Whilst managers may have long known that ‘if you can’t measure it, you can’t manage it’, what does that maxim mean in the context of a wiki implementation? In other words, when evaluating a wiki implementation what can be measured and how should it be done? In an earlier post I indicated that:
“Measuring users’ progress through adoption stages and how often people are using wikis will provide some elementary figures on wiki diffusion and infusion in the organisation, and may provide grounds for investigating any barriers to the implementation process. However, more difficult issues relate to evaluation of wikis’ impact on bottom-line performance and development of organisational learning practices. Measurements focusing solely on bottom-line performance improvement in terms of accelerated project cycle times, reduced email overload and search costs may provide some hard data to support ROI, but they do not consider more important effects of wiki management/usage on organisational learning and collaborative capability development. Not only is it more difficult to establish direct causal connections between wiki management/use and improvements here, any evidence would be in the form of people’s opinions/perceptions.”
During my discussion with Euan Semple regarding the issue of wiki evaluation, he highlighted how the emergent nature of wiki usage and the wiki itself requires “conversations and actions, not pre-planning and control”. Consequently, in his experience ‘evaluation’ has been a continuous process requiring managers to be (i) awake to how people are working/using the wiki by engaging with and being open to user feedback and (ii) prepared to amend original ideas about the implementation and allow/encourage users to take responsibility for ensuring the wiki meets their needs.
Contrasting that ‘emergent’ approach to evaluation, Ross Mayfield described a more directed/planned approach where initial goals, milestones and indicators/measurements are identified at the outset and later used to establish progress and/or reassess plans. However, he did highlight the difficulties of measuring benefits associated with fostering transparency, innovation and culture change, or establishing whether any improvement in a targeted work process is directly attributable to the wiki, which tends to rely on soft data.
To investigate what is actually happening in practice, I asked interviewees and survey respondents whether and how they evaluate their wikis. Whilst most interviewees indicated that no formal evaluation takes place, when asked whether their companies have learnt to better manage/use wikis, they identified a range of initiatives to improve wikis including change of the wiki-structure to reflect active communities of practice, improved training mechanisms, and seeking out of best practices (e.g. from WikiPatterns).
Likewise, the majority of survey responses (30% of responses) indicated that no feedback was sought/given regarding the wiki. However, where evaluation had been undertaken it focused largely on the wiki itself (e.g. content maintenance, the ability to locate information and ease of use) rather than the implementation process (e.g. identifying collaboration and training needs)
So despite the above maxim, and the key theme espoused throughout ‘learning organisation’ literature (i.e the need for continual and live attention to ensure processes, skills and structures encourage the best possible feedback from outside the organisation, and between all elements within the organisation), it appears that evaluations tend to be ad-hoc rather than formal assessments of the wiki and the implementation process, with many organisations not actively seek feedback, and instead relying on more subtle forms of evaluation that occur seamlessly as people use the wiki and adapt it to their needs accordingly. Furthermore, where feedback has been sought, it has focused on wikis’ utility rather than internal usage patterns, transition mechanisms and user commitment/drop-out levels (i.e. the implementation process and barriers thereto).
Such approaches overlook the value of evaluating the implementation process and fail to view evaluation as central to that process. In other words, evaluation must be a continuous process providing opportunities to engage in dialogue, discover barriers to wiki use/growth and possible solutions thereto, not an activity which is tacked on at the end. That approach reflects the importance of understanding the needs wiki use/management is endeavouring to satisfy, which can aid in setting flexible goals to guide development of capabilities at all organisational levels, changes to organisational systems, and subsequent assess thereof.
