EVALUATION

Although Contextual Design is primarily aimed at a systems design environment, for the purposes of this study, Contextual Design provided a methodological framework that suited the aims of the study and that allowed the easy handling of a mass of qualitative data. This was an interesting application of the method and resulted in some interesting insights. In general, the method proved to be practical and useful, but showed that there were some areas that needed to be adapted.

The data gathering techniques worked exceptionally well for the purposes of KM research when focusing on CoPs. It is not surprising that this stage worked so well, for it has its roots in ethnographic procedures. In later stages when the focus became tighter, the contextual inquiry techniques were too exploratory and more focused questions were needed alongside. This, too, was not unsurprising, as Contextual Design also has roots in the Discovery of Grounded Theory (Glaser & Strauss, 1967), which observes that data should be gathered iteratively in order to inform the findings of previous iterations. The iterations become more focused and the questioning tighter.

Contextual Design provided five models and an Affinity with which to handle the mass of ethnographic data. These models are intended to be the beginning of a work re-design process, but in the case of this project, they were used for the analysis of the qualitative data. The analysis of Contextual Design data is intended to be a team activity. In the case of this study, I was working alone. This did not prove to be a disadvantage as it meant I obtained a comprehensive view of the data. In order to avoid too narrow a view, however,

1. colleagues and interested parties were brought in to "walk the Affinity" and to provide different insights, and

2. a set of propositions was created to take back to the interview respondents. I could then use the feedback to the propositions to provide further insights to the Affinity and models and provided a form of validation.

Taking these two steps overcame the limitations imposed by working alone.

The Affinity is a standard qualitative technique that sorts the data into themes and categories and as such was an essential stage in the process, providing a number of insights.

It had initially been expected that the Artefact Model would be a useful model as the research was exploring the use of artefacts in a CoP. Unfortunately, it did not prove to be useful as it focused on the structure of an artefact with a view to embodying this in a system. For example, it would explore the structure of a calendar and how people used it, in order to use those insights in a personal information management tool. A better picture of the use of the artefact in the CoP was obtained by creating a different model, in the form of a timeline, to track the creation, iterations, and use of the artefact. This provided a picture of the development of the planning document over time and its use in the community. It showed the people involved, some of the work that went into its preparation, the number of iterations, where it was used, when it crossed the boundaries between the cores, and also how it was stored and displayed (for example, used in a meeting or placed on the intranet for comment). As the planning document became something of a focus, the timeline provided a means of pulling all the data about its development and use into one model. It did not, however, show the application of the softer aspects of people's knowledge. This understanding was built up through the use of the timeline, the Affinity, and the Flow Model. For the purposes of planning KM issues, it would be useful to further develop the timeline model to include more details on the development and use of an artefact by the CoP members.

The purpose of the Flow Model was to abstract the work done by the members of the CoP into roles. It was also helpful in mapping some of the movements of the planning document. This worked well and provided a number of insights into the working of the CoP.

The Sequence Model also worked well in that it provided a means of representing the actions taken, suggesting reasoning behind the actions, and steps taken to accomplish an action. This provided a number of insights into how the CoP accomplished some of its tasks.

The Cultural Model was satisfactory to the extent that it provided a visual representation of the impact of culture on the CoP. The notion of culture in the model, however, is restricted to issues (and people) that hinder work and enable it and breakdowns in the process. Culture in the broader sense, that is, national, organisational and group cultures, was not addressed but was a theme that came out strongly in the Affinity. A useful development would be to further develop the Cultural Model to cater for a less-restricted notion of culture.

The model that contributed the least was the Physical Model. This may be partly due to the fact that, although it was used properly as per the method, it was not adapted sufficiently. The model was created as directed, but it became clear during the analysis stage that the model would have benefited from being broadened to reflect the wider context of the CoPs operation, as opposed to the immediate physical environment of each member. The Physical Model was used to record the immediate physical environment of the members. This provided no insights at all, other than to emphasise the importance of ICT. It would perhaps have been preferable to record the wider physical environment and illustrate the difficulties of the distributed environment.

Once all the models and the Affinity had been created, they were "read," and any insights or findings recorded. These were then used as a focus for the next stages, which in turn further informed the findings to increase the understanding of the situation. The timeline was not used at this stage, being developed at the beginning of Stage Two of the main case study in order to bring the development of the planning document up-to-date.

Using the Contextual Design as a methodological basis worked very satisfactorily from a practitioner point of view. It must be emphasised, however, that it was only satisfactory as a basis, and that the method had to be adapted by:

• adding an extra stage of creating propositions with which to return to the respondents;

• changing the Artefact Model;

• using structured, open-response interviews in the later stages [In the later stages, the only models that were used were the Sequence Model and the adapted Artefact Model (timeline). It was not felt that the other models would add anything extra at this stage.]; and

• extending the Physical Model to reflect work in the distributed environment.

Further improvements that could be made to the method for use in investigating work from a KM standpoint with a view to developing a KM initiative would be:

• Further developing the Artefact Model to reflect the application of people's softer knowledge.

• Adapting the Cultural Model to reflect a richer definition of culture.

• Developing a model to reflect the social networks of the organisation. The Flow Model does this to some extent but translates the flows into work roles. A model to map the social network in an organisation would be particularly useful. It could be mapped in stages or smaller units and consolidated to cover a wider group.

It must also be borne in mind that the focus of this project was an organisational CoP that operates in a distributed international environment. It is unlikely that the method would have proved so effective in exploring the interactions of a totally virtual community.