Roel W. Schuring
University of Twente, The Netherlands
Ton A. M. Spil
University of Twente, The Netherlands
Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.
User-adoption of new IT-applications is the proof-of-the-pudding when it comes to IT-success in healthcare. As a consequence, many studies are made of the role of the users in the introduction of new IT in both theory and practice. This paper introduces relevance and micro-relevance as key determinants of IT-diffusion and IT-use, respectively. Relevance is the degree to which the user expects that the IT-system will solve his problems or help to realize his actually relevant goals. Micro-relevance is the degree to which IT-use helps to solve the here-and-now problem of the user in his working process. Central to both concepts is the degree to which goals or problems that IT is related to are actually pressing in nature. Goals and problems that are less pressing do not result in relevance. Goals that are pressing may still not be micro-relevant on the level of executing work. A study amongst 56 general practitioners (GPs) on the introduction of an Electronic Prescription System (EPS) demonstrated the importance of relevance and micro-relevance. To these GPs, time-pressure and communication with pharmacy and hospitals were highest on the agenda. In that light, the innovation of EPS was not relevant. Lack of micro-relevance level obstructed intensive use of EPD by those who had introduced the innovation. As a consequence, those who adopted the EPD only used it sparsely.
Every improvement is a change but not every change is an improvement. (Heraclitus)
The central perspective of our approach is that user-adoption of new IT-applications is the proof-of-the-pudding when it comes to IT-success. This contribution will elaborate on two factors that determine the diffusion and the use of IT: relevance and micro-relevance. We will use an example in the healthcare sector to illustrate how these factors work. Healthcare is changing (Tange, 1997), and information and communication technology is a driving force for many of these changes (Suomi, 2001). At the same time, many authors report cases of failure of ICT innovations (Southon et al., 1999; Gelderman, 1998). The cause of these failures is often searched for in resistance and user participation (Ballantine et al., 2000; Offenbeek & Koopman, 1996). Professionals in healthcare organization get the blame for opposing changes in their working processes. But Pare & Elam (1999) state that they are positive about using information systems to access up-to-date knowledge, for continual medical education, for access to healthcare in rural and remote areas, for the quality of patient care and for the interaction within a healthcare team. Also, Timpka (1989) demonstrated the fundamentally positive attitude of physicians towards the use of IT. So, there is no resistance to change, there is only resistance to bad change (Barlow, 2001). Therefore, the assumption is made that there is more to IS success than resistance alone. While resistance is still important, there is an increasing awareness that there is a wide range of more complex organizational and people-related factors to be taken into consideration (Lorenzi & Riley, 1994). Southon et al. (1999) concludes that the capability to bring IS benefits is severely compromised by our inability to adequately address the problems of the healthcare working process. So, user-perspectives are by no means new to IT-evaluation in general, or in the healthcare sector specifically. Still there is a degree of "magic" in successful IT-introductions, as compared to the "disappointing surprise" after failures. We contribute to the topic by focusing on the relevance factor as an important success factor for ICT implementation in healthcare and probably beyond healthcare. Or as Whyte et al. (1997) sharply observes, "we must understand the attributes of information systems that users perceive to be important."
We can use a wide range of sources that discuss user-perspectives in IT-introduction. This section gives a short overview of intriguing literature. The aim is to demonstrate that relevance and micro-relevance are not the determinants of user-adoption. Rather, they are important determinants among other factors. One of the ultimate goals of our research project in this field is to propose a model that neatly balances the role of such factors.
First, such factors may be looked for in general literature on change and on the introduction of new technologies. For example, in the balance model of organizational change risks, Leavitt (1965) introduced four domains in which these risks will occur: tasks, structure, technology and people. People, at that time, were not the most important domain. Offenbeek & Koopman (1996) connect people with resistance potential because they can feel that the quality of their working life will be decreased. Mumford (1983) observed that user participation contributes to effective organizational change. Wissema (1987) defines resistance as willingness to change and the difference between results and expectations.
When we focus on IT introduction more specifically, we again see a number of interesting sources. Thong & Yap (1995) discuss the user-satisfaction approach to IT effectiveness. They mention the debatable operationalization, poor theoretical construct and misapplication as critics to the approach. On the basis of their review, they conclude that attitude is the construct that lies at the root of user-satisfaction, and suggest ways to improve operationalization and measurement of attitude. Paré & Elam (1999) studied attitudes, expectations and skills in relation to physicians' acceptance of IT systems. Physicians with formal training on computers were more knowledgeable about informatics concepts and reported that computers would be more beneficial to healthcare, although it is not clear whether the training causes this attitude. Also, it becomes clear that user-priorities regarding IT-innovations vary strongly. The functional uncertainty is often described in information systems literature. It occurs in the task domain of Leavitt. In each situation, the interpretation and the meaning can be different. Therefore, it is necessary to establish a functional specification with user and providers of the information systems. Henry & Stone (1999) state this to be information quality. Within the healthcare sector, Walley & Davies (2001) conducted a study to the internal barriers to technological IT-advancement in the healthcare sector. The involvement of stakeholders is arguably one of the most distinctive characteristics of IT projects. There are instruments to identify user-needs, but they question whether they are actually used. Van der Pijl (1994) shows that there is more to say about people than just resistance or user participation. Both users and providers of information systems have their own targets, not necessarily going hand-in-hand. A central question is whether the provider intention is the same as the user interpretation (Sperber & Wilson, 1986). We will re-define relevance in the next section. Finally resources, (human, physical and monetary components, Ansoff, 1965) are needed to implement the new information system into the organization. The human resources can both be insufficient in time and in experience (risk of technology). Insufficient material resources (Offenbeek & Koopman, 1996) will have a limiting influence on the other three risk domains.
In this chapter, we will focus on a single determinant to user-adoption of IT in healthcare, i.e., relevance. We keep in mind that, although relevance plays a major role in determining IS success on user-level, it only plays its role within the context of other factors that were briefly mentioned in this section (Schuring & Spil, 2001). It is most important to elaborate the construction of a framework that brings these factors together. Saarinen & Sääksjärvi (1995) point out that different factors act as critical success factors under different circumstances. This will also apply when a framework of success factors is limited to user-related factors. However, in this chapter we will solely concentrate on relevance and the empirical findings in this area.
Saracevic (1975) defines relevance is a measure of the effectiveness of a contact between a source and a destination in a communication process. This is a somewhat abstract wording of what we would define as: "the degree to which the user expects that the IT-system will solve his problems or help to realize his actually relevant goals." There are three dimensions that are kept implicit in Saracevic' definition that we altered. We use the word "expects," since we want to stress that relevance is a factor that is important in the course of the adoption process, not only in evaluation. Second, instead of effectiveness we use "solve problems and goals." By doing so, we imply that effectiveness has two dimensions: to take away existing negative consequences (problems), and to reward with positive consequences (reach goals). Third, the word actual is crucial in our view of relevance. Relevance is not to be confused with the degree to which the user considers outcomes as being positive. The set of outcome-dimensions that someone considers "positive" is larger than the set of outcome-dimensions that are relevant. Imagine a physician, who basically considers IT-outcomes of a computer decision support system, such as assistance in diagnosis, disease prevention, or more appropriate dosing of drugs (Thornett, 2001), as "positive." This does not automatically imply that the IT-adoption is relevant to him. We would say it is only relevant if these dimensions are high on his goal agenda. That is why we use the word actual. Again, this is a more explicit wording of a dimension that is implicitly included where Saracevic' uses the word effectiveness in his definition. The actually relevant goals may be a mix of short-term goals and long-term goals. If, for example, smooth communication with hospitals or pharmacy is his prime actual problem or goal, the user will only consider the IT-innovation as relevant when it actually helps to improve that communication, notwithstanding the fact that he might have a positive attitude towards that innovation as long as the innovation helps to solve other problems or other goals that are on the lower positions in his agenda-ranking. We discovered in our case studies that it is not sufficient for an innovation to effectuate a positive attitude amongst users. The IT-innovation should be relevant.
Micro-relevance is a related concept that can be used to describe a similar phenomenon once the new IT is installed. Micro-relevance is defined as "the degree to which IT-use helps to solve the here-and-now problem of the user in his working process." The use of new equipment or new IT-procedures is a conscious activity. In every conscious activity that is goal-oriented to a specific goal, there is a reason why that course of action is being chosen. Similar to what was discussed above on "relevance," not every course of action that a user basically considers as "positive" is "micro-relevant." Again, let's illustrate this with an example. Imagine a patient with virus infection visits a physician. The physician might notice the similarity to a number of other patients he has met that week and decide on diagnosis and treatment fairly quickly. To this doctor, the use of a decision support system to determine diagnosis is not micro-relevant. However, a colleague of his may not feel so confident and thus use the diagnostic support system. We discovered that micro-relevance is a key factor in explaining IT-use in our case studies. Box 1 gives an overview of relevance as we propose to use it.
Box 1: Relevance and micro-relevance
FRAMEWORK OF IT RELEVANCE TO USERS
Relevance and micro-relevance are notable refinements of the way the role of the user is being discussed in the existing literature. Thornett (2001) implicitly refers to relevance and micro-relevance when he discusses limited adoption and use of DSS by primary physicians where "consultation time is lengthened by their use and there is no appreciable impact on patient satisfaction." It is an example where other outcomes that are basically considered as positive (as mentioned above: better diagnosis, more appropriate dosing of drugs, and other) are overruled by limited relevance and micro-relevance.
Saracevic (1975) provides a historic positioning of relevance. The roots lay in the 1930s and 40s, where the distinction between information and relevant information is made by Bradford (Saracevic, 1975, p. 324). In order to make the distinction between relevant and non-relevant information, he discusses the nature of communication. By doing so, he recognizes that relevance to a subject depends on specific dimensions, for example, the subject's knowledge, representation and values. He discusses a number of (philosophical) approaches to relevance. The elaboration we propose above builds on the radical pragmatism-perspective or, more specifically, Cooper's (1971) utility function: "Relevance is simply a cover term of whatever the user finds to be of value about the system output, whatever its usefulness, its entertainment, or aesthetic value, or anything else." Wilson (1973) adds to this that relevance is situational. Ballantine et al. (1999) put it in the following way: "Depending on the type of task, the information generated by the system may be more or less appropriate, which will affect its success or failure." Saracevic (1975) distinguishes various other approaches to relevance, of which a number focus on the basic source of relevance, such as logical relevance, the nature of interference and the pertinence view of relevance. We are very much aware of the fact that our elaboration of relevance does not, in full, retain the differences between those points of view. It is merely a practical elaboration that we use to predict user-adoption.
The pragmatic perspective of relevance that we choose resembles the notion of "relative advantage," as discussed in the Innovation Diffusion literature by Rogers. Rogers (1983, 1997) reserves a central role for "relative advantage," which is the user's view of "the degree to which an innovation is better than the idea it supersedes." Relative advantage can be economic or social. Rogers adds that, "The nature of the innovation largely determines what specific type of relative advantage is important to adopters, although the characteristics of the potential adopter also affect which dimensions of relative advantage are most important." Based on a review of hundreds of empirical studies, Rogers concludes that relative advantage explains 49% of the rate of adoption of innovations.
It is most notable that the organizational factors are not explicitly included in our user-relevance framework. It should be kept in mind that the user's agenda of problems and goals depends on his role in the society (Barnard, 1938). The influence of the organization on this agenda depends on many aspects, including the involvement with other organizations, on time and on place. As a consequence, our framework reflects the actual impact that organizational goals and preferences have on the user, and thus, on organizational behavior.
Also, other scholars have detailed relevance to levels or factors, for example, Saracevic (1996), Schamber (1994), and Cosijn & Ingwersen (2000). Our distinction between relevance and micro-relevance takes a different starting point, as it distinguishes between the situations where the user is or is not actually working in a process in which he may use the IT system as a major difference.
Our research was set up to both assess the situation regarding the electronic prescription system "EVS" in the Netherlands and the theory that is described above that was set up to provide an instrument that could be used to unravel the diffusion-situation of the prescription system. This resulted in a case-study protocol that covers all the topics that are mentioned in the framework in openended questions. In line with the case-study approach by Yin (1984), we discerned different case-situations on the basis of our theoretical framework. Particularly, the network-situation of general practitioners and the degree of adoption of previous ideas served as a basis to make categories of general practitioners. A total of 56 case studies were conducted. Each general practitioner was visited in his/her own working situation and interviewed for over an hour. We had data available on the size of each category, which enabled us to quantify the qualitative data that we gathered.
Under the header relevance for the GP, questions were asked about problems or wishes that the GP experienced as important at the moment of asking, during implementation of the Electronic Prescription System. Figure 1 gives an overview of all the situational relevance factors mentioned. Here we will summarize the results of the main seven:
In more than half of the cases, improvements to communication with colleagues, pharmacists and hospitals were crucial. The EPS system does not deliver these features. Saarinen & Saaksjarvi (1992) measured the improved internal communication and improved inter-organizational communication under the header "impact of the IS on the organization." None of these success factors was satisfied in our cases.
In 55% of the cases, and independently of each other (the term was not mentioned by the interviewee), the GPs stated that there should be a diminishing of the time pressure. Both in the description of the EPS, as well as in international literature, it is made assumable that EPS will not diminish the time of consult (Mitchell & Sullivan, 2001; Thornett, 2001).
Forty-five percent of the case studies reported that the GP expected a fee in return for going through the trouble of implementing and using EPS. At the moment of interviewing, it was not clear what financial profit the new system would deliver for the GP, although it was expected that the revenues for the government would be substantial by saving on expenses on drugs.
In about 20 case studies, lack of trust in the existing software and in the software supplier were mentioned as a barrier for (wanting to) use the new EPS. They said that first things had to change in the GP-IS market and in the GP-IS itself before EPS could be a success.
About the same amount of GPs want to retain freedom of choice for medication of the patient. Although this seems to be a resistance matter, it is also a relevance matter because the EPS does not comprehend new ideas and new treatments, which are already known in the general practice.
Although the use of ICPC seems useful to many GPs (in structuring and communicating), the time that it will cost to find the right code and the omissions of some codes will form a barrier for EPS use.
Twenty percent of the GPs make use of a personal or regional formulary. The EPS makes use of a formulary of the Dutch council for GPs and often does not have the possibility to keep the own formulary when an update of the software is installed.
Figure 1: Case study results
Finally, once the computer-system was installed, use of the system was mostly sparse. The way of working was relatively complicated and added relatively little value in most patient-doctor contacts.
We analyzed the empirical results of the previous section with the characteristics of relevance and micro-relevance as described in Box 1. User relevance tells us more about the adoption of the system and micro-relevance shows improvements on the working process. The relevance of the EPS to general practitioners was, on the average, very low. The economic and social benefits were scarce, as the main problems on the agenda of general practitioners were (1) communication between practitioners, with the hospital and with the pharmacy and (2) time pressure. The EPS did not help to solve these problems. Also, the initial cost was high to many practitioners, as the IT-use in their starting-situation was by far not sufficient to be able to use the EPS system. About 27% of the general practitioners make sufficient professional use of the computer to be able to start right away with the EPS-use. The remainder first would have to further structure their electronic patient records, or would even have to start using the computer professionally at all. The economic benefits of the system are potentially substantial, but they do not appear for the user but for the ministry and for the healthcare insurance companies. The costs involved with updating the system are higher than the benefits. A promise is made on practical support, but not for every practice such service becomes available. The immediacy of the reward is therefore not recognizable. Socially, the new system is more seen as a threat than a benefit. The patient-GP interaction changes, as also noticed for micro-relevance, and this change is not for the good. The quality of the GP service can improve as Sullivan & Mitchell (2001) pointed out. Saving in time and effort is not to be expected, as the same authors point out, which is seen as most wanted by the GP's. This characteristic is most critical in our opinion.
Beside the general relevance of the new system we want to know if the system improves the working process of the user, the micro-relevance. Although GPs say double-checking prescriptions by use of the EPS would be relevant to them, it is more micro-relevant to stay in personal touch with his patient during the visit than to double-check if the intended prescription can be further optimized. Where the barrier to use the ICPC code was already high in general terms, we see that the micro-relevance is really problematic. There is a high start-up barrier to start to search for such coding during the prescription process for one specific patient (In fact, the initial use of any particular code takes learning-time since the codes were not sufficiently clear and accessible. The micro-relevance of that learning is negative, since it will take a lot more time and effort to complete the prescription for that initial patient). This also made the immediacy of the reward within the use process meager. During the study, we discovered that relatively many General Practitioners used the EPS system after the consult. We tend to conclude that the micro-relevance after the patients' consult is higher than during the consult. When GPs double-check their prescriptions after the consult, there is generally less time-pressure (as this will only be done for those few patients where the GP feels uncertain). Also, there are few other ways to double-check prescriptions afterwards. This also contributes to the micro-relevance.
Relevance has long since been a central notion to IT-theory. In our study, we monitored a great number of factors that (co) determine IT diffusion and use in 56 case studies. These cases provided us with enough evidence that, for this particular (electronic prescription) system in this particular (healthcare) branch, relevance and micro-relevance were by far the most important determinant for failure of diffusion and use of the system.
We agree with Wilson (1973) that, "relevance is not a single notion but many." We feel that the distinction between user relevance and micro-relevance follows the more philosophical thoughts of Schutz (1970) when he introduced motivational relevance (course of action to be adopted) and topical relevance (perception of something being problematic).
In terms of our framework new ICT innovations will not be adopted without relevance and micro-relevance to the user.
Althuis, T. R., & Rikken, S. A. J. J. (2000, April). Electronic support for general practitioners in prescribing Drugs. Health Information Developments in the Netherlands, pp. 62–66.
Ballantine, J., Bonner, M., Levy, M., Martin, A., Munro, I., & Powell, P. L. (1998). Developing a 3D Model of IS Successs. In Garrity & Sanders, (eds.) Information Systems Success Measurement. Hershey, PA: Idea Group Publishing.
Barnard, C.I. (1938). The Functions of the Executive. Cambridge, MA: Harvard University Press.
Barlow, C. (2001). Following and accelerating the design evolution curve in healthcare. First International Conference on Management and Healthcare and Medical Technology. Enschede.
Bergeron, B. P., & Bailin, M. T. (1999). Medical Information Technology: a vehicle for change. International Journal of Healthcare Technology and Management, 1 (1), 29–45.
Cooper, W.S. (1971). A definition of relevance for information retrieval. Information Storage and Retrieval, 7 (1), 19–37.
Cosijn, E., & Ingwersen, P. (2000). Dimensions of relevance. Information Processing and Mangement, 36, 533–550.
Delaney, B. C., Fitzmaurice, D. A., Riaz, A., & Hobbs, F. D. R. (1999). Can computerized DSS deliver improved quality in primary care? British Medical Journal, 312, 1008–1012.
Finley, P.N., & Forghani, M. (1998). A classification of success factors for decision support systems. Strategic Information Systems, 7, 53–70.
Garrity, & Sanders. (1999). Information systems success measurement. Hershey, PA: Idea Group Publishing.
Gelderman, M. (1998). The relation between user satisfaction, usage of is and performance. Information & Management, 34, 11–18.
Henry, J. W., & Stone, R. W. (1999). End user perception of the impacts of computer self-efficacy and outcome expectancy on job performance and patient care when using a medical information system. IJHTM, Special issue on advances in the management of technology in Healthcare, 1 (1/2), 103–124.
Lagendijk, P.J.B., Schuring, R. W., & Spil, A. A. M. (2001). Het Elektronisch Voorschrijf Systeem, van kwaal to medicijn. Universiteit Twente, Enschede.
Linnarson, R. (1993). Decision support for drug prescription integrated with computer-based patient records in primary care. Medical Information, 18, 131–142.
Lorenzi, N. M., & Riley, R. T. (1994). Organizational aspects of health informatics. New York: Springer-Verlag.
Mason, R. O. (1978). Measuring information output: A communication systems approach. Information & Management, 1 (5), 219–234.
Mitchell, E., & Sullivan, F. (2001). A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980–97. British Medical Journal, 322, 279–282.
Mumford, E. (1983). Designing human systems for new technology. Manchester Business School.
Offenbeek, M. van & Koopman, P. (1996). Interaction and decision making in project teams. In M.A. West, (ed.), Handbook of Work Group Psychology. New York: John Wiley & Sons, Ltd.
Pare, G., & Elam, J. J. (1999). Physicians' acceptance of clinical information systems: an empirical look at attitudes expectations and skills. International journal of Healthcare Technology and Management, 1 (1), 46–61.
Pijl, P.G. J. van der (1994). Measuring the strategic dimensions of the quality of information. Journal of Strategic Information Systems, 3 (3), 179–190.
Rogers, E. M. (1983). Diffusion of innovations. New York: The Free Press.
Rogers, E. M., & Scott, K. L. (1997). The diffusion of innovations model and outreach from the National Network of Libraries of Medicine to Native American Communities. Retrieved from http://www.nnlm.nlm.nih.gov/pnr/eval/rogers.html
Sperber, D., & Wilson, D. (1986). RELEVANCE, communication and cognition. Oxford: Basil Blackwell.
Saarinen, T., & Saaksjarvi. (1992). Process and product success in information system development. Journal of Strategic Information Systems, 1 (5), 266–277.
Saracevic, T. (1975). Relevance: A review of and framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26 (6), 321–343.
Saracevic, T. (1996, October 13–16). Relevance reconsidered: '96 Proceedings of COLIS 2: Second International Conference on Conceptions of Library and information Science: Integration in Perspective (pp. 201–218). Copenhagen, Denmark: The Royal School of Librarianship.
Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29, 3–48
Schuring, R. W., & Spil, T. A. M. (2001). Relevance as a major driver of Innovation and diffusion of ICT in Healthcare Organizations. First international conference on management and healthcare and medical technology, Enschede.
Schutz, A. (1970). Reflections on the problem of relevance. New Haven, CT: Yale University Press.
Southon, G., Sauer, C., & Dampney, K. (1999). Lessons from a failed IS iniative: Issues for complex organisations. Medical Informatics, 55, 33–46.
Spil, T. A. M. (1996). The effectiveness of strategic information systems planning in professional organizations. PhD thesis, Enschede.
Sullivan, F., & Mitchell, E. (1995). Has general practitioner computing made a difference to patient care? British Medical Journal, 311, 848–852.
Suomi, R. (2001). Streamlining operations in healthcare with ICT. In R. A. Stegwee & T. A. M. Spil, (eds.), Strategies for Healthcare Information Systems. Hershey, PA: Idea Group Publishing.
Tange, H. (1997). Medical narratives in the electronic medical record. PhD Thesis, Maastricht.
Thornett, A. M. (2001). Computer decision support systems in general practice. International Journal of Information Management, 21, 39–47.
Timpka, T. (1989). Introducing hypertext in primary healthcare: A study on feasibility of decision support for practitioners. Computational Methods and Programs in Biomedicine, 25, 49–60.
Walley, P. & Davis, C. (2001). Implementation IT in NHS Hospitals: internal barriers to Technological Advancement." Paper presented on the 1st Hospital of the Future Conference, Enschede, The Netherlands.
Whyte, G., Bytheway, A., & Edwards, C. (1997). Understanding user perceptions of IS Success. Journal of Strategic IS, 6, 35–68.
Wilson, P. (1973). Situational relevance. Information Storage and Retrieval, 9 (8), 457–471.
Wissema, J. G. (1987). Angst om te veranderen? Een mythe! Assen: Van Gorcum.
Yin, R.K. (1984). Case study research, design and methods. London: Sage Publications.
Zviran, M., & Armoni, A. (1999). Integrating hospital information systems: a bottom up approach. IJHTM, Special issue on advances in the management of technology in Healthcare, 1 (1/2), 168–179.