Study Findings

---

Round 1-Inventory Round

In October 1999, a total of 117 "inventory" round questionnaires were distributed to individuals who had been substantially involved in the five government agencies' SAP Financials Project. Before the e-mailout, the survey questionnaire (Word attachment) and covering email were pre-tested for clarity and ease of understanding by several senior personnel in the government agencies. Minor cosmetic changes resulted. In all, 78 questionnaires were returned, yielding a 67% response rate. A total of 61 valid questionnaires were eventually obtained from the first-round survey (Table 1), providing a net response rate of 52%. More than two-fifths (44%) of the respondents were from Agency A, the lead agency on the implementation and a corporate services provider to the other agencies. Other agencies had comparatively fewer participants.

Table 1: Inventory Round Survey Responses.

Organization	#	%	Role	#	%	Level	#	%
IP	7	11	IP	7	11	Strategic	12	22
Agency A	27	44	Agency	54	89	Operational	42	78
B	12	20
C	7	11
D	2	3
E	6	10
Total	61	100	Total	61	100	Total	54	100

Respondents from the five agencies were further differentiated by organizational level of involvement, where (1) strategic = steering committee members, project sponsors, project managers, and (2) operational = business process team members, power users, help-desk team members, change-management team members. Approximately four-fifths (78%) of the respondents were involved at the operational level, the rest (22%) representing the strategic level.

Respondents were asked to identify any issues regarding implementing, managing, and supporting the SAP Financials throughout their life cycle in their 'home' agency. The 61 respondents identified 274 issues, or an average of 4.5 issues per respondent. Approximately 41% or 115 of the issues identified originated within Agency A. This is not surprising given the lead role played by this agency and given that 44% or 27 of the total respondents are from this agency. Approximately one-tenth of the issues identified were from the implementation partner. From within the agencies, approximately 28% of the issues identified were from the strategic level and 72% from various operational levels. In general, the number of issues identified by the various respondent groups was in proportion to the number of respondents in these groups. Table 2 shows responses by stakeholder groups.

Table 2: Cross-Tabulation of Responses by Stakeholder Groups.

	Issue		Response		I/R		Issue		Response		I/R		Response		Issue		I/R
Organization	#	%	#	%		Role	#	%	#	%		Level	#	%
IP	26	10	7	11	3.7	IP	26	10	7	11	3.7	Strategic	75	28	12	22	6.3
Agency A	115	41	27	44	4.3	Agency	248	90	54	89	4.6	Operational	173	72	42	78	4.1
B	48	18	12	20	4.0
C	34	12	7	11	4.9
D	14	5	2	3	7.0
E	37	14	6	10	6.2
Total	274	100	61	100	4.5	Total	274	100	61	100	4.5	Total	248	100	54	100	4.6

Having identified 274 issues from 61 survey respondents, the study then sought to distill these issues into a summary set of major-issue categories and related sub-issues. This resulted in a "tentative set" of 12 major-issue categories, with 40 sub-issues pending further validity and reliability testing in Round-Two.

As a validity test, and in order to establish a summary set of major-issues representing the respondents' main concerns, a domain experts' workshop was conducted soon after the "tentative set" of major-issues was derived during July 2000. Four out of five representatives from the government agencies and five research team members agreed to participate in this "synthesis" workshop. The workshop was organised to allow time for information sharing and discussion with the participants. The workshop yielded valuable insights and a greater level of understanding of SAP Financials issues in the agencies and resulted in a "preliminary set" of major-issues that were more relevant and meaningful to the study stakeholder groups, pending confirmation from all survey respondents.

Round 2-Confirmation Round

Having rationally synthesized and logically restructured the "preliminary set" of issue categories and related sub-issues through the coding and synthesis exercises and domain experts' workshop, in the second "confirmation" or interim survey round, the study sought respondents' comments on and confirmation of the "preliminary set" of major-issues. For each respondent from Round-One, a custom report was prepared. The report included the hierarchy of major- and related sub-issues in the "preliminary set." The report also clearly indicated the link between each of the respondent's original round-one issues and the related sub-issues with which they had been associated. A total of 61 Round-Two reports were distributed to individuals who had responded in the Round-One survey. Although participants were instructed that there was no need to formally respond if they agreed in principle with the "preliminary set" of major-issues, about one quarter of questionnaires were returned showing their further comments and agreement.

The comments on and confirmation of the issue categories from the domain experts' workshop and the Round-Two survey respondents, resulted in a minimally revised "master set" of 10 major-issue categories from M-1 to M-10 with 38 sub-issues from S-1 to S-38 (Appendix A). Figure 1 shows the incidence of the initial 274 issues from the 61 respondents across the 10 major-issues categories.

Figure 1: Distribution of Issues Across the 10 Major Issue Categories.

Using the incidence of overall citation as an early crude indicator of severity, it is noted that 63% (172) of all 274 initial issues cited pertain to: Operational-Deficiencies (67 issues); Knowledge-Management (55 issues); and System-Development (50 issues). We recognise that the number of sub-issues in other major-issue categories were relatively fewer, accounting to some extent for the lesser citations, and that not all issues listed are issues for all respondents (a further fallibility of citations as an indicator of issue severity). Nonetheless, the aim of the study was to be as inclusive as possible in this master set of issues, with further relative evaluation in the next "weights" round of the survey.

Round 3-Weights Round

During September-October of 2000, a total of 100 Round-Three questionnaires were sent to Round-One contacts, excluding those who had indicated they would be unable to participate but including those who had not responded in the previous rounds. Respondents were asked to rate the importance of each of the 38 sub-issues on a scale from 1 to 10 where 1 means "not important" and 10 means "very important." Prior to its e-mailing, the survey was pre-tested for clarity and ease of understanding by several senior personnel in the government agencies. Slight changes were made.

Approximately one week after the due date, in an effort to boost the response rate, follow-up e-mail messages and phone calls were made to those who had not yet responded. When necessary, another copy of the questionnaire was e-mailed to those respondents who had "misplaced" the survey. The follow-up phone calls resulted in 15 additional returns. A total of 58 questionnaires were returned, yielding a 58% response rate. A total of 42 valid questionnaires were eventually obtained from the final round survey, providing a net response rate of 42%. Known reasons for non-response were: some respondents had discontinued their SAP responsibilities; others had left their organization; some were on holiday or maternity leave; several respondents did not wish to participate because of the time required to complete the questionnaires. The distribution of the survey respondents in this final-round survey by agency, role, and organizational level is shown in Table 3.

Table 3: Third-Round Survey Responses.

Organization	#	%	Role	#	%	Level	#	%
IP	6	14	IP	6	14	Strategic	11	26
Agency A	15	36	Agency	36	86	Operational	25	74
B	7	17
C	3	7
D	3	7
E	8	19
Total	42	100	Total	42	100	Total	36	100

Table 4 shows the overall mean scores and rankings of the 10 major-issue categories from the "weights" round survey (where the mean for the major-issue is simply the average of the mean scores for its constituent sub-issues). A total of 1,134 valid scores for the 38 sub-issues were received from the 42 respondents (71 % = 1,134 / (42*38)). The number of respondents varies between 29 and 34 across the sub-issues from which major-issue scores are derived.

Table 4: Overall Ranking of Major Issue.

M-#	Mean	Std Dev	Rank	Major Issue Categories
3	6.19	2.53	1	Knowledge-management
9	6.00	2.34	2	System-development
8	5.79	2.68	3	Support
2	5.69	2.97	4	Data-conversion
5	5.62	2.73	5	Operational-deficiencies
4	5.58	2.53	6	Lack-consultation
1	5.25	2.86	7	Cost-benefit
6	5.06	2.70	8	Organizational-context
7	4.79	2.84	9	Intransigence
10	4.28	2.82	10	System-performance
Overall	5.57	2.67

Though OPERATIONAL-DEFICIENCIES ranked highest based on number of citations in the survey Round-One (see Figure 1), they have moved to fifth place based on Round-Three weights. This may suggest that though OPERATIONAL-DEFICIENCIES were prominent in many respondents' consciousness during Round-One, subsequently when listed alongside other sub-issues in Round-Three, they were felt by respondents to be somewhat lower in importance than the earlier relative incidence of citations implied. Note that frequency of citation was known from the outset to be a much cruder indicator of issue importance than weights. Many may feel that something is an issue, while at the same time universally believing it to be a relatively lesser issue. Regardless, with a mean score of 5.62, OPERATIONAL-DEFICIENCIES are yet marginally above the scale mid-point (5.5), suggesting that these at a minimum, are perceived to be moderately important issues. Even SYSTEM-PERFORMANCE issues with a mean score of 4.28, more than a full point below the scale mid-point, and ranked last (10^th) based on weights, should not be overly discounted. These too are issues cited by multiple respondents in Round-One, and here in Round-Three scored as moderately important. KNOWLEDGE-MANAGEMENT major-issues have moved from second place based on citations to first place based on weights. SYSTEM-DEVELOPMENT, SUPPORT, and DATA-CONVERSION are ranked 2^nd through 4^th respectively based on weights.

The detailed mean scores and ranks of the 38 related sub-issues (Appendix A) and comparisons between the strategic and operational personnel within agencies are discussed in the following section.

---