Validation of OCWS Process

It was important to involve experienced project managers and executives in the validation effort. Invitation letters were sent to member companies of CII as well as non-CII companies that attended the 1999 CII Annual Conference. This group of companies represented a cross-section of the North American construction industry and provided a source for experienced project managers and executives. In addition to these individuals, participants in the case studies that were conducted prior to the Delphi study were also contacted. Out of the ninety individuals who were invited to participate, forty-seven participated in the study.

The design of the Delphi study was based on a series of discussions with experienced researchers and past research that used similar approaches for developing consensus (Arditi and Ferreira Martins 1996; Hartman and Baldwin 1995; Martorella 1991; Zinn 1998; Kastein et al. 1993; Raskin 1994). The design involved having each of the participants rate various aspects of the OCWS process over successive administrations or rounds of the study. The degree of agreement among the participants and the reliability of their ratings were computed at the end of each round of the study. The agreement and the reliability that were achieved at the end of the second round, and the fact that no new issues emerged at that point, suggested that a third round would not add any further value to the results.

Design of the Delphi Protocol

The first and foremost question to be addressed before commencing validation was what the participants needed to validate. This question was answered by developing research propositions, termed as themes, which described what the OCWS process was designed to accomplish. The themes were defined as follows:

• Theme I: The owner-contractor work relationship framework, illustrated in Figure 1, will enable project management to create optimal owner-contractor work relationships, given that all other conditions remain unchanged.

• Theme II: Steps in the OCWS process and the associated worksheets, illustrated in Figure 2, will assist owners and contractors in the creation of optimal work relationships, given that all other conditions remain unchanged.

• Theme III: Owner-contractor work relationships based on this process will contribute positively to capital project performance, given that all other conditions remain unchanged.

• Theme IV: The knowledge-base created by using this process, that is, the core/non-core classification of competencies and corresponding owner-contractor work relationships, will help owner companies to mitigate the effect of knowledge lost by way of retiring experts and the loss of experienced personnel to other companies, given that all other conditions remain unchanged.

• Theme V: This process will be useful for creating and sustaining a potentially successful strategic alliance between two firms, given that all other conditions remain unchanged.

A second question to be answered was how to conduct the validation. It was answered by constructing multiple statements that corroborated the proposition of each theme, and then having the participants rate each statement on a one-to seven-point Likert scale, where one represented the option strongly disagree and seven represented strongly agree.

Test-Retest Reliability of the Delphi Protocol

The response protocol for Delphi validation was tested for consistency of responses over time by conducting two administrations within a gap of three weeks to measure the test-retest reliability of the participant response protocol. Four individuals, including two graduate students and two faculty members at Texas A&M University participated in this exercise. Three of these individuals had more than five years of experience in the planning and development of capital projects. The fourth individual had no such experience. Each person's scores on the first administration of the test were related to the person's score on the second administration to provide a reliability coefficient. The Spearman rank correlation coefficient was used to measure the test-retest reliability and compare the measured values with estimated critical values at a 5 percent significance level (Tuckman 1999). The test-retest reliability measurement helped identify unreliable statements that had to be modified or deleted from the response protocol.

Delphi Validation of the OCWS Process

A key characteristic of the Delphi approach is the iterative feedback from participants in the study. At the time of commencing the Delphi study, all participants were informed that they would be required to provide input over a minimum of two and a maximum of three iterations. The upper limit of three iterations was set considering both time and effort required by the participants, and past research which suggested that two to three iterations were sufficient for achieving satisfactory results in a Delphi study (Zinn 1998; Martorella 1991). The AI and the ICC, explained earlier, were used to draw inferences regarding the outcome of each Delphi iteration and determine the need for a subsequent iteration.

Background of Participants

From the fifty-two professionals who had initially agreed to participate in the Delphi validation, forty-seven professionals representing thirty-two owner companies and two consulting companies from North America participated in the first round. The participants had an average of twenty-one years of experience in the development and execution of capital projects, with a standard deviation of eight years. The composition of the participants' expertise at various levels in the owner organization is shown in Figure 3. The terms executive management, project management, plant operations, and other were used, based on a universally recognized distinction between levels of involvement in capital project work.

Figure 3: Composition of Participants' Experience Level

The data on the participants' level of experience provided an important basis to split the participants into two subgroups; those working at the level of executive management, and those working closer to the level at which work is accomplished. Since individuals working at the latter level should be expected to have a substantially different perspective on using contract services (Gibson et al. 1998), splitting the data accordingly was considered very important. Moreover, the OCWS process has a two-level focus, one involving a project application and the other involving a corporate or capital program application. Depending on their level of experience, the participants could respond differently to questions pertaining to the process. The information about participants' level of experience was used to assess whether or not differences existed in the perceptions of these two groups regarding various aspects of the OCWS process.

Another aspect of the participant characteristics that was particularly important in this research was their familiarity with the CII OCWS process. As shown in Figure 4, slightly more than half the participants indicated that they had some prior knowledge of the OCWS process, published by the CII in 1997, under the title Owner-Contractor Work Structure Process Handbook (Construction Industry Institute 1997). This information was used to assess whether or not there were differences in the perceptions of those that were familiar with or had used the OCWS process, and those who had no prior knowledge of the process. This background was considered very important since it served to assess if there was any bias involved in the participants' responses due to their prior knowledge of the OCWS process.

Figure 4: Participants' Familiarity with the OCWS Process

Analysis of Round 1 Data

A summary of data obtained from the first administration of Delphi is shown in Table 1. Each theme included between three and eleven statements that were rated by each of the forty-seven participants. The AI and ICC were used to measure participant agreement and reliability of the data, respectively. The results showed general agreement among the participants for all the themes. However, a low ICC for Themes I and IV and a negative ICC for Theme V indicated that the results lacked consistency between the participants.

Table 1: Summary of Round 1 Results

Theme Number	Mean	Standard Deviation	Agreement Index	ICC	P Value for ICC
I	5.8	0.93	0.95	0.46	0.12
II	5.6	1.02	0.97	0.88	≅ 0
III	6.0	0.84	0.95	0.96	≅ 0
IV	5.7	0.97	0.94	0.47	0.11
V	5.4	1.12	0.87	−0.17	0.43

The descriptive statistics suggested moderate to general agreement based on the Likert scale. The mean ratings for all themes were between five and six. The variability in terms of the standard deviation of participants' responses ranged from 0.84 to 1.12. The standard deviation for individual statements under each theme helped in identifying the statements that showed a relatively high degree of variance among the participants.

The AI ranged from 0.87 to 0.97, clearly suggesting that there was substantial agreement among the participants on the five themes. For the Themes I, IV, and V, the ICC values were unsatisfactory given a level of significance of 0.05. Also, since this was the participants' first opportunity to evaluate the OCWS process, it was possible that they had different perspectives with regard to the process and what it could accomplish. To help resolve the differences in the participants' perspectives to some extent, and to provide them with an opportunity to reconsider their first round ratings in light of the knowledge they could gain from the ratings and comments made by other participants in Round 1, another round of Delphi was used.

The second round was conducted using the same response protocol in Round 1, except the addition of one statement under Theme II. The addition of the statement was based on comments made by some participants that indicated that the strategic basis for core/non-core competency decisions had to be emphasized. In addition to the response protocol, each participant was provided with an assessment of the findings of Round 1 and a comparison of his responses with the group response.

Analysis of Round 2 Data

Some attrition of the participant pool is expected in a Delphi study, due to the limited time frame in which the participants are expected to read and understand Round 1 results and complete the Round 2 response protocol. In this study, the number of participants in Round 2 dropped to forty-two.

Round 2 of the Delphi validation showed an improvement in the rating of various elements of the process by the Delphi panel. The results of Round 2 are summarized in Table 2. These results suggested that the participants in the Delphi study changed their earlier response in Round 1 to some extent, based on the knowledge they gained from Round 1 results. The mean ratings did, however, indicate that on a one to seven scale, the mean was above 6.0 only in the case of Theme III. On all other themes the mean ratings were between five and six, suggesting that the agreement was conditional. This was not a cause of concern though, since each user organization is very likely to adapt the OCWS process to fit within its own business approach.

Table 2: Summary of Delphi Round 2 Results

Theme Number	Mean	Standard Deviation	Agreement Index	ICC	P Value for ICC
I	5.82	0.85	0.96	0.49	0.10
II	5.65	0.96	0.98	0.86	≅ 0
III	6.07	0.76	0.96	0.96	≅ 0
IV	5.69	0.80	0.96	0.05	0.38
V	5.46	1.08	0.88	−0.18	0.43

The mean ratings for all themes increased marginally, while the standard deviations decreased in Round 2. These data suggest that the panelists rated the process more favorably in the second round, while moving closer to each other on the rating scale. The ICCs improved in Round 2 over the values in Round 1, with the exception of Theme IV. This theme dealt with the value and the utility of the information created by using the OCWS process for future projects. On this theme, some participants expressed their skepticism about the value of such information for future projects since, in their opinion, it constitutes an initial plan only. Moreover, this knowledge may not be useful since staffing and training decisions are driven by factors that may change over time. This realization may have persuaded the participants to change their response in the second round, leading to an increased inconsistency in responses to Theme IV.

The reliability of the results of Round 2, measured in terms of the ICC, improved in comparison with that of Round 1 for Themes I and V, stayed the same for Theme III, reduced marginally for Theme II, and reduced substantially for Theme IV. Theme II focuses on the steps in the OCWS process and the associated tools. Theme III focuses on the contribution that using the OCWS process makes to improving capital project performance. Although both agreement and reliability were satisfactory for Themes II and III, the reliability indicated by the ICC for Themes I, IV, and V was not satisfactory. Theme I focused on the owner-contractor work relationship framework, Theme IV considered the utility of knowledge base created by using the OCWS process, and Theme V dealt with the applicability of the OCWS process for creating alliances. The low reliability for Themes I, IV, and V indicated that the participants remained inconsistent in their responses to these themes despite having two opportunities to make their decision.

Additional analysis of Themes I, IV, and V concentrated on participants' familiarity with the OCWS process prior to this study, and the extent of their executive management experience. In order to assess if the participants' familiarity with the process influenced their ratings, the participants were segregated by the criteria of familiarity with the OCWS process. A statistical comparison of the means for each of these two subgroups, presented in Table 3, indicated that the means were statistically not different, suggesting that participants' familiarity with the OCWS process did not influence their rating of the process. This was an important finding since participant familiarity with the process could have biased the results.

Table 3: Statistical Comparison of Means

Theme Number	Familiar with or Used the OCWS Process			Not Familiar with the OCWS Process			D.O.F.	t Statistic (t0.025 = 1.645)
	N	μ	σ2	N	μ	σ2
I	140	5.86	0.58	70	5.74	0.92	114	0.92 [**]
II	336	5.65	1.02	168	5.66	0.68	400	0.11 [**]
III	84	6.02	0.75	42	6.17	0.29	118	1.14 [**]
IV	140	5.71	0.67	70	5.64	0.64	141	0.54 [**]
V	84	5.55	1.26	42	5.29	0.89	96	1.38 [**]
* N = (Number of Statements under the Theme) * (Number of Participants)
[**]H0 Cannot be Rejected, where H0: Means Equal and H1: Means Not Equal

The possible influence of the participants' level of management experience was evaluated by stratifying the participants into two subgroups. The first group included twenty-two participants with more than two years of executive management experience. The second group included twenty participants with no executive management experience. These individuals were primarily involved in engineering, project management, and plant operations. Individuals in both groups did, however, have an average of ten years of project management experience. Table 4 provides a statistical comparison of mean ratings of the two subgroups.

Table 4: Statistical Comparison of Means

Theme Number	With Executive Management Experience			Without Executive Management Experience			D.O.F.	t Statistic (t0.025 = 1.645)
	N	μ	σ2	N	μ	σ2
I	110	5.83	1.06	100	5.82	0.29	168	0.06
II	264	5.70	0.84	240	5.60	0.98	488	1.13
III	66	6.09	0.67	60	6.05	0.52	124	0.30
IV	110	5.82	0.63	100	5.54	0.65	205	2.52 [**]
V	66	5.62	1.41	60	5.28	0.82	120	1.81 [**]
* N = (Number of Statements under the Theme) * (Number of Participants)
[**]H0 Cannot be Rejected, where H0: Means Equal and H1: Means Not Equal

Since the t-statistic for Themes IV and V was higher than the t-statistic at a significance level of 0.05 for a two-tailed analysis, the means were significantly different between the two subgroups, for these themes. This result suggests that the level of participants' experience had some influence on their ratings for these themes. A likely explanation for this difference was that individuals without executive management experience were relatively more concerned about the influence of project circumstances and conditions on decisions about capital projects. Since differences in mean ratings for the two subgroups were found to be statistically significant as shown in Table 4, further analysis was deemed necessary to test the reliability of responses within each subgroup for Themes I, IV, and V. The results of this analysis are shown in Table 5.

Table 5: Results of Delphi Round 2—Analysis by Subgroup for Themes I, IV, and V

Theme Number	Subgroup	Agreement Index	ICC	p-Value
I	With Executive Management Experience	0.93	0.62	0.04
	Without Executive Management Experience	0.98	−0.39	0.58
IV	With Executive Management Experience	0.96	0.45	0.13
	Without Executive Management Experience	0.96	0.04	0.39
V	With Executive Management Experience	0.85	0.64	0.07
	Without Executive Management Experience	0.93	0.74	0.03

The ICCs indicated that the reliability was reasonably high for the first subgroup in Theme I, and for both subgroups in Theme V. The p-values were also satisfactory for the first subgroup under Theme I and the second subgroup under Theme V, suggesting that the probability (on repeated trials) of obtaining ICC values at least as high as 0.62 and 0.74, respectively, was higher than the significance level of 95 percent. This probability was 93 percent for the first subgroup under Theme V, which was marginally lower than the significance level of 95 percent that is typically used in scientific research. The reliability coefficient for the first subgroup in Theme IV was fairly low, while the reliability coefficients for the second subgroup in Theme I and the second subgroup in Theme IV were very low.

Based on these computations, the data suggested that the executive managers were consistent in their response to Theme I, while those without any executive management experience were not consistent. One possible explanation for this finding relates to the broader perspective of executive managers, placing them in a better position to understand the conceptual framework of the OCWS. In the case of Theme IV, the two subgroups, as well as the whole group are not consistent in their responses. This finding is indicative of an inherent flaw in the proposition of Theme IV; that the knowledge base created by using the OCWS process will help owner companies mitigate the effect of knowledge lost by way of retiring experts and the loss of experienced personnel to other companies.

Each of the two subgroups provided consistent responses to Theme V. However, the responses were not at all consistent when the entire participant pool was evaluated together. One possible explanation for this inconsistency could be that the two groups viewed alliances differently, given that the term "alliance" was not defined in the response protocols. The participants' comments did not offer other explanations for the inconsistent overall response to Theme V.

In summary, the data provided evidence to suggest that based on reliability considerations, Theme IV could not be considered valid. Also, Themes I and IV evoked a different response from those with executive management experience and those without such experience.

Results of the Delphi Study

The Delphi rounds suggested that some propositions or themes about the applicability and usefulness of the process were valid, while others were not entirely valid. Although this observation is based on the assumption that the statements under each theme were accurate measures for the respective themes, no attempt was made to introduce new statements for two reasons: 1) Comments made by the participants in support of their numeric ratings did not suggest the need for introducing new statements; and 2) The AIs were reasonably high for all themes, including the ones with low or negative ICCs. The following results provided the evidence regarding validity of the five themes:

• The mean ratings for each theme ranged from 5.46 to 6.07, suggesting that the participants were in moderate to complete agreement with the themes.

• AIs for the five themes were between 0.96 and 0.98, with the exception of Theme V, which had an agreement index of 0.88, which is also very high.

• The ICC was 0.86 and 0.96 for Themes II and III, respectively. Although the ICC was low and unacceptable for Themes I, IV, and V, additional analyses identified satisfactory ICCs for one subgroup under Theme I and both subgroups under Theme V. Theme IV, however, failed the reliability criteria.

• No new issues that would lead to changes in the OCWS process steps or tools emerged in the Delphi rounds. This suggested that the participants did not envisage further changes to the OCWS process.

These results eliminated the need for conducting another Delphi round or for making any changes in the OCWS process steps and tools. Therefore, it was concluded that with the exception of Theme IV, the OCWS process constituted a valid approach for creating and sustaining well-aligned owner-contractor work relationships.