Research Methodology


The Center for Construction Industry Studies (CCIS), jointly funded by CII and the Alfred P. Sloan Foundation, performed follow-up research with the objective of testing, validating, and improving the OCWS process. Due to the subjectivity inherent in an assessment of owner-contractor work relationships, with the help of the OCWS process, the nature of this research was qualitative. Also, given the limited existing knowledge base on the determination of owner-contractor work structures, investigations in this area were exploratory in nature, making it difficult to construct hypotheses based on quantifiable parameters. Therefore, the research methodology relied on an owner survey, case studies, and the Delphi method (Arditi and Ferreira 1996; Linstone and Turoff 1975). Three research phases were conducted.

Phase I—Owner Survey

Phase I of the research covered a survey of owner companies in North America, the formulation of a research methodology, and the development and testing of instruments and proposals for the case studies that followed in Phase II. As part of Phase I of the research, owner companies were contacted in the summer of 1998 to study the use of the OCWS process developed by CII. The results of this survey established the premise for subsequent phases of this research (Anderson et al. 2000). Out of the total of sixty-two owner companies contacted, twenty-three responded to the survey. Of the twenty-three companies, only three had used the OCWS process and eleven had reviewed the process but not used it. The remaining nine owner companies had not reviewed the OCWS process even though they had acquired a copy of the publication from CII. The relatively low use of the OCWS process and concerns regarding practicality and ease of use, as expressed by respondents, established the premise to undertake more in-depth, exploratory research through case studies.

Phase II—Case Studies

The case studies in Phase II involved a combination of interviews and process applications for gathering data. To initiate the case studies, company-specific proposals were developed with the objective of addressing each company's particular problem through the implementation of the OCWS process. Two owner companies in North America accepted the proposal to implement the OCWS process. The OCWS process implementation in one company involved the development of a new proposed alliance with a large contractor. In this case, the owner was a large refining company. The second implementation effort involved the restructuring of the owner company's capital projects organization based on a business model approved by top management. In this case, the owner was a large power company. A third company agreed to participate in an interview to evaluate a recent application of the process for developing an OCWS for an overseas chemical plant project.

The conceptual framework of the OCWS process was explained to participants in the case studies with the help of an owner-contractor work relationship continuum, as shown in Figure 1. The case studies provided valuable insight into the OCWS approach for developing owner-contractor work structures, and identified several weak areas in the existing process (Anderson et al. 2000). As a result, the OCWS process was modified and improved upon. The modifications in the OCWS process provided the necessary elements to help owner companies accomplish the following four key objectives:

  1. Forming optimal work relationships with designers, constructors, suppliers, and consultants.

  2. "Rightsizing" in-house capital project development and execution capabilities.

  3. Addressing the problem of loss of expertise due to attrition of experienced personnel.

  4. Forming potentially successful strategic alliances.

The steps and associated worksheets in the improved OCWS process are illustrated in Figure 2. The worksheets provide a mechanism to document decisions and create a common language that can be used throughout the life cycle of capital projects.

click to expand
Figure 2: Steps in the OCWS Process and Process Worksheets

Phase III—Delphi Validation of OCWS Process

Validating the OCWS process posed a unique problem. Unlike most research problems that involve the study of organizational phenomenon based on some available data or surveys/interviews of personnel, validating the OCWS process involved having the research participants assess various aspects of the process. The participants' assessment of the process was based on their experiences with forming effective owner-contractor work relationships. Since the reliability of this approach was a function of how well the participants understood the process, it was inappropriate to conduct the validation in a single administration. A Delphi approach that involves multiple iterations of data collection was considered most suitable for testing the validity of the OCWS process that resulted from Phase II. The three key characteristics of a Delphi study are: 1) Anonymous response; 2) Iteration and controlled feedback; and 3) Statistical group response (Dalkey 1969).

Prior to beginning the Delphi validation the participants were first selected from a wide range of organizations, based on the extent of their experience in the area of capital project development and execution. Each participant was provided with an overview of the modified OCWS process, including Figure 2, and a structured response protocol. Ordinal Likert scales were used for soliciting responses from the participants, by having each of them rate various aspects of the OCWS process on a one to seven, strongly disagree to strongly agree scale. The responses were based on the participants' knowledge of capital project development and execution, and the relevance of the process described in the overview. Reliability and validity issues were addressed by computing appropriate indices.

Two types of reliability were critical in the study. The first, known as test-retest reliability, measured the consistency of results on repeated trials. In this procedure, each person's scores on the first administration of the test are related to his scores on a second administration, to provide a reliability coefficient (Tuckman 1999). The test-retest reliability was measured on a pilot basis prior to conducting the Delphi process. The second type of reliability measure that was relevant to this study was internal or inter-item consistency. The internal consistency was measured after each Delphi round.

Internal consistency was one of the two separate indicators that were used to draw inferences regarding the outcome of each Delphi round and determine the need for a subsequent round. It was measured by computing the intraclass correlation coefficient (ICC), which is a function of the true score variance and the error variance based on an analysis of variance (McGraw and Wong 1996; Nunnally and Bernstein 1994; Kastein et al. 1993; Kozlowski and Hattrup 1992; Bravo and Potvin 1991; Shrout and Fleiss 1979). The ICC measures the degree of consistency for measurements that are averages of the k independent measurements on randomly selected participants. In this research, the ICC was based on a two-way random effects analysis of variance, which is mathematically the same as Cronbach's alpha in psychometrics (McGraw and Wong 1996; Cohen and Swerdlik 1999).

The other indicator that was used to draw inferences regarding the outcome of each Delphi round was an Agreement Index (AI) (James, Demaree, and Wolf 1984, 1993; Kozlowski and Hattrup 1992). The AI is a function of the ratio of observed or sample variance and the random measurement variance, based on a uniform distribution. Since high inter-observer agreement is not sufficient to insure the quality of the data collected, evidence of the reliability of the data was also necessary (Mitchell 1979). Therefore, both the ICC and the AI were used to evaluate the data. It is important to note that the two measures are not related mathematically or otherwise.

The Delphi method served as an approach for testing the validity of the OCWS process. The ICC, used to estimate internal consistency, also provided evidence of construct validity (Cohen and Swerdlik 1999). The multi-round process of Delphi enabled the identification of problems related to judgments of the participants, as well as possible biases involved in their opinions (Arditi and Ferreira Martins 1996). The Delphi rounds were also very effective in resolving differences among the participants and addressing various issues that surfaced through participants' comments during each administration.




The Frontiers of Project Management Research
The Frontiers of Project Management Research
ISBN: 1880410745
EAN: 2147483647
Year: 2002
Pages: 207

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net