Research Methodology
|
As outlined previously, both the research-based literature on project management competence and the standards that have been developed to define aspects of project management competence have been developed primarily from the collective opinion of project management practitioners and others as to what project personnel need to know and what they need to be able to do in order to be considered competent.
The assumption behind the development and use of project management standards is that the standards describe the requirement for effective performance of project management in the workplace and that those who meet the standards will therefore perform, or be perceived to perform, more effectively than those whose performance does not satisfy the standards. To date, no research has been conducted to validate this assumed positive relationship between project management competence as described in the literature and assessed against standards and perceptions of effective performance in the workplace.
Using two recognized project management standards as described previously, A Guide to the Project Management Body of Knowledge (Project Management Institute 2000) and the Australian National Competency Standards for Project Management (Australian Institute for Project Management 1996), data was collected to explore the relationship between performance against project management standards and perceived performance in the workplace.
Five instruments were used in data collection. One instrument was used to gather general demographic information about respondents and their project management role. Two instruments were used to collect information on project management knowledge and practices of participants:
-
A knowledge test, using multiple-choice questions drawn from sample questions for the Project Management Professional (PMP ) certification examinations, with five questions for each of the nine PMBOK Guide knowledge areas.
-
A self-assessment against the nine units of the Australian National Competency Standards for Project Management with responses against a five-point scale from 1 to 5 where:
Two instruments were used to gather information on perceived effectiveness of project management performance:
-
A self-rating questionnaire
-
A supervisor-rating questionnaire.
Based on review of the project success criteria literature, the self- and supervisor-rating questionnaires addressed the issue of perceived success according to differing stakeholder perspectives, seeking ratings of the participating project personnel on a 1 (low) to 5 (high) scale of perceived value to clients, value to their organization, effectiveness of relationships in achieving project goals, and ability to inspire and encourage the performance of others.
A deficiency of the perceived performance rating instruments was that information was only sought from the participant and his supervisor (or suitable equivalent). Ideally feedback would also have been sought from other stakeholders, such as clients, but this was beyond the achievable scope of the study. Another problem encountered was in securing completed rating forms from supervisors. Subsequently, although the supervisor rating appears to be a more reliable indicator (Cronbach's alpha 0.8650) than the self-rating (Cronbach's alpha 0.6214), responses are not available for all participants. Difficulties in obtaining supervisor ratings highlighted an interesting dimension of the project manager role. In a number of cases, organizations claimed that there was no one in a position to rate the performance of the participating project personnel.
It must also be noted that the standards do not directly address two important factors identified in the literature review, namely leadership, drawing primarily on personality characteristics, and technical performance, which tends to be application area specific and is therefore not directly addressed in generic standards.
The sample for the study was obtained by asking organizations in Australia, the United Kingdom, and the United States to identify between five and twenty of their project personnel to participate in the study. Data collection was conducted on the premises of participating organizations.
Reluctance on the part of respondents to provide either age or gender, or both, reduced the value of these variables for analysis. However, based on those who did respond, the demographics of the study resemble those of PMI membership (Project Management Institute 1999b) in terms of both age (Table 4) and gender (Table 5).
| Age Category | Study Group Demographics | PMI Member Demographics |
|---|---|---|
| 31–40 Years Old | 31% | 30% |
| 41–50 Years Old | 43% | 40% |
| All Other Ages | 26% | 30% |
| 100% | 100% |
| Gender | Research Sample | PMI Member Demographics |
|---|---|---|
| Male | 74% | 75% |
| Female | 26% | 25% |
| 100% | 100% |
The industry sector of organization and regional location of the study group is shown in Table 6. As the research was funded by an Australian Research Council grant for conducting it in Australia, the Australian sample is the largest and best distributed. Engineering and construction organizations in the United States were reluctant to participate in the study, largely due to the time commitment required.
| Industry Sector of Organization | Australia | USA | UK | Total |
|---|---|---|---|---|
| IS/IT & Telecommunications | 59 | 39 | 19 | 117 |
| E & C | 104 | 50 | 154 | |
| Business Services | 46 | 28 | 7 | 81 |
| Total | 209 | 67 | 76 | 352 |
Self and Supervisor Ratings—Perceived Performance
Competence is a socially constructed concept (Burgoyne 1993) and studies which have endeavored to identify high performance competencies (Boyatzis 1982; Schroder 1989; Gadeken and Cullen 1990; Cockerill 1989) have encountered difficulties in identifying the best or most effective performers.
As a measure of job performance, Boyatzis (1982), in his study of general managers, suggests three types of performance or criterion measure:
-
Supervisory nominations or ratings
-
Peer nominations or ratings
-
Work-output measures.
Supervisory ratings are, of course, subjective, and it is recognized that subjective measures are excessively prone to contamination, especially by supervisor bias (Campbell 1990). However, Nathan and Alexander (1988) conclude that objective measures are not more predictive than subjective measures. The most important issues appear to be awareness of the potential construct validity threats of any measure used (Bommer et al. 1995) and recognition that performance is not a single construct (Campbell 1990).
In the field of project management competence and effectiveness, some studies have used a supervisor's subjective rating of the degree of effectiveness of the project participant under examination. Thamhain and Wilemon (1977) asked superiors to rate project managers relative to their peers on overall project performance on a 0-100 percent scale.
Gadeken (1991) having experienced the problem on a previous occasion in his studies of US defense systems procurement program managers (Gadeken and Cullen 1990) has explained the dilemma of identifying project management effectiveness:
The first and most difficult step in the job competency assessment process is to identify truly outstanding performers to study. For project managers, this is problematic because there are no clearly objective performance measures that can be applied. Overall assessment of project success and hence project manager success is difficult because of the complexity and extended time duration of most projects. Also many projects are significantly affected by external funding and political factors. Since projects usually involve several project managers over their duration, the current project manager may benefit or suffer from the efforts of his predecessors. Consequently, the only reasonable and acceptable approach for this study was to ask for nominations from senior officials (Gadeken 1991, 7).
For the purposes of this study, scales were constructed to distinguish top from lower performers by taking the total scores for the self- (n = 347) and supervisor-(n = 208) rating instruments, and identifying those above median score as top performers and those below the median score as lower performers. Those cases with scores on the median were removed to ensure a clear split between top and lower performers, leaving a sample size of n = 278 for self-ratings and n = 176 for supervisor ratings.
It is important to note that due to the method of selecting the sample, all participants in this study should be considered to be relatively effective performers. All participants were, at the time of the study, employed in project roles by organizations that recognized the value of project management sufficiently to support the study and valued the participants sufficiently to support the time that they spent in providing data for the study. Therefore, any differentiation in terms of performance must recognize that such performance is relative only. Those who are identified as lower performers are still gainfully employed in project roles.
Using this scoring, and relating self-ratings and supervisor ratings, 31 percent of the sample were rated as having low perceived performance by both themselves and their supervisors. Twenty-seven percent rated their performance as high, and this was supported by supervisor ratings. Twenty-four percent of the sample rated themselves high but their performance was considered low by supervisors, and 19 percent of the sample was rated more highly by their supervisors than they rated themselves.
Project Management Knowledge as a Predictor of Perceived Effective Performance
Analysis of the previous results (independent samples T-Test) on the multiple choice knowledge test using the nine knowledge areas identified in the PMBOK Guide and related to perceived performance as outlined above, provided no direct evidence that more knowledge in these areas lead to greater perceived effectiveness of performance. However, investigation using logistic regression methods to explore the relationships between variables, revealed that increasing levels of project management knowledge appear to have a positive rather than a negative effect upon supervisor perceptions, with the possible exception of quality knowledge although this is at a lower level of significance.
Project Management Practices as a Predictor of Perceived Effective Performance
No direct relationship between the use of practices and supervisor perceptions of performance was identified (independent samples T-Test). However, exploratory analysis suggested complex relationships between the use of practices and supervisor perceptions of performance with some practices (e.g., contract finalization, time management) enhancing the odds of being perceived as a top performer while other practices (e.g., communication activities, stakeholder management) were associated with decreasing odds.
Two levels of analysis were applied to the results collected in the self-assessment against the Australian National Competency Standards for Project Management.
-
Summary scores were calculated for each of the nine units in the Australian National Competency Standards, identifying patterns of use of the practices and then exploring their predictive value relative to perceived effective performance.
-
Summary scores were calculated for literature-derived constructs.
Analysis at Unit Level
Each of the nine units in the Australian National Competency Standards for Project Management is made up of a number of elements and performance criteria—integration (11), scope (8), time (9), cost (9), quality (11), human resource management (12), communications (11), risk (9), and procurement (14). Data was collected by asking each respondent to indicate, on a 5-point scale (Table 7), their level of experience in each of the ninety-four performance criteria.
 |
| 1 = I have never done or participated in doing this. |
|
The nine Units have good reliability with Cronbach's alpha ranging from 0.8384 for integration to 0.9494 for procurement. From investigation of patterns of use of practices, the spread for use of integration, scope, time, cost, and communications practices is fairly evenly distributed across the sample.
Quality, human resource management, and risk practices are not evenly distributed, with fewer people using practices in these units, than those who do not. Procurement is particularly interesting. There are some who use procurement practices extensively, some who don't appear to become involved in procurement practices at all and others in the middle range.
Quality is the only one of the nine items for which there is an apparent distinction between the perceived top and lower performers (Figure 1). Those perceived as less effective performers have slightly more experience in use of quality practices. Quality results also provide an opportunity to demonstrate an overall tendency for the supervisor scores to be different between countries. The distinction is particularly marked between the UK and the rest, interpretable as a cultural difference where in the UK being understated and "objective" is valued more highly.
Figure 1: Quality Scores by Region
The use of communications practices, associated in logistic regression models with decreasing odds of being perceived as a top performer, provides a further example of cultural differences on perception of performance. Analysis (Figure 2) reveals that those perceived as lower performers have slightly more experience in the use of communications practices for both the Australian and US samples, while the reverse is true in the UK where those perceived as top performers have higher levels of experience in the use of communications practices.
Figure 2: Communication Scores by Region
Analysis Based on Literature-Derived Constructs
Each of the ninety-four performance criteria against which data was collected were grouped according to the literature-derived constructs presented in Table 2. A reliability analysis was conducted, identifying those constructs that had a reasonably high value (above 0.80) for Cronbach's alpha. Only a few of the more interesting constructs will be discussed here.
Planning (Integrative) (alpha = 0.8319)
Use of this group of practices, consistently considered in the first rank of importance according to the literature analysis (Table 2), shows that there is a tendency for all self-assessed, high-performing individuals to have a higher rating on the planning (integrative) scale. However, this pattern is not supported by supervisor perceptions. In logistic regression models based on supervisor ratings, integrative planning is associated with decreasing odds of being perceived as a top performer.
Monitoring & Controlling (Integrative) (alpha = 0.9005)
This variable represents a consistent construct, based on eleven items. This construct, identified in the first rank of importance according in post-1995 research-based literature for both success factors (Table 1) and project manager competence (Table 2), was also identified in logistic regression analysis as a variable associated with increasing odds of being perceived as a top performer. Examining the supervisor ratings, this tendency is apparent (Figure 3) for both the Australian and UK, but not for the US sample, where those identified as top performers tend to have lower scores for use of these practices.
Figure 3: Monitoring & Controlling (Integrative)—Supervisor Ratings
This construct also provides a useful illustration of the tendency for self-ratings to be less affected by cultural differences than supervisor ratings, with higher scores for use of monitoring and controlling (integrative) practices being more strongly and consistently associated with self-assessed top performers (Figure 4).
Figure 4: Monitoring & Controlling (Integrative)—Self-Ratings
Monitoring & Controlling (Risk) (alpha = 0.9024)
This is a very reliable construct, even though it is only based on four items. However, the sample is divided between those who use risk management practices and those who don't. There is a tendency for higher scores on this item to be associated with higher self-rating of success. On the other hand, it does not appear to be associated with higher supervisor ratings (Figures 5 and 6).
Figure 5: Monitoring & Controlling (Risk)—Supervisor Ratings
Figure 6: Monitoring & Controlling (Risk)—Self-Ratings
Team Development (alpha = 0.8974)
There is a very small group scoring highly on this scale, and quite a large group scoring well below 2.5, indicating that there are many people who are not using these practices. This is interesting given the high ranking this construct received based on the literature. Team development is not identified in logistic regression analysis as a predictor of perceived performance.
Lessons Learned (alpha = 0.9351)
Despite considerable rhetoric concerning the importance of capturing, sharing, and utilizing lessons learned, this construct does not appear in the results of the literature review. This may be a result of focus, in research and literature, on management of single projects. More recently interest has been widened to consideration of organizational project management and multiproject contexts (Engwall and Kallqvist 1999; Turner and Keegan 2000) and further research in this area may justify inclusion of lessons learned as a required area of project management competence. As it stands, there are a number of performance criteria in the Australian National Competency Standards for Project Management that relate to lessons learned that could not be grouped with variables identified from the research-based literature. It is interesting however, that this is a very reliable construct, based on fourteen items, but that most people score low on this scale. In other words, most project management personnel in the sample do not use lessons learned practices. There is a slight tendency for those with higher scores to be perceived by supervisors in all three countries (Australia, US, and UK) as lower performers. Higher scores are associated with top performers in self-ratings.
|
EAN: 2147483647
Pages: 207