TIPS Pilot Trial Findings and Discussion

Usefulness of the Action Research Approach in the TIPS Pilot Trial

During the first week of the pilot trial, participants were provided with a protected copy of the Excel spreadsheet template and the senior author introduced TIPS philosophy at a one-hour seminar on company premises. Project managers A and C used monthly time-blocks (project duration six months to two years) and B, X, Y, and Z used weekly time-blocks for their shorter projects as suggested. The Friday visits to project managers were particularly useful in encouraging regular project reviews. A copy of each project's updated TIPS file was made and retained by project managers at the end of each week, primarily for trial record and review.

Regular interaction during project reviews with project managers using TIPS was valuable because they confronted the senior author with problems, conflicts, frustrations, and insights first hand. The immediacy of the interaction also helped motivate four software upgrades during the trial (8 August, 7 September, 20 September, and 29 September). General issues identified at visits could be aired at subsequent visits and with other project managers, and were often discussed or resolved through the email updates. The action research approach therefore stimulated both progressive learning and TIPS system development.

TIPS Implementation During the Pilot Trial

While all project managers were well-intentioned, external commitments, work priorities, project complexity, the developmental status of TIPS, and the ready availability of data strongly influenced the rate at which the TIPS project sheet was initially completed. For example, at the first weekly meeting project manager Y was able to enter all data into all fields of the ten-week upgrade project within one hour. The two participants on project C (a two-year "blue-skies" research project) took the longest. They were unable to complete data entry until the last week of the trial when the data fields and TIPS software functionality had been developed to accommodate their reasonable requirements. They were further delayed by the time taken to extricate two auxiliary projects, initially co-scheduled for the purposes of arranging continuity of work for the research group staff.

The original intention was to send TIPS project and portfolio files to project portfolio managers G and H at the end of each time-block for progress reporting and to assign the updated project priority rank. However, the practice for most projects was that submission did not begin until the end of the trial period. Therefore, TIPS interaction between these management levels was not adequately tested.

TIPS Information Development

The final format for the pilot trial TIPS worksheets is shown in Figures 4 and 5. Calculating and expressing project risk was clarified by defining three types of probability:

1. The probability of ever achieving a block goal.

2. The probability of achieving a block goal on time (using optimistic and pessimistic elapsed time estimates for each block goal).

3. The probability of commercial success (incorporated into the dollar value of project "net income").

The ratings for overall "usefulness" of the project management information fields were assessed and counted at values between one and five, with 2.0 or less being "not useful at all" or worse, and equal to or above 4.0 being "extremely useful" (Figure 7).

Figure 7: Usefulness of Twenty-Seven Fields of TIPS Project Information

Overall TIPS data fields were rated at an average of 3.4 or more than merely "useful." Company R1, which had a less structured existing innovation project management process, found more fields more useful than E1 (nine compared to one of the twenty-seven fields rating four or higher). Portfolio managers and business managers rated information fields as most useful (twenty-one and thirteen fields, respectively, rating four or higher). Technical managers were least impressed, rating four of the fields (priority rank, strategic category, project income, and company specified data) at two or less. Both company E1 and the project manager groups rated the priority rank and strategic category fields poorly. Technical managers thought that the field project income data was not useful, nor were company specified data fields (incorporated at the request of portfolio managers).

In a fully operational TIPS environment, it is envisaged that priority ranking will help empower staff to self-select day-to-day work and will strongly influence project funding and resource levels. The fields currently ranked poorly by the technical managers will consequently attract increasing attention.

TIPS Software Format

The usefulness of the TIPS format was surveyed and presented in Figure 8.

Figure 8: Usefulness of Fifteen Aspects of the TIPS Format.

The overall rating of 3.3 indicates that the TIPS format is useful. Company R1 (mean 3.8) and the project managers (mean 3.4) were the most positive.

Company R1 and the technical managers group considered that familiarity with Excel across functions and levels was particularly useful (rating 4.0 and 3.6). Six of the nine respondents verified that Excel was useful as the platform for TIPS, whereas the other three had reservations especially about tidiness, flexibility, and robustness at the user interface.

Specifying achievements instead of activities and tying these into the calendar on a weekly, monthly, and quarterly basis was rated positively (3.4 and 3.2, respectively) with little variation across groups (range: 2.5–3.6). The Excel cell, comments feature, was found to be a useful (if cumbersome) facility for expanding descriptions, listing tasks, and allocating jobs.

Company R1 considered the probability of block goal achievement and the consistent approach to estimating schedule achievement as very useful. This was in sharp contrast to the views of E1 who rated both these features very poorly. Similarly, the presentation and transfer of project information and the convenience of updating TIPS was rated at 4.2 to 4.8 for R1, compared with only between 2.5 and 3 by E1. This may reflect the multi-site operation for R1 compared to the single-site operation of E1.

TIPS Project Information Flow

Useful project management information must be identified and then acted upon to effect good monitoring and control. Overall, respondents slightly agreed that TIPS could be expected to increase useful project management information compared to existing arrangements (3.4 compared to 2.6, where one represented strong disagreement and five strong agreement).

TIPS identified project information more effectively and earlier than did existing arrangements in both companies (3.8 mean).

R1 thought that existing information exchange between R&D and marketing functions was unsatisfactory (1.8 mean) where it was acknowledged that no formal arrangements were in place. However, E1 managers agreed they had satisfactory arrangements (4.0 mean). Similarly, project information exchange across management levels was rated as poorly in R1, at 1.8, and neutrally in E1, at 3.0.

R1 considered that the amount of useful project information exchanged would increase when using TIPS (3.5), but E1 did not (2.8). However, when asked if TIPS was expected to improve useful communications between R&D and marketing, R1 expected no change, while E1 expected a mild improvement (3.3).

TIPS Implementation

A better user interface, perhaps using icons, was urged by three of the nine respondents, suggesting also that TIPS should be linked to a company database. Respondents suggested that staff education was needed for upper management "buy-in" of TIPS. An organizational culture change was also needed to accommodate the philosophical differences with existing arrangements, for example, to celebrate achievement rather than finish activities, and to accept that past expenditure is an investment in new knowledge. Awareness of the TIPS approach would be required for all staff, with project manager and higher levels needing specific training, both in project management principles and TIPS software use.

The estimate for the time to establish an operational TIPS system ranged from "as long as it takes for you to write a specific program" to five years. The broad median consensus was twelve months, which is considerably shorter than the three to seven years expected (Englund and Graham 1999).

The Rated Value of Implementing TIPS

Company R1 project managers and technical managers rated the potential value of implementing TIPS, or a similar approach, for monitoring technological innovation projects as high (4.0) (Figure 9). The mean rating of all respondents was moderate to high (3.6). Only one respondent gave a value at less than moderate (3.0). If dedicated software was used, this respondent's rating increased dramatically from one to five! One advantage of TIPS is that it can be introduced even if the project has started. Also, retrospective information is not required.

Figure 9: The Rated Value of Implementing TIPS (In Your Organization)

The mean for probability of TIPS implementation was moderate and ranged from 2.7 to 3.1, perhaps reflecting the developmental nature of the pilot trial and the lack of information about its performance elsewhere. R1 perceived the most value (4.0) from implementing TIPS, having no consistent innovation project approach and projects operating from dispersed sites. Project managers placed the same high value on TIPS, which was encouraging, because they had the greatest interaction during the pilot trial and were potentially most affected. Likewise, technical managers rated TIPS highly, the survey data indicating positively their expectations of improved communication of project information, particularly to higher levels of management.