Summary

Point-Counterpoint

A point-counterpoint approach is presented to discuss the pros and cons associated with various issues and their operational impacts. The reader should be somewhat familiar with both the CMMI effort and the previous models developed by the Software Engineering Institute.

The authors of this book do not make any claims as to these arguments. We believe the readers should be able to make up their own minds.

Issue: No tailoring guidance is given for tailoring the CMMI for small organizations.

Point: The previous CMM was often criticized as having been written by large DoD organizations for large DoD organizations. Projects within the DoD realm generally consist of many people devoted full time to one project, or many people devoted full time to one of several sub-projects. These projects run for years , and cost millions of dollars. This type of thinking is completely opposite to that of small organizations. One example of problems that small organizations currently have with the CMM is the number of "groups" suggested by the CMM to achieve Level 3. The number of groups suggested is 13. Even if a group ranges from one person, part-time , to several people full-time , if your entire organization only consists of 20 people maximum, 13 groups become quite an expenditure of resources. The CMMI, relying heavily on practices maintained by large organizations, will be even more difficult to implement in smaller ones. CMMI has been slow to catch on in the commercial community. CMMI tends to attract DoD/aerospace organizations. The only tailoring guidelines given are called "Discipline Amplifications," which reside in the margins of most Process Areas and consist only of one or two sentences. These are often not clear or detailed enough to follow. The tailoring guidelines suggested in Chapter 6 are similar to those found in the original CMM for Software; that is, the more tailoring done, the less likely an organization is to improve. This warning seems to address only the negative aspects of tailoring "out" a practice or process area. It appears as if tailoring is actively discouraged. Also, software shops are not necessarily interested in systems engineering it just may not apply to them. So how and when can this be tailored out?

Counterpoint: While it is true that more guidance is needed as to how to select those Process Areas and practices that are most relevant to a small organization, the CMMI still allows the organization to tailor its process improvement objectives to the organization's business goals. The point of the CMMI is to improve the processes of the organization, not to just worry about maintaining fidelity to the model for Maturity Level ratings.

Issue: CMMI is simply too big for small organizations to handle.

Point: The CMM was criticized for having too many Key Process Areas and too many Key Practices. Just doing a visual comparison of CMMI against the CMM, CMMI appears to be three times as big. Also, the CMM was criticized for not containing everything necessary to promote the development of effective, efficient systems. So, the CMMI has removed the term "key" from its Process Areas and Practices. It now seems as though the CMMI is trying to prescribe everything necessary to produce good systems. The CMM has over 300 Key Practices. Processes and supporting procedures needed to be written by the organization in order to describe how these practices were to be followed. This effort was seen by most organizations as a major part of the time it took for their software process improvement efforts. The CMMI contains more Practices, and most of these are not detailed enough to be understood in such a way as to promote consistent application across organizations. Writing procedures will be long and difficult. The time, resources, and costs associated with implementing the CMMI appear to have expanded exponentially, compared to the already major investment required by the CMM.

One response heard at an SEPG Conference was that an organization no longer had to write as many procedures. That response was based on the fact that the CMMI rarely uses the word "procedures," whereas the CMM did rely on that word. However, a close reading of the CMMI will reveal that most of what an organization used to call "procedures" is now included in the "plans" that are a major part of each Process Area. Without the detail included in documented procedures, it is very difficult to ensure that processes are being followed in a consistent manner across your organization. So, whether they are called "procedures" or "plans," an effective process improvement program still has plenty of documentation to prepare.

Counterpoint: The CMMI allows small organizations, as well as large ones, to realize the benefits of following a structured process. CMMI allows for tailoring of the process, and for aligning the process to the needs of the organization. However, this alignment requires more thought than was previously required with the CMM, as the CMM was directed specifically at software engineering. As stated previously, the SEI responded to the cries from the marketplace for more information regarding those areas where the CMM was lacking specifically systems engineering. The organization can also choose how much of the model it wishes to follow. For example, if an organization is not interested in applying the Systems Engineering guidelines cited in the model, that organization may select the continuous representation of the model, or only the Software Engineering aspects.

Issue: Return on Investment (ROI) from CMMI has not been validated , especially as it relates to small organizations.

Point: The Return on Investment (ROI) quoted by proponents refers to the ROI gained from the CMM, not for the ROI from the CMMI. With some organizations reporting that it takes three to five years to realize its ROI from following the CMM, how long will it take an organization to realize its ROI from following the CMMI? Small organizations do not have the deep pockets and overhead of the larger organizations. Small organizations must worry about meeting payroll every two weeks. Expected ROI from using a model is often the argument used to justify the large expenditures necessary to institute and maintain a process improvement effort. Yet no real studies are available (at time of print) that can help a small organization calculate ROI from using this model.

Counterpoint: As for citing statistics for ROI from usage of the CMMI, it is still too early to have enough data points gathered to be statistically accurate. However, the SEI is conducting pilots and gathering this information. As both models are predicated on applying common sense to the practice of developing systems, one can extrapolate the benefits from one model to the next .

Issue: CMMI emphasizes Systems Engineering over Software Engineering.

Point: Historically, the biggest seller of all the models was the CMM for Software. For those customers who bought into and applied the CMM for Software, Systems Engineering may not be part of their work, and simply may not apply. The biggest growth sector in the marketplace right now for technical work is not large, bureaucratically structured organizations, but small, software- oriented shops. Why make taxpayers pay for a model that is not universally needed?

Counterpoint: Systems Engineering is necessary, no matter how large or small the organization. Very few organizations develop just software there are always systems issues to be taken into consideration, platform and hardware requirements, as well as interfacing with other groups or individuals responsible for some part of the system being built. Good systems engineering practices flowing into and out of software engineering tasks can only improve the software engineering effort.

Issue: CMMI is too prescriptive for small organizations.

Point: The CMM was sold as being what to do, with the organization responsible for defining how to do the whats . The CMMI is structured so similarly to large, bureaucratically controlled efforts that there seems to be little room to maneuver. For example, published results of CMMI pilot assessments have reported that the assessors had difficulty not asking interviewees leading questions, because the model is basically black or white, yes or no, do you do this or not. As for examples in the Process Areas themselves , Risk Management seems to require a risk mitigation plan. Is this really necessary if you are a maintenance shop and your projects only last three weeks? Verification and Validation may be too rigorous for small teams . Can independent testing and configuration audits satisfy the tenets of these Process Areas instead?

Counterpoint: The CMMI is a balanced model, promoting not only Systems Engineering concepts, but also Software Engineering, IPPD, and Supplier Sourcing. The organization may choose which areas to focus on, as well as the degree of focus that fits their business objectives. The developers of the CMMI realized that small organizations, as well as those organizations not as mature as early adopters of the SEI process improvement approach, should not be penalized . Therefore, Level 2 in the Staged representation mostly mirrors the basic concepts expressed in the CMM itself at Level 2, that is, basic project management. Because going from Level 1 to Level 2 is generally the most difficult for most organizations, Level 2 remains basic. Small organizations may also choose to follow the Continuous representation, although these shops may need more time to implement the concepts and to understand the terminology (as most software shops used the Staged Model of the CMM for Software). With the Continuous representation, an organization may select which Process Areas best fit their needs and concentrate on those areas only. Organizations may also elect to focus on only one discipline. So, the CMMI is actually much more flexible than the original CMM.

Issue: Small organizations are run differently from large organizations and face different challenges.

Point: The primary business driver in small, high-tech companies is time-to-market. Decisions must be made quickly, and all relate to the bottom line. While CMMI promotes quality by elongating the process used to develop and deliver systems (because of preplanning and embedded checkpoint mechanisms), time-to-market does not seem to be considered . Ignoring time-to-market concerns is simply not practical in today's marketplace. Although the public decries poor-quality systems, it seems to prefer speed over functionality. And delivering products quickly is the life-blood of small organizations.

Counterpoint: This is the reason models such as the CMMI were written. What good does it do to deliver a system on time if it does not work? While small organizations may need to consider time-to-market questions, inevitably, if the organization cannot deliver good products, it will go out of business. The CMMI is written based on the best practices of highly mature organizations that have used the various models and methodologies, and have learned from them. The CMMI has leveraged this information, consolidated it, and presents it in a format that can be tailored to the needs of any organization. While several Process Areas within CMMI may prolong the development and delivery of systems, using the Process Areas of this model will result in better products delivered and better decision making by executives.

Issue: CMMI was written for already-mature organizations.

Point: Early, introductory material from the Staged Representation of an earlier version of the CMMI states that organizations currently rated at the higher maturity levels, or pursuing Malcolm Baldridge or ISO 9000 certification, should consider using the CMMI. These organizations are already working on, or have achieved, some notion of process improvement. But is it not true that most organizations are still functioning at the lower levels? Is this an elitist model?

Consider the following example. The Measurement and Analysis Process Area is now a stand-alone PA. Measurement and Analysis used to be a Common Feature in the previous CMM. As such, it served as an effective check-and-balance for the entire Key Process Area (KPA). If an organization had performed the activities preceding it, then that organization should have been able to measure those activities. Then, the measurements could be reported to Management and Software Quality Assurance (as discussed in the Verifying Implementation Key Practices). The effectiveness of the processes for a particular KPA, as well as any deficiencies, could then be determined by reviewing the metrics collected. In the CMMI, we now have Directing Implementation. This Common Feature has far fewer examples of practical metrics. While the Measurement and Analysis PA has useful information, much of this information seems to relate to higher maturity levels. If an organization is just beginning the road to process improvement, how much of this PA must be implemented? The reason this PA was pulled out of each Key Process Area and is now its own PA came as a result of input from higher-maturity organizations. They all stated that measurement is the key to successful improvement efforts. But is creating this as a stand-alone PA the answer? Hindsight is a wonderful thing. Most organizations at Levels 4 and 5 are large organizations with deep pockets. Do these organizations really believe that they can institute this PA in their organizations, as they were back when their organizations were not quite as sophisticated as they are today? Do they believe that small organizations can implement this PA as written? Comprehensive metrics programs for small organizations, while beneficial, are too difficult and expensive to implement at Level 2.

Counterpoint: Level 2 in the CMMI still remains somewhat comparable to what it was in the CMM for Software. This should allow organizations, no matter what their size , to readily achieve this level if they had already been assessed at Level 2 or above using the CMM. Measurement truly is the key to focused improvement. As stated, if you have done it, you should be able to measure it. Using the measurements is also key. It does no good to simply collect the numbers and report them the numbers must be used. At the lower levels and in less-mature organizations, the metrics were not always used. Most organizations only began focusing on metrics when they were attempting to reach Level 4; and not all organizations continued in process improvement once they achieved Level 3. They dropped out before really turning their attention to measurement. By separating out this PA, emphasis is placed on collecting, analyzing, reporting, and using the measurements. Once again, an organization can tailor its process improvement efforts and the usage of this PA, as related to business goals.

Issue: The CMMI is too vaguely written to be used in assessments.

Point: Organizations attempting to use the model for an assessment report greatly extended timeframes. Interpretation appears to be based on personal experience, not on careful reading of the model. Most assessment teams have difficulty interpreting the model and state that the model is too vaguely written. Therefore, they must rely on their own personal experience. Because individuals have different experiences, this reliance does not promote consistent assessment results across organizations. While this consistency has always been a problem with CMM-based assessments and evaluations because of their dependence on the individuals who make up the appraisal teams, it is magnified with this model. Assessment teams could go to the CMM and read its practices and subpractices for guidance. Users of the CMMI report that this guidance cannot be found. Also, assessment teams have been directed to assess three projects from the Software area and then three different projects from the Systems area. Where is the integration?

This is not just a problem for small organizations.

Counterpoint: These issues have been raised by early pilot assessments and have been addressed in the SCAMPI method. Early CMM-based assessments had significant interpretation issues, so this is nothing new.