SG 1 Evaluate Alternatives Decisions are based on an evaluation of alternatives using established criteria. Issues requiring a formal evaluation process may be identified during any phase of a product or project life cycle. The objective should be to identify issues as early as possible to maximize the time available to resolve them. SP 1.1-1 Establish Guidelines for Decision Analysis Establish and maintain guidelines to determine which issues are subject to a formal evaluation process. Not every decision is significant enough to require a formal evaluation process. The choice between the trivial and the truly important will be unclear without explicit guidance. Whether a decision is significant or not is dependent on the project and circumstances, and is determined by the established guidelines. Typical guidelines for determining when to require a formal evaluation process include the following: When a decision is directly related to topics assessed as being of medium or high risk When a decision is related to changing work products under configuration management When a decision would cause schedule delays over a certain percentage or specific amount of time When a decision affects the ability to achieve project objectives When the costs of the formal evaluation process are reasonable when compared to the decision's impact Refer to the Risk Management process area for more information about determining which issues are medium or high risk. | Examples of when to use a formal evaluation process include the following: On material procurement when 20 percent of the material parts constitute 80 percent of the total material costs On design-implementation decisions when technical performance failure may cause a catastrophic failure (e.g., safety of flight item) On decisions with the potential to significantly reduce design risk, engineering changes, cycle time, and production costs (e.g., to use lithography models to assess form and fit capability before releasing engineering drawings and production builds) |
Typical Work Products Guidelines for when to apply a formal evaluation process Subpractices Establish guidelines. Incorporate the use of the guidelines into the defined process where appropriate. Refer to the Integrated Project Management process area for more information about establishing the project's defined process. SP 1.2-1 Establish Evaluation Criteria Establish and maintain the criteria for evaluating alternatives, and the relative ranking of these criteria. The evaluation criteria provide the basis for evaluating alternative solutions. The criteria are ranked so that the highest ranked criteria exert the most influence on the evaluation. This process area is referenced by many other process areas in the model, and there are many contexts in which a formal evaluation process can be used. Therefore, in some situations you may find that criteria have already been defined as part of another process. This specific practice does not suggest that a second development of criteria be conducted. Document the evaluation criteria to minimize the possibility that decisions will be second-guessed, or that the reason for making the decision will be forgotten. Decisions based on criteria that are explicitly defined and established remove barriers to stakeholder buy-in. Typical Work Products Documented evaluation criteria Rankings of criteria importance Subpractices Define the criteria for evaluating alternative solutions. Criteria should be traceable to requirements, scenarios, business case assumptions, business objectives, or other documented sources. Types of criteria to consider include the following: Define the range and scale for ranking the evaluation criteria. Scales of relative importance for evaluation criteria can be established with non-numeric values or with formulas that relate the evaluation parameter to a numerical weight. Rank the criteria. The criteria are ranked according to the defined range and scale to reflect the needs, objectives, and priorities of the relevant stakeholders. Assess the criteria and their relative importance. Evolve the evaluation criteria to improve their validity. Document the rationale for the selection and rejection of evaluation criteria. Documentation of selection criteria and rationale may be needed to justify solutions or for future reference and use. SP 1.3-1 Identify Alternative Solutions Identify alternative solutions to address issues. A wider range of alternatives can surface by soliciting as many stakeholders as practical for input. Input from stakeholders with diverse skills and backgrounds can help teams identify and address assumptions, constraints, and biases. Brainstorming sessions may stimulate innovative alternatives through rapid interaction and feedback. Sufficient candidate solutions may not be furnished for analysis. As the analysis proceeds, other alternatives should be added to the list of potential candidate solutions. The generation and consideration of multiple alternatives early in a decision analysis and resolution process increases the likelihood that an acceptable decision will be made, and that consequences of the decision will be understood. Typical Work Products Identified alternatives Subpractices Perform a literature search. A literature search can uncover what others have done both inside and outside the organization. It may provide a deeper understanding of the problem, alternatives to consider, barriers to implementation, existing trade studies, and lessons learned from similar decisions. Identify alternatives for consideration in addition to those that may be provided with the issue. Evaluation criteria are an effective starting point for identifying alternatives. The evaluation criteria identify the priorities of the relevant stakeholders and the importance of technical challenges. Combining key attributes of existing alternatives can generate additional and sometimes stronger alternatives. Solicit alternatives from relevant stakeholders. Brainstorming sessions, interviews, and working groups can be used effectively to uncover alternatives. Document the proposed alternatives. SP 1.4-1 Select Evaluation Methods Select the evaluation methods. Methods for evaluating alternative solutions against established criteria can range from simulations to the use of probabilistic models and decision theory. These methods need to be carefully selected. The level of detail of a method should be commensurate with cost, schedule, performance, and risk impacts. While many problems may need only one evaluation method, some problems may require multiple methods. For instance, simulations may augment a trade study to determine which design alternative best meets a given criterion. Typical Work Products Selected evaluation methods Subpractices Select the methods based on the purpose for analyzing a decision and on the availability of the information used to support the method. For example, the methods used for evaluating a technical solution when requirements are weakly defined may be different from the methods used when the requirements are well defined. |
Typical evaluation methods include the following: Select evaluation methods based on their ability to focus on the issues at hand without being overly influenced by side issues. Results of simulations can be skewed by random activities in the solution that are not directly related to the issues at hand. Determine the measures needed to support the evaluation method. Consider the impact on cost, schedule, performance, and risks. SP 1.5-1 Evaluate Alternatives Evaluate alternative solutions using the established criteria and methods. Evaluating alternative solutions involves analysis, discussion, and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, or simulations may be needed to substantiate scoring and conclusions. Often, the relative importance of criteria is imprecise and the total effect on a solution is not apparent until after the analysis is performed. In cases where the resulting scores differ by relatively small amounts, the best selection among alternative solutions may not be clearcut. Challenges to criteria and assumptions should be encouraged. Typical Work Products Evaluation results Subpractices Evaluate the proposed alternative solutions using the established evaluation criteria and selected methods. Evaluate the assumptions related to the evaluation criteria and the evidence that supports the assumptions. Evaluate whether uncertainty in the values for alternative solutions affects the evaluation and address as appropriate. For instance, if the score can vary between two values, is the difference significant enough to make a difference in the final solution set? Does the variation in score represent a high risk? To address these concerns, simulations may be run, further studies may be performed, or evaluation criteria may be modified, among other things. Perform simulations, modeling, prototypes, and pilots as necessary to exercise the evaluation criteria, methods, and alternative solutions. Untested criteria, their relative importance, and supporting data or functions may cause the validity of solutions to be questioned. Criteria and their relative priorities and scales can be tested with trial runs against a set of alternatives. These trial runs of a select set of criteria allow for the evaluation of the cumulative impact of the criteria on a solution. If the trials reveal problems, different criteria or alternatives might be considered to avoid biases. Consider new alternative solutions, criteria, or methods if the proposed alternatives do not test well; repeat the evaluations until alternatives do test well. Document the results of the evaluation. Document the rationale for the addition of new alternatives or methods and changes to criteria, as well as the results of interim evaluations. SP 1.6-1 Select Solutions Select solutions from the alternatives based on the evaluation criteria. Selecting solutions involves weighing the results from the evaluation of alternatives. Risks associated with implementation of the solutions must be assessed. Typical Work Products Recommended solutions to address significant issues Subpractices Assess the risks associated with implementing the recommended solution. Refer to the Risk Management process area for more information about identifying and managing risks. Decisions must often be made with incomplete information. There can be substantial risk associated with the decision because of having incomplete information. When decisions must be made according to a specific schedule, time and resources may not be available for gathering complete information. Consequently, risky decisions made with incomplete information may require re-analysis later. Identified risks should be monitored. Document the results and rationale for the recommended solution. It is important to record both why a solution is selected and why another solution was rejected. |
