7.1 Evaluation and deployment of complex systems

---

The deployment process must be carefully planned, prepared, and executed. In large organizations, the deployment process can take a long time, perhaps even several years, and it may happen that the results of the deployment are not directly visible. Before the tool is deployed, it must be selected. Large investments require a thorough evaluation process in which the most suitable tool is selected, with the selection based on clearly identified requirements and expected results.

Very often, personnel working on deployment are focused on tool features and neglect other aspects of changes caused by the introduction of a new tool. The deployment of a tool involves much more than beginning to use it. The organization introducing a new complex tool must also consider the following facts:

• It may be necessary to change the company development, maintenance, and production procedures. New roles and activities might be introduced, relationships between procedures and data might be changed, the types of data to be processed might change, and it may even be necessary to replace certain other tools affected.

• There is often a general resistance to learning something new, which is correlated with an initial reduction in productivity and with psychological barriers when leaving a known area and entering the unknown. Some may also lose their positions as experts and be required to learn a complex new system from the beginning.

• Personal return on investment may vary. PDM and SCM are administrative systems introducing an overhead to many of the users. To many of these users, getting an increased administrative overhead is no problem because they can also see the benefits. However, to some, all extra work is only a burden without any gain in return. Because no chain is stronger than its weakest link, it is crucial for a successful deployment to motivate the entire organization. Only a few people/roles neglecting to use a new model/tool will greatly reduce its positive influence. Machines can be ordered to do whatever we want, while humans have to be motivated, engaged, and committed. Preferably all people involved should benefit from using a new model or tool themselves; every person should have a positive personal return on investment.

• Education and training will be necessary. To understand why the new and often extra procedures have to be performed is necessary— especially if there is a low personal return of investment. Knowing that the extra work put into the system actually is good or even necessary for the project as a whole can motivate a person to do it without personal gain. It is also important to minimize the extra work invested without decreasing the positive effects. Training—learning to use a new tool effectively—is therefore a good investment. In some cases it not only reduces the time it takes to perform a certain action, the action would not otherwise actually be performed at all.

• Inexperience in the use of new tools and misunderstanding their functions may lead to poorer performance or errors, which have a negative impact on productivity and the final result.

• Administration activities will increase during the transition period. When a complex new tool is introduced, it most often replaces an obsolete tool or some manual procedure. Replacement of an old tool with a new one requires concurrent execution of both tools during the transition period as well as dealing with undefined and even inconsistent states.

• Conversion of data and its import into the new system might be required. This conversion can be complex, and it can require the use of, or even development of, certain conversion tools.

• The maintenance of old products may become questionable. The conversion of all data and the maintenance process as whole can be very expensive, and in many cases its cost is unacceptable. An example of such a situation could be the use of a newSCMtool running on a certain computer platform, but with the product development and maintenance environments located on another platform, probably an old version, on which it is not possible to install the new SCM tool. An alternative is to retain the old environment for old products (this being a requirement in many cases, for example, for safety-critical systems) and to use new tools only for new products. This solution also might become very expensive due to the cost of maintenance of old hardware and software, and the availability of knowledge of the use of the old environment would probably decline.

• Complex tools cannot be used directly, but must frequently be adjusted to the processes in the organization. SCM and PDM tools are typical examples of tools by which the processes are implemented.

• Using a new complex tool with new functions requires continuous support. The more functionality it provides, the more administration support it will require. Introducing a new tool will not necessarily decrease the administration costs. It is more likely that the administration and support costs will remain the same, if not increase. The return on investment hoped for will instead be increased by the new functionality resulting in increased productivity, higher quality, and a shorter time to market.

• Introduction of a new tool may require changes in the organization on temporary and permanent bases. For example, a new support and training team may be required.

7.1.1 The organization of the evaluation and deployment process

As the introduction of a complex tool can require considerable effort and time, it is most appropriate is to perform the introduction as a project. Using the project form has many advantages: the process is planned as a standard project, resources are allocated, costs are calculated, time frames are determined, and the project goals are identified.

The entire process includes several phases:

1. Preparation phase includes the activity:

   • Identification of needs of a new tool.

2. Evaluation and tool selection phase includes the activities:

   • Planning of the deployment process;

   • Identification of requirements for the new tool;

   • Tool evaluation and selection.

3. Deployment phase include the activities:

   • Tool deployment;

   • Analysis of the deployment process.

4. Full use of new tool phase includes the activity:

   • Verification of the requirements in full use of the tool.

For this reason, it is reasonable to divide the process into several projects: an evaluation project and one or even several deployment projects.

The evaluation project should deal with requirements specifications for the tool and its selection. The deployment project (or several projects) will deal with the deployment of the selected tool in the organization. This phase can take a considerably longer time than the evaluation phase. Figure 7.1 shows the entire process divided in several phases and the deliverables.

Figure 7.1: The entire process divided in several phases with documentation delivered.

7.1.1.1 Identification of needs of a new tool

A need for a new tool need grows as the development and production processes become more complex. Examples of such situations are when a company developing hardware devices begins the development of software embedded in their products, the development moves from one place to several places, the demands for shorter time to market increase dramatically, new customers and new forms of placing the product on the market appear, or new technology is used in the products or in the product development. In many cases, this need is not recognized in the organization. Instead, temporary compensation for the need for a new tool may be provided by alternative measures, such as optimization in the process, a decrease of the profit margin, or an unawareness of the new tools needed. In due course, the need for a specific tool may be recognized. Selecting the wrong tool (a tool not needed or not appropriate for the particular processes in the organization) may be as unfortunate as not selecting a tool at all. For this reason, it is important to identify the main requirements of the procedures in the PLC. These requirements are the result of an analysis of the procedures and business goals. The analysis should identify any bottlenecks in the procedures, the main problems, and the possibilities for improvements. The analysis should also identify the main purpose of the new tool, the main benefits, and the expectation of the return on investment. Possible risks of unsuccessful deployment of the tool and the possible consequences should also be considered.

7.1.1.2 Planning of the deployment process

Deployment process planning is similar to planning of a development project. The analysis of the need for new tools is used as input to the project. A project plan document, similarly to a development project plan, should include the following elements:

• The project organization, the project members, and to whom the project will be reported. The project members may change, depending on the project phases. It is important that relevant representatives of potential users, administration, and support personnel are involved in the process. The management must be involved in the process to be able to make as correct a decision as possible.

• Project activities, time schedule, and milestones. The entire project may occupy a relatively long period of time and the possibility of dividing the entire project into several projects should be considered.

• Deliverables. The deliverables from the first part of the project are a requirements specification for the tool to be selected, and an analysis report about the tools proposed for the selection. This report should include a recommendation for a tool, the bases for the recommendation, and an overview of possible scenarios of the tool deployment. The result of the second phase, or later projects, should be the integration of the tools and their acquirement by the users.

• Costs. As any project plan, this project should list the project costs. The costs will include costs of the use of internal resources, possible costs of the evaluation licenses, consultant help, and training of the project participants. For the deployment phase, the costs will be significantly larger than for the tool selection phase.

• Risk analysis. The main risk in such a project is that the conclusion drawn from the tool evaluation is incorrect due to inadequate analysis. Typical risks could be a misunderstanding of the organization’s processes and a misinterpretation of the tool’s characteristics. There is also a risk if the tool’s behavior in a complex environment cannot be properly tested and evaluated. Such risks should be analyzed, and measures to reduce their effects should be taken.

7.1.1.3 Identification of requirements for the new tool

The requirements for the tools should be derived from the process requirements and the business goals of the organization. They can be classified as functional and nonfunctional requirements. The functional requirements describe the demands on the functionality of the tool. The nonfunctional requirements cover a large spectrum of features related to the tool and activities that cannot be expressed as functions but are important for a smooth and effective functionality.

Examples of nonfunctional features are performance, usability, portability, robustness, and reliability. Other types of features include the ability to maintain and improve the tool. Some of these features are modifiability, maintainability, modularity, and flexibility.

Moreover, requirements of the tool vendors are also important in evaluating their ability to ensure the quality of the products as required, to provide support and to maintain the product, to be able to continue with the tool development, and to have a recognized position on the market. In the long run, these requirements may be even more important than the functional requirements.

The requirements should be prioritized. It is certain that not all requirements will be met. The prioritization of the requirements indicates those that must be fulfilled unconditionally, those that are important, and those that are desirable.

7.1.1.4 Tool evaluation and selection

Selecting a tool is a multistep process that begins in parallel with or after the completion of the requirements specification. During the first phase, information about many tools is gathered. The goal of this phase is to select the most interesting candidates for the final selection. Many references available on the market can be used in this process of evaluation (see Chapters 9 and 10 for references of PDM and SCM tools). The experiences of other companies, available through news groups, other sources on the Internet, or through direct contacts, are very valuable.

The result of the investigation is a list with a limited number of tools (between five and 10 candidates). The process can continue through several iterations with deeper analysis until the two or three most interesting tools are selected. The final step is an extensive comparative analysis of the short list of final candidates. Different types of analysis should be provided:

• Quantization of the tool features related to the requirement specification;

• Demonstrations of the tools;

• Tool test in an experimental environment similar to the real environment in the organization;

• Vendor’s position on the market and its capacity to further develop the tool;

• Cost analysis and estimation of return on investments.

The result of this phase is an analysis report describing the different alternative tools, their advantages and disadvantages, and finally one tool recommended for selection. The final decision should be left to management and a decision group that was not directly involved in the evaluation process.

The entire evaluation process is described in more detail in Section 7.2.

7.1.1.5 Tool deployment

Tool deployment is a separate phase, most probably performed as a separate project. The final goal of this phase is a successful use of the tool in everyday work. The procedure may be executed in very different ways, depending on the type of tool, the organization, or the processes in which the tool will be used. Most often, there is a strong requirement that development and production efficiency are not prejudiced during this phase. This requires a gradual introduction of the tool and a systematic training of its users. Stepwise integration is intended to increase the number of people in stages using the tool, the number of new functions used, and finally the interoperability between the new and other tools. As the organization deploying a new tool is usually not familiar with this tool, it might consider including some advice on using consultants to assist in the selection or deployment processes. However, the use of consultants can be a double-edged sword; on one hand they can provide much-needed expertise, while on the other hand they can add cost and inhibit the team from finding its own way forward. All of the tool companies will offer support for the deployment of their tool, and some guidance would be useful.

7.1.1.6 Analysis of the deployment process

An analysis of the project itself is included as a standard part of a project. For example, the project efforts, costs, and results should be analyzed. The experience of the deployment of a complex tool is very valuable to an organization; the experience is related not only to the tool, but also to the organization and to its procedures.

7.1.1.7 Verification of the requirements in full use of the tool

Many tools may appear to work perfectly in a simple, experimental environment but are not able to provide the same services in a large-scale industrial environment. The scalability is one the most frequent pitfalls of the evaluation process. Another pitfall is the inadequate communication and misunderstanding between customers and users. The customers buy the products, but in large organizations they are not necessarily the users—and they may have different criteria for the acceptance of the tool (e.g., the price, the vendor market position, and long-term goals) than normal end users (to whom the most important factor is usability). As a consequence, the final result, the utilization of the tool in the everyday process, may be different from that envisaged during the evaluation and deployment procedures. To avoid the creation of an incorrect impression of the tool (this is especially important for the management), the tool’s performance in its full use should be verified. In the verification process, the everyday users of the tools should be involved and their experience should be compared with the initial requirements. The result of the process is a verification document, which can be used for possible negotiations with the tool’s vendor and as input to other tool evaluation and deployment projects.

---