7.2 Evaluation

---

The most appropriate form of the evaluation process is a project that identifies the resources, the requirements, the constraints, the activities, the deliverables, and the time planning.

7.2.1 Evaluation team

As different categories of people work with PDM and SCM tools, either as pure users, as developers, or as support and administration staff, it is important that representatives of these groups should be involved in the evaluation process. On the other hand, it is counterproductive to build a group too large. The best solution is to form a core operational group that will work continuously in the project, while other representatives can be co-opted for particular activities. A typical core operational group would consist of the following representatives:

• Administration staff will deal with software administration. Typically, they are concerned with the resources that the tool may require (e.g., primary and secondary memory, CPU and network loading, and network and node configurations).

• Supporting staff will be responsible for the administration and support of the particular tool. They will analyze the possibilities of tool configuration, the requirements of its administration, and security issues.

• Experts will be concerned with the functional features of the tool. Which requirements can the tool meet? What is its potential (i.e., the possibilities of extending its functionality or of adjusting it to the processes involved)?

• Developers and other users of the tool will be focused on the tool’s behavior in everyday use. They will be concerned with its usability, its interoperability with other tools, its possibilities of adjustment to the local environment, its usability, and its performance.

• Quality assurance and process improvement groups will be concerned with processes in which the tool will be used. The new tool may have a significant impact on the processes, requiring extensive changes.

7.2.2 Evaluation project activities and milestones

The main project activities and milestones are similar for both PDM and SCM tools. They were already mentioned in Section 7.1, and are now presented in Table 7.1. The particular activity flows are described in detail in the following sections.

Table 7.1: Activities and Milestones of the Evaluation Project

Activity	Milestone/Deliverable
Specify requirements for the tools to be evaluated	Requirements specification approved
Investigate the market and find the most appropriate tools	Market investigation report approved The main candidates selected
Collect experiences of using specific tools from other companies and other sources
Evaluate the most interesting tools: Evaluate their functional features Evaluate the vendors’ positions on the market Estimate the initial and the total costs of the tool deployment Estimate return on investment	A comparative analysis report A tool is recommended
Propose the deployment process
Analyze the finding from the reports	A tool is selected

7.2.3 Cost analysis

There is a large variety of PDM and SCM tools on the market—more than 100 different SCM tools (see Chapter 9) and many PDM tools. The costs of SCM tools vary considerably, from free software to expensive products that can only be afforded by large companies. Most PDM tools are expensive. License costs, however, are not the total costs, and an analysis of the total costs should be performed in order to compare the alternatives. The following types of costs should be considered:

• Product/license costs, including the licensing model;

• Maintenance costs, including the conditions and terms for upgrades;

• General support costs, including costs for consulting and system customization services;

• Training costs;

• Deployment costs;

• Hardware costs;

• Costs for additional internal development;

• Costs for additional software required.

It should be stressed that the costs for consulting and training tend to dominate over the license costs. In some cases, the vendor can supply an estimate of the total cost of ownership, which includes all costs for upgrades and maintenance as well.

The costs are of two kinds—external, with costs for the external support paid by the company, and internal, the costs for internal activities. The external costs are visible, while internal costs can be hidden and often underestimated. The costs should also be analyzed in a time frame: the initial costs and annual costs. Advanced and complex tools usually have large initial costs that discourage potential purchasers. Figure 7.2 shows an example (based on [1]) of the initial and maintenance costs of two tools during 2 successive years.

Figure 7.2: Distribution of internal and external costs.

Examples of two different categories of tools are shown. Tool A is an expensive tool that requires a large investment in licenses, equipment, and training. The external costs are considerably higher, especially for maintenance. Tool B is an inexpensive tool with low license fees but also providing a low level of functionality. To achieve a better functionality, additional functions must probably be developed internally, which increases the internal costs. The maintenance of internally developed functions may require more effort than the maintenance of the more expensive tool.

7.2.4 Return on investment

The total cost is not the only information needed to make the correct decision about the deployment of a tool, and, likewise, only calculating the total benefits of the use of the new tool is an insufficient basis for making a decision. The difference between the benefits and the costs is the main factor to be considered. This difference is designated as return on investment. It can be expressed in an absolute value or as a relative value renormalized to the investment. In the second case, the total benefit is divided by the total costs and expressed as a percentage. The investment is justified if the return on investment is greater than one (100%). The costs (especially the external costs) are easier to estimate. The benefits should include all savings and improvements that are the result of the use of the tool. This is difficult to estimate. A detailed analysis of all activities affected by the new tool should be made. In addition, reports of similar analyses and experience reports should be used. It is also useful to perform a further analysis of the return on investment when the tool is integrated and used in normal production, and compare the results with the estimated values.

Return on investment is not a static value but varies over time. Most often the initial investment is much higher, especially for large investments. Figure 7.3 shows an example analysis of three different tools. Tool A is a more advanced tool, and its deployment requires higher direct costs initially (for product licenses, change in the infrastructure, and training). Tool B has lower licenses costs but requires more adjustments and internal improvements of the tool. The tool costs, direct and indirect, may be consistently lower, but the functionality and the return on the investment can never reach the same level as for the advanced tool. Tool C is an example of an internally developed tool. There are no license costs, but the costs of the tool development and its maintenance are high and will increase, while it will be more and more difficult to achieve the desired functionality and return on investment in the future. Deciding when the internally developed PDM or SCM tool should be replaced with one of those available on the market depends on when the return on the investment for the new tool will exceed the return on the investment of the internally developed tool ( t0 in Figure 7.3), and when the total cost of using a new tool will be less than the cost of the development and maintenance of the old tool ( t1 in Figure 7.3).

Figure 7.3: Estimated investments and return on investment.

7.2.5 Evaluation of the tool vendor

It is also important to evaluate the tool vendor for the assurance of the future use of the tool and for the negotiation strategy. Because the choice of a system is a long-term decision, the financial status of the vendor is of interest. A vendor with poor finances might be bought by another vendor, which might merge the development of the acquired vendor’s tool to the purchasing company’s tool or simply stop further development. As experienced over the last 8 years by the Ericsson sourcing community, in a period of 2 years about 20% of the existing companies were merged with or bought by other companies.

To improve the cooperation and increase the quality of a service, some companies (mostly large companies) may choose to start a partnership with tool vendors. If the consumer company is enough large and strategically important, the vendor company will also be interested in a partnership. Such partnership in principle makes it easier to acquire the tool and integrate it with other tools. Other companies may want to sign a special agreement with the vendor company. Also, in this case, a mutual business interest must exist to make this happen. In most of the cases, however, the customer company cannot directly influence the vendor. In any case, independent of any type of cooperation, an evaluation of the vendor should be a mandatory part of the tool evaluation.

The following points are some examples of what should be studied more closely:

• The market position of the vendor and of the tool;

• The vendor’s financial statement;

• The vendor’s internal organization, including the experiences of persons making up the board of directors and the management team;

• The present core business of the vendor and relationship of the tool to the core business;

• The vision and strategy of the vendor—the technology roadmap;

• Strategic alliance of the vendor to their customers and suppliers;

• Customer list and reference users;

• The vendor’s competitive advantages and disadvantages;

• The ownership of the tool.

Many SCM tools exist as free products, some of them largely widespread (for details, see Chapter 10). Most of them belong to a category of simpler tools, which may be either used directly for simpler development processes or as a basis on which additional support can be built. There are no free PDM tools, but parts of them are free. Document management and content management tools (see Chapter 11), underlying databases, and implementation of certain middleware products are available and used for further building of PDM functions. How should companies acquiring free products treat them? The answer is simple: for the end user and for the company, the result should be the same, independent of whether the tool is commercially available or free of charge. This means that it must provide a customer support and maintenance support. This can be obtained either directly by the company itself or by a consultant company. In any case, all criteria discussed in the previous sections should be applied when evaluating this type of software. The ownership of the free tools and free software is often not precisely defined. Different types of free software exist (see Chapter 10), and the company using this software should be aware of these differences.

7.2.6 SCM evaluation example

The concrete requirements for the tool should be a result of an analysis of the organization’s business goal, development, and other processes. There are no unique requirements, and no one tool is the best choice for all cases.

The requirements can be classified according to the main SCM functions, and then other requirements related to integration, flexibility, and the possibilities of modification and adding new functions can be added. Table 7.2 includes an example of such a classification.

Table 7.2: Classification of SCM Requirements

Classification	Feature
Version management	Check in/check out process CIs history information CIs version attributes Configuration and baseline process
Configuration management	Possibility of finding differences between two baselines Possibility of merging differences between two baselines
Build management	Build support (tools such as Make) Possibility of reproducing the build procedure with exactly the same results Possibility of obtaining information on how the build was performed
Workspace management	Relationship between repositories and workspace Flexibility in changing workspace Cooperation between workspaces of different developers
Change management	Mapping CRs to the changes made in the code Finding changes (CRs) implemented in a baseline Propagating changes from one baseline to another Providing measurements and statistics related to changes and efforts
Release management	Automation level in generating a product release Automatic generation of release documentation Possibility of showing differences between two releases
Parallel development	Team support (awareness) Merging possibilities
Distributed development	Data synchronization and replication Merging possibilities, export/import functions
Integration	Integration and interoperability possibilities
Administration	Installation efforts required (for servers and for clients) Ability to convert current data to format required by the tool Efforts required for everyday administration
Usability	Graphical user interface Simple use
General	Tool quality and tool reliability Requirements on resources Performance

The evaluation is performed as follows. The features are tested and graded (for example, from 0 to 10). The tests should be performed in different ways, as mentioned before, by different groups that will set grades independently. An average value, or the value defined by the consensus between the groups, should be set as the final grade. The grades can be graphically presented for all tools tested, for example as shown in Figure 7.4. The figure shows three different tools and their characteristics. The inner thick circle in the figure defines the minimum grade required. From the figure, it is apparent that only one tool meets all of the requirements (which still does not mean that this tool will be selected).

Figure 7.4: Graphical presentation of the fulfillment of functional requirements.

7.2.7 PDM evaluation example

Choosing a PDM system involves more than choosing a technical solution. It is therefore recommended that the vendor, commercial system information, general system information, and the PDM functionality of the system should be studied to obtain a complete picture.

Similar to the example for SCM tools, a list of functional requirements for PDM tools should be prepared. It is recommended that the evaluation of the PDM functionality should be based on a broadly accepted terminology to avoid confusion of terms. The required functionality differs between companies. It is therefore recommended that the required functionality should be grouped according to, for example, CIM Data’s classification [2] of PDM functionalities:

• Data vault and document management;

• Workflow and process management;

• Product structure management;

• Program management;

• Classification management.

In addition to the PDM functionality of the system, its general properties should be investigated:

• User friendliness;

• Scalability;

• Possibility of integration with other systems and tools;

• Flexibility.

User friendliness includes such characteristics as the user interface and response times. Scalability means that the system will retain its performance as it expands (e.g., with the number of users, the number of sites, and the volume of data managed). Flexibility is a measure of how easy it is to customize the system.

---