6.3 Examples of integrations

---

More advanced PDM and SCM tools provide support for integration through their APIs, integration languages and commands, Web-based services, and components. These facilities make it possible for end users to perform integration. The integration process is very complex, and it often requires considerable knowledge of both systems and different technologies. For this reason, many end users are not capable of performing the integration alone and need the assistance of the vendors or consultant companies providing such service.

This section gives an overview of two integrations: the integration of the SCM system ClearCase [3] and PDM tool eMatrix [4] developed by an end user, and the integration of ClearCase and the PDM systems Metaphase [5] performed by the vendors and available as a commercial product.

6.3.1 Case study: Integration of eMatrix and ClearCase

The research and development department of the Ericsson Company realized that the SCM tool used (ClearCase, adjusted to the Ericsson environment) did not fully support the entire development process. More concretely, it was found that even for pure software development (i.e., no hardware involved), there is a need to manage much metadata, or additional information, connected to the software items. In many SCM tools, including ClearCase, metadata management is provided through different attributes connected to the versioned items. These attributes are user defined and can be used to store all kinds of metadata. In practice, attributes are seldom used for this purpose due to poor usability in this context; the use of many and complex attributes during software development has been found to be impractical. Even for the CM group that manages software products and requires a support similar to that which PDM systems provide, the utilization of attributes alone was insufficient for providing the support required. For this reason, it was decided to use a PDM tool to handle metadata and implement the development process, using the SCM tool to store and retrieve the source code and other software items. A PDM tool eMatrix was selected for integration with ClearCase.

6.3.1.1 User functionality

The idea of using both a PDM and an SCM tool is to provide better support for the management of metadata related to the source code being developed. This provides project managers and developers with additional support. The users obtain improved support within at least three important activities—configuration identification, baseline handling, and configuration control including change request handling.

6.3.1.2 Configuration identification

Configuration identification is intended to give each CI a unique identity (see definition in Section 3.3.1), which is required in several different environments. For example, eMatrix uses a triple name, type, and revision to identify a CI. ClearCase uses version extended pathname. Mappings between these identities are performed in the interface product, MxCC, by which the product identity was converted to ClearCase attributes. It is also possible to define the entire business model (information model) within eMatrix. The type hierarchy attributes and their relationships can be modeled. It is also possible to define the states and the life cycle of all defined types of objects.

6.3.1.3 Baseline handling

An important type of object stored in eMatrix is the baseline object (BL object). All data connected to this object, including its processes, is also stored within it. Important data referred to from a BL object are the source code files included in the baseline. These files are represented as CI objects in eMatrix. A CI object contains only metadata and a reference (called link) to the real storage in ClearCase (using a foreign vault facility in eMatrix).

A baseline process may vary in different projects, but some core functionality is provided by the integration to cope with the most common procedures. The project management is benefited principally by being able to work entirely within the PDM tool. Metadata from the source code can now be stored and managed together with the other project data. Support for software developers is also improved. A BL object has a status flag, which is initially set to “preliminary.” When the status flag is set to “approved,” a baseline label is set in ClearCase. All CI objects referred to by the BL object are labeled (i.e., the specific version referred to is labeled). The developers can thus be aware of baselines being set in the PDM system by the managers. This makes it possible to select versions from any branch to define a baseline and to be able to freeze the specified version of each variant separately. This also provides support for different types of development processes and generation deliverables (such as feature based or incremental).

6.3.1.4 Configuration control

An important part of configuration control is the management of changes. The change management is usually realized via change requests, which are stored in eMatrix as CR objects. The project management benefits mostly CR objects, being able to manage data uniformly and independently of which type of change is concerned, but the integration also provides a communication link between the management and the developers used during the process. A CR object also has a status flag. When set to “approved,” this information is propagated to ClearCase by updating attributes for all related files. This provides certain support in the form of warnings and process guidance:

• When a developer checks out a file he or she is informed of all CRs registered on that branch, including CRs merged from other branches. This makes it possible to check the satisfaction of the requirement that the CR he or she is to work on is registered.

• When the file is checked in, the system prompts the developer to mark the CR as implemented, if all changes related to it are completed. When a CR is marked as implemented, this is automatically registered in ClearCase and, after certain manual communication, also in eMatrix.

• According to the CR process, all CI objects referred to from a CR object must also be included in a baseline (i.e., referred to from a BL object). This can be checked both in eMatrix, where the same physical CI object is referred to, and in ClearCase, where the developer is warned if a version has a baseline label name but the CR is not registered in the attributes.

Figure 6.8 shows the architecture of the two tools and their integration. In eMatrix, a BL object, BL1, is stored and refers to three CI objects. A CI object is a link to a specific version of a file stored in ClearCase. A CR object is also stored in eMatrix, referring to the two CI objects implementing the CR.

Figure 6.8: Relationship between CIs in eMatrix and ClearCase.

Baseline establishment is always initiated from eMatrix. When a baseline item is approved by a CCB, each ClearCase CI in this baseline item is uniquely identified with a baseline item label. This label is sent from eMatrix when the CM manager approves the baseline item in eMatrix. When the designer in ClearCase checks out a baseline item, he or she will be informed (warned) if there is no approved CR connected to that CI version.

CRs are managed from eMatrix. CRs can only be written against an approved baseline (i.e., CRs can only refer to CIs that are also referred to by a BL object). When a CR is approved by the CCB, which affects a ClearCase CI, the CR number is forwarded to ClearCase and applied on the actual CI version. When the ClearCase designer checks out the actual CI, he or she will be informed that a CR is approved on that CI. When the designer implements the CR, he or she then inputs the implemented CR number. Metadata from ClearCase CIs is forwarded to eMatrix.

In ClearCase, each branch has its own set of attributes. CRs are registered in ClearCase using four attributes: sum_CR_list, sum_CR_finished, sum_merged_CR_list, and sum_merged_CR_finished. These list all CRs—all implemented CRs for that branch and CRs from branches merged into that branch, respectively. Each attribute has a value consisting of a list of CRs following the syntax in CC: of CR (e.g., 2:CR4 2:CR7 4:CR8—see Figure 6.8). When a CR is registered, it is appended to the attribute sum_CR_list. Note that attributes to all files referred to from the CR object are affected. When the CR is implemented, the same procedure is followed for the attribute sum_CR_finished.

Attributes in ClearCase are connected to a branch. This means that when a new branch is created, a new set of attributes is also created. CRs are not copied or moved when a new branch is created, but the new attributes are empty. When a branch (e.g., a branch designated bugfix) is merged (e.g., with the branch main), all CRs registered in the bugfix branch are copied to the merged lists in the main branch. When a developer checks out or checks in a file, the system presents all CRs registered for that branch and CRs from the merged branches.

6.3.1.5 Roles and information exchange

Software developers normally work in ClearCase. Some updates in eMatrix also result in ClearCase being updated. Information imported from eMatrix can be used to keep developers informed about up-to-date states in the project and to initiate the activities required. However, information flow from SCM to PDM is not obtained automatically. When a developer reaches a phase where metadata stored in eMatrix should be updated, he or she will send a notification to the project management using some other tool (e.g., e-mail).

The project management (e.g., project leader or configuration manager) normally works only in eMatrix. The configuration manager creates CRs in eMatrix. A CCB, which consists of a project leader, configuration manager, and, if necessary, developers, analyzes the impact of new CRs and creates the structure in eMatrix and the links to the correct versions in ClearCase.

6.3.1.6 Integration architecture

Figure 6.9 shows the integration architecture. Note that the two types of users, the configuration manager and the ClearCase manager, work with their respective tools.

Figure 6.9: Architecture ClearCase and eMatrix interoperability.

The interface MxCC uses the standard Adaplet™ to access ClearCase from eMatrix. In addition to links, the interface consists of attributes mapped between the tools and ClearCase commands executed from eMatrix. The Adaplet™ provides a dynamic view of ClearCase information and is used for searching, displaying, and updating ClearCase information from eMatrix clients. For example, a user working within eMatrix can retrieve all information about a file stored in ClearCase. Metadata stored in eMatrix cannot be retrieved from within ClearCase. The interface is partly event driven. Certain operations in eMatrix also result in actions within Clear-Case. The reverse, however, is entirely manual. All modifications to data stored in eMatrix are made through the eMatrix standard interface and are not the result of operations in ClearCase. CIs may be stored in both Clear-Case and eMatrix and are updated as required by the process. Updates initiated in ClearCase are displayed on demand (manually) by a refresh function in eMatrix related to CRs and baseline items. Customizations in eMatrix are performed using standard eMatrix scripting (MQL/Tcl), and the ClearCase scripts are built using Perl.

6.3.1.7 Client-server and server-server architecture

The eMatrix server and the ClearCase server do not communicate with each other directly. Instead the eMatrix client uses a local ClearCase client to set the dynamic view and to communicate with the ClearCase server. The ClearCase designer works with the standard ClearCase client and can thus only reach information stored in the ClearCase server (i.e., the integration is asymmetric). Figure 6.10 shows an example of client-server architecture. Two types of PDM clients exist. A thick client includes business logic and the integration part between eMatrix and ClearCase. The thick PDM client communicates with the ClearCase server through the built-in ClearCase client working as a proxy. The ClearCase client makes it possible to define Clear-Case settings (such as views) locally. A thin client is encapsulated in a Web browser, and it communicates only with the PDM server. The PDM server consists of the eMatrix server and the integration software with ClearCase. The integration part is implemented in the same way as for the thick PDM client. The ClearCase client is used for assessing information from the ClearCase server. In this solution, users cannot set up the ClearCase setting locally, but the same settings are valid for all clients.

Figure 6.10: eMatrix and ClearCase client-server architecture.

Figure 6.11 shows a solution for distributed development. As described in Chapters 3 and 4, SCM tools often provide replication of data in many distributed servers. ClearCase, for example, has its MultiSite synchronizing two servers. However, PDM tools such as eMatrix do not provide this support, and thus clients at different sites must be connected to the same server.

Figure 6.11: Architecture of distributed systems.

6.3.1.8 Conclusion

Both tools are used where they are most effective and use information from other tools, thus producing a synergistic effect. There are, however, some solutions that could be improved:

• Manual update from ClearCase to eMatrix is potentially problematic. Apart from slowing down the process, it may lead to data being inconsistent in eMatrix and ClearCase when a change in ClearCase has not yet been registered in eMatrix. It should be possible to implement an event-driven interface from ClearCase to eMatrix also. An automatic update of eMatrix through polling has been tested but significantly decreased the performance of the process.

• The developers are noticed/warned by the system at check out and check in (i.e., at late stages in the process). The information stored could be made available to the developers earlier to enable them to take positive action on their own initiative, instead of being informed of their errors by the system or prompted when considered necessary by the system.

• When a CI object is created in eMatrix, it refers to a specific version of a file stored in ClearCase. There is support that simplifies finding the correct version, as an alternative to only presenting the version graph of a selected file. The support for searching for files that belong to a specific configuration or baseline is very limited.

6.3.2 Integration of Metaphase and ClearCase

Integration of Metaphase and ClearCase exists in the form of a product, and a more detailed description can be found in [6]. The goal of this integration is to make it possible to retrieve data from ClearCase into the Metaphase environment. Data imported from ClearCase is treated as software components that are managed in the PDM system, along with all of the hardware components within the same product structure and within the delivery structure to the customer. Later releases will cover the import of information from Metaphase in ClearCase. The interface is built on a data exchange facility, in which Metaphase runs ClearCase commands with arguments through Perl scripts. The results from ClearCase are stored in an XML file. Figure 6.12 shows how the exchange is performed.

Figure 6.12: Data exchange architecture.

Because the ClearCase system manages files and directories as seen through different views and Metaphase objects, the integration functionality focuses on operations to provide access to the managed files and directories in a manner consistent with the Metaphase look and feel. To access any ClearCase data, a view is required; therefore, the Metaphase/ClearCase integration requires views from ClearCase to provide access to the contents. In Metaphase, access to directories is available on two levels: access to the file system followed by access to a directory location. In the case of the Metaphase/ClearCase integration, this model maps naturally to the two levels associated with accessing ClearCase views. The top-level directory containing the views maps directly to a file system in Metaphase. The views under the top-level directory map directly to Metaphase Work or Vault Locations. To obtain full information from ClearCase, the ClearCase items must be registered as objects in Metaphase. There are two different ways of accessing data from ClearCase—through the ClearCase way of describing the actual version, or through a static view defined in Metaphase. This view is not the same as a ClearCase view. A third way of finding data in ClearCase is by using the configuration specification rule files, but this function will not be available until later releases.

The following conditions are assumed to be satisfied to obtain a successful-integration.

• The file system managed by ClearCase must be defined in Metaphase first.

• A particular ClearCase view must be set up before being used in Metaphase.

• A ClearCase view must exist before it is possible to register in Metaphase.

• The owner of the Metaphase installation software must be the owner of the ClearCase VOB mount point.

• A view in ClearCase cannot be updated from Metaphase.

• Only metadata of one file is available at one time in ClearCase; concurrent transfer of metadata of a number of files is not possible.

• Asoftware product, built in ClearCase, managed in Metaphase, must be registered in the PDM system. This means that the product will be placed under version control in both systems.

• A software product managed in Metaphase is stored within ClearCase, but metadata is placed in both systems.

These requirements indicate the complexity of the integration. The end user must have an in-depth technical knowledge of both systems to understand how to use the interface and to understand the mixed terminology within the manuals and the interface. To reach a mutual understanding, the terminology must be common to the two systems or a translation table must be available.

---