Chapter 9: Going About the Development

This chapter introduces some of the steps used by development teams for building communications software. It builds on the details of the previous chapters to outline the various steps during the design and development. The issues addressed include the stages of development, “build versus buy,” and using a simulated environment as well as an OS Abstraction Layer. Tools of the trade like development environments and test equipment will also be covered.

9.1 Product Development Steps

This section outlines the typical steps followed in the design and development of software for communications devices. Individual document titles and formats may vary from company to company according to the software development methodology, but the concepts are the same.

Product development starts from two documents which are provided typically by the marketing and product management teams. These are the market requirements document (MRD) and the product requirements document (PRD). The market requirements document covers the target market, customer requirements, competitive analysis, revenue potential, product fit within company strategy, and so on. The product requirements document is derived from the market requirements document and details the features, feasibility studies, engineering tradeoffs, usability criteria, performance requirements, development and delivery schedules, and roadmap or projected life cycle. Some companies combine these into a single document.

The product requirements document forms a framework for the engineering team, which uses it as input to the high-level engineering design (HLD). The high-level engineering design details specific tasks and modules in the system at an architectural level, including functionality and interfaces. While the document is being prepared, an engineering team works on the system test plan, based on the features specified in the market and product requirements documents. The system test plan (STP) details the test strategy, test setup, tools and equipment required, test topologies, and so on.

The next engineering document is the low-level design (LLD) for each major module, which drills down deeper into the design, including data structures, state machines for the protocols, and so on. Once management approval has been granted to each of the above documents, the coding may begin. After coding and a code walkthrough process, unit testing is usually done on each module, followed by integration testing, which combines multiple modules to form the complete system. System testing is near the final stage of development, when the system test plan is executed.

The complexity of individual stages depends on the nature of the system. For example, unit testing for a simple buffer management scheme may be different from unit testing for a protocol. Similarly, unit testing of a hardware device driver may be more complex if the hardware is still under development.

The following sections cover the typical steps in the design and development of a Layer 3 IP Switch (IPS), using the software development life cycle given above.

9.1.1 Layer 3 Switch Product Requirements

In this phase, which follows the market requirements document, the product manager (in consultation with the engineering team) outlines the features that are required in the Layer 3 IP Switch (IPS) development, including the following:

• System requirements (single-board, multi-board, multi-processor)

• Operating system (Commercial UNIX, RTOS)

• Number and Type of Ports

• Software Protocols

• User Interface and Management (CLI, SNMP, CORBA, XML)

• Performance requirements

• EMI, Power requirements

• Standards Conformance (hardware, software, power, EMI)

• Phases of Development (Phased delivery schedule and expectations)

• Scalability (hardware and software designs to accommodate future expansion and scalability)

The product manager is the bridge between engineering, marketing, and the customer. The product manager plays the role of arbitrator in the event of a mismatch between external market demand and internal constraints. For example, the product may need to use the code from another product line for feature compatibility, or it might need an ASIC to be developed (to handle the performance specified). These constraints are clearly identified at the product requirements stage. Mature product organizations nail several of these issues up front, without the need to revisit them during development.

Build versus Buy

It is common in the product requirements stage for both development and marketing teams to discuss the “build versus buy” question. The engineering team can develop software modules such as protocol stacks from scratch or license the stacks from a third- party source code vendor or from a Linux distribution. Several equipment vendors prefer to license the stacks or components from vendors. This is more common with startup companies building new products. The decision of “build versus buy” is not restricted to software. Hardware designers have to choose between using merchant silicon like switching chipsets instead of developing their own ASIC.

Time to market (TTM) can be one of the most important considerations. Protocol stack software takes time to develop and requires a significant amount of testing for compliance and interoperability. The effort and time required for these steps may be significant. Moreover, since the protocols are standards based, there is very little scope for enhancement or differentiation, which can be another reason to license the stacks.

Development teams struggle with the issue of build versus buy since it requires a careful analysis of the tradeoffs. Table 9.1 provides a quick overview of the issues to be considered when choosing between internally developed code and third-party protocol stacks (“licensed code”).

Table 9.1: Build versus buy issues.

Issue to be Considered	Code Developed Internally	Licensed Code
Time to Market	Longer—since effort & resources required to develop the individual protocols may cost more than the price charged by vendor for code	Shorter
Code Stability	Requires some level of testing until the code is stable and fulfills requirements	Tested and deployed widely—more stable
Complexity	Can be made less complex than third- party code if new code conforms to specific requirements of the system being built	Less or more complex depending upon the architecture—portability considerations may cause code to be more complex than normal
Performance	Optimization can be done for specific system for which it is being built	Cumbersome to optimize for specific system since it has been written to be portable to multiple systems
Cost	Engineering effort + testing effort—can be quite high	Dependent upon vendor pricing
Support	Developers support the code base— highly flexible but can be an issue with limited resources	Vendor provides support and updates to latest version of standards

Despite some of the advantages of licensing, engineers are divided in their opinion about the value of licensing code like protocol stacks from third-party vendors. The architecture of the licensed stack, its fitness for the system being designed, and the quality of the code are some of the factors that contribute to the decision.

9.1.2 High-Level or System Design

For the high-level design phase, the system is decomposed into individual tasks and modules, along with the interfaces between them. In the IPS, this consists of the following:

• Modular Decomposition into the individual routing tasks, IP Switching Task, Driver Details

• Internal interfaces between the modules—including interprocess communication for message passing between modules along with message types

• External system interfaces—including user interface and serial port connectivity

• Global Data Structures used in the system (common routing table) and access routines to manage these

• Events and notifications for each of the modules

• Provide a mapping to product requirements, such as routing table sizes, split control plane in multi-board systems

9.1.3 Low-Level Design

The low-level design (often called detailed design) is the second level of decomposition. Individual modules are broken into submodules with data structures, the interfaces between the submodules, and dependencies between the data structures. In the IPS, this consists of the following:

• Decomposition for individual tasks such as OSPF, IP Switching Task, drivers for the various types of interfaces

  • Depending upon the complexity, each of these modules may have its own detailed design document

  • If a third-party stack is licensed, the low-level design is a porting plan, in which the changes to the licensed software and its interfaces are specified.

• Key data structures in the individual tasks such as interface tables, neighbor tables, and protocol control blocks

• Pseudo code for key functions such as the main loop of the task, the shortest-path- first algorithm for OSPF/IS-IS

• Sizing and performance for each of the modules or tasks

9.1.4 Coding

This phase translates the detailed design into code. In the case of third-party stacks, it includes translating the porting design into code. Coding guidelines, as specified by the project, need to be followed for easy maintenance.

9.1.5 Testing

Unit testing can involve developing stubs for routines that are called by the individual modules being tested, test tasks in a multiple-task environment, and test code that drives the testing. Test packets and messages need to be constructed during this stage. The test packets could be constructed using packet generator test tools.

Integration testing can be done when individual modules are combined together in a phased manner. At each integration phase, the stubs are replaced by the individual routines in the interfacing module, while the test tasks are replaced by the newly integrated tasks. For example, during unit testing of the RIP task, engineers may stub out the interface routines to IP and have a test task that plays the role of the IP Switching task. A similar effort is required for the IP Switching Task. At integration, the “real” RIP task and IP Switching Task are tested together.

Test Plan

All types of testing stages require a test plan. This document specifies the following for each test:

• The scope of the test, including the objectives, and what is not included

• Test Tools to be used, including Analyzers, Packet Generators, external nodes

• Test Setup and data to be used

• Dependencies for the test

• Actual Test Procedure (connections, initialization, packet generation)

• Required Result

• Special instructions or comments

The format of this document will vary according to the organization, but it is important that all test cases include the information specified.

System Testing Criteria

System testing tests the product as a whole, after all the modules have been combined through integration testing. The specific criteria used for system testing are:

1. Functionality, including conformance

2. Stability

3. Scalability

4. Interoperability

5. Performance

Functionality and conformance testing

indicates the testing to be performed to verify that the product satisfies the requirements specified in the product requirements document. With IPS, it can mean forwarding between all ports. Conformance indicates whether the product conforms to the individual protocol specifications and standards. RFC 1812, for example, specifies the behavior for an IP router. Conformance testing will verify that IPS functionality is in accordance with RFC 1812.

Stability testing

is required to ensure that the product satisfies the reliability and quality requirements as specified in the product requirements. Generally this means that the application runs flawlessly for a specified period of time. The IPS can be tested for forwarding of packets between interfaces for extended periods of time to ensure that buffer and memory leaks do not occur.

Stress testing

verifies that the system is able to perform correctly at “full load,” both in terms of traffic and the configuration. In IPS, this can mean that the system has been configured with the maximum number of interfaces, has the maximum number of routes in its routing table, and is tested with traffic destined for multiple routes on all interfaces. In combination with stability testing, this is used to verify that the system performs correctly and in a stable manner.

Interoperability testing

verifies that all components perform seamlessly and flawlessly together. This testing is key when standard protocols are involved. With IPS, it can include testing with third-party routers such as those from Cisco Systems or Juniper Networks. Since IPS needs to be deployed in a network which may include routers from other vendors, this is an essential part of the testing. This stage is less involved, if the protocol stack has been licensed from a vendor that previously verified interoperability with third-party routers. However, if the protocol was developed from scratch, interoperability testing can take longer.

Performance testing

is used to verify the performance requirements specified in the product requirements document. For IPS, this can include forwarding rates between ports at full load, routing protocol convergence, time required to failover to a backup control card, and so forth. Performance numbers provided during this phase are used by marketing to showcase their product.

It should be noted that testing is not linear but a highly iterative process. Some test results such as scalability and stress testing may require a significant amount of time to fix the issues that caused the tests to fail. Regression testing requires that the entire set of tests be rerun after a fix—to ensure that the fix provided does not “break” any of the earlier functionality or tests.