Continuous Integration

I'm sure you all recognize how big a showstopper manual integration on a monthly basis, for example, can beeven if your team just consists of two developers on a project. The problem increases exponentially when you add more developers. Among other things, it costs lots of time, it is error prone, and you will find that it won't happen as planned. "Just one more day" will be a common sentence used in the project. My friend Claudio Perrone is just the right guy to describe the popular solution to this, known as continuous integration.

The Integration Problem

By Claudio Perrone

Sooner or later, all the components that have been created and modified by different developers need to be built and assembled together to form a single system.

In the past, I've witnessed (and, yes, occasionally caused) spectacular delays in projects originated by last-minute integration efforts where unexpected defects were discovered in the final phases of a development cycle.

A build may fail for a variety of reasons, often in combination, such as

• Poorly tested components

• Wrong (but often plausible) assumptions about exception handling, null values, parameters, global variables, and so on

• Weak design

• Unpredicted behavior on different platforms

• Unexpected differences between release and debug builds

• Obfuscation issues

• Setup and permission issues

• Bugs in the underlying frameworks

Note that, despite all the best efforts, only some of these problems can be limited by the discipline and communication capabilities of the developers involved. The reality is that if the team does not integrate often and if the number of new classes in the system is sufficiently large, you may find yourself anxiously hunting combined bugs foreveran unpleasant scenario commonly known as "Integration Hell."

The Solution (Or at Least a Big Step in the Right Direction)

The key to significantly reducing integration problems is to generate your builds automatically and to use an incremental integration strategy where all the code is rebuilt and tested at least daily, if not continuously. The idea is that if your recently added code breaks a build, you either fix it immediately or roll back the changes to restore the system to the last known good state.

The fundamental parts that constitute a continuous integration system are as follows:

• A machine dedicated to the build process

• A source control system that acts as a central repository for all source code

• A monitoring service that checks the source control system for changes to the source code

• A scripting engine that, when triggered by the previous service, is able to create builds

• A reporting system that can give immediate feedback about the results of a build

There are several integration products available that you may want to investigate. Currently, my favorite is CruiseControl.NET [CC.NET], which I use in combination with NAnt, a very popular XML-based build engine. Both tools are open source. It takes quite an effort to configure CruiseControl.NET but, once up and running, it takes care of calling NAnt scripts whenever there is a change in the codebase. It notifies you of the progress and status of all current builds using a wide variety of client modules, including a little application that uses a Windows tray icon to show summary information at a glance through color coding and notification balloons.

Lessons Learned in My Organization

When we first contemplated the possibility of implementing a continuous integration system at InnerWorkings, most of us thought that it was a really great idea. Although we were already under severe scheduling pressure, we knew that it was definitely worth investing a few days to implement the system. If I think about it now, however, we certainly underestimated the profound impact that such a system would have on the quality of our products and the confidence of our team.

Today we have about 500 continuously integrated solutions, and the number is still increasing. A third of these solutions share a custom-built common framework, which is also integrated. Integration steps for all these solutions include compilation, obfuscation, packaging, testing, and deployment using different configurations and platforms.

Before we started this project, we were told that it would be impossible to integrate all of these solutions continuously. However, I'm convinced that it would have been impossible to do otherwise and that this effort constitutes a critical factor in our success.

Further Information

A good starting point for learning more about continuous integration is a paper written by Martin Fowler and Matthew Foemmel called "Continuous Integration" [Fowler/Foemmel CI].

Thanks, Claudio! Now let's move on to some operational aspects.