Empirical Investigation

Case Study

We propose that industrial organizations structure a case study with related surveys to provide quantifiable, empirical results. In a case study, key factors that affect outcome (such as the transition to an agile methodology) are identified. Then, specific, predetermined outcome measurements are tracked and analyzed. Case studies are generally performed in a "typical project," and they can be thought of as "research in the typical."

Often, a more preferred research technique is to run a formal experiment. Formal experiments can be referred to as "research in the large," and their results have greater external validity or applicability to many settings [Kitchenham+1995]. However, formal experiments generally require a control group (a group that uses the organization's existing development methodology) and an experimental group (a group that uses the new methodology), often replicating the same or very similar projects.

It is very difficult to affordably run formal experiments in industry [Fenton+1998]. As a result, we suggest that organizations structure a case study that compares pre-agile metrics with post-agile metrics. "Although they cannot achieve the scientific rigor of formal experiments, case studies can provide sufficient information to help you judge if specific technologies will benefit your own organization or product" [Kitchenham+1995]. We propose a multiproject study as defined by Basili, Selby, and Hutchens [Basili+1986], whereby the case study examines objects across a single team and a set of projects. It is beneficial for the projects to be as similar as possible in complexity and scope and for as many of the same people to be involved in the pre- and posttransition studies.

Surveys could be used to poll a set of data from the software engineers involved in the study. Pretransition and posttransition surveys could assess their reaction. Additionally, surveys could be used to measure customer satisfaction with software releases.