8.2 Data to Collect

If you're not already collecting historical data, you can start with a very small set of data:

Size (lines of code or something else you can count after the software has been released)
Effort (staff months)
Time (calendar months)
Defects (classified by severity)

This small amount of data, even if you collect it only at the completion of two or three projects, will give you enough data to calibrate any of several commercial software estimation tools. It will also allow you to compute simple ratios such as lines of code per staff month.

In addition to the fact that these four kinds of data are sufficient to calibrate estimation models, most experts recommend starting small so that you understand what you're collecting (Pietrasanta 1990, NASA SEL 1995). If you don't start small, you can end up with data that's defined inconsistently across projects, which makes the data meaningless. Depending on how you define these four kinds of data, the numbers you come up with for each can vary by a factor of 2 or more.

Issues Related to Size Measures

You can measure the size of completed projects in Function Points, stories, Web pages, database tables, and numerous other ways, but most organizations eventually settle on capturing size-related historical data in terms of lines of code. (More details on the strengths and weaknesses of using LOC measurements are discussed in Section 18.1, "Challenges with Estimating Size.")

For size in lines of code, you'll need to define several issues, including the following:

Do you count all code or only code that's included in the released software? (For example, do you count scaffolding code, mock object code, unit test code, and system test code?)
How do you count code that's reused from previous versions?
How do you count open source code or third-party library code?
Do you count blank lines and comments, or only non-blank, non-comment source lines?
Do you count class interfaces?
Do you count data declarations?
How do you count lines that make up one logical line of code but that are broken across multiple lines for the sake of readability?

There isn't any industry standard on this topic, and it doesn't particularly matter how you answer these questions. ^[1] What does matter is that you answer these questions consistently across projects so that whatever assumptions are baked into the data you collected is consciously projected forward in your estimates.

Issues Related to Effort Measures

Similar cautions apply to collecting effort data:

Do you count time in hours, days, or some other unit?
How many hours per day do you count? Standard 8 hours or actual hours applied to the specific project?
Do you count unpaid overtime?
Do you count holidays, vacation, and training?
Do you make allowances for all-company meetings?
What kinds of effort do you count? Testing? First-level management? Documentation? Requirements? Design? Research?
How do you count time that's divided across multiple projects?
How do you count time spent supporting previous releases of the same software?
How do you count time spent supporting sales calls, trade shows, and so on?
How do you count travel time?
How do you count fuzzy front-end time—the time spent firming up the software concept before the project is fully defined?

Again, the main goal here is to define the data you're collecting well enough so that you know what you're estimating. If your data from past projects includes a high percentage of unpaid overtime and you use that historical data to estimate a future project, guess what? You've just calibrated a high percentage of overtime into your future project.

Issues Related to Calendar Time Measures

It's surprisingly difficult in many organizations to determine how long a particular project lasted.

When does the project start? Does it start when it gets formal budget approval? Does it start when initial discussions about the project begin? Does it start when it's fully staffed? Capers Jones reports that fewer than 1% of projects have a clearly defined starting point (Jones 1997).
When does the project end? Does it end when the software is released to the customer? When the final release candidate is delivered to testing? What if most programmers have rolled off the project a month before the official release? Jones reports that 15% of projects have ambiguous end times (Jones 1997).

In this area, it's very helpful if the organization has well-defined project launch and project completion milestones. The main goal, again, is simply to understand the data you're collecting.

Issues Related to Defect Measures

Finally, defect measures also vary by a factor of 2 or 3 depending on what's counted as a defect:

Do you count all change requests as defects, or only those that are ultimately classified as defects rather than feature requests?
Do you count multiple reports of the same defect as a single defect or as multiple defects?
Do you count defects that are detected by developers, or only those detected by testers?
Do you count requirements and design defects that are found prior to the beginning of system testing?
Do you count coding defects that are found prior to the beginning of alpha or beta testing?
Do you count defects reported by users after the software has been released?

Tip #37

In collecting historical data to use for estimation, start small, be sure you understand what you're collecting, and collect the data consistently.

Other Data Collection Issues

Historical data tends to be easiest to collect if it's collected while the project is underway. It's difficult to go back six months after a project has been completed and reconstruct the "fuzzy front end" of the project to determine when the project began. It's also easy to forget how much overtime people worked at the end of the project.

Tip #38

Collect a project's historical data as soon as possible after the end of the project.

While it's useful to collect data at the end of a project, it's even more useful to collect snapshots of a project as it's underway. Collecting data on size, effort, and defects every 1 to 2 weeks can provide valuable insight into your project's dynamics.

For example, collecting a snapshot of reported defects can help you predict the rate at which defects will be discovered and will need to be fixed on future projects. Collecting data on effort over time can help you understand your organization's ability to mobilize staff to support a project. If one project staffs up more slowly than desired, it might be a fluke. If your historical data says that the last three projects have each staffed up at about the same rate, that suggests that you're facing an organizational influence that can't easily be changed on the next project.

Tip #39

As a project is underway, collect historical data on a periodic basis so that you can build a data-based profile of how your projects run.

^[1]The closest the software industry has to a standard definition is a non-blank, non-comment, deliverable source statement that includes interfaces and data declarations. That definition still leaves a few of the questions unanswered, such as how to count code reused from previous projects.