9.9 CAPTURE INFORMATION REQUIREMENTS

< Day Day Up >

This is the core activity of this stage. It is important because there are various techniques that can be used to derive a set of Software Metrics that can then be used by the organization. One technique is to go into a development team, a project or the data from a previous project and then to measure everything you can possibly think off. This is the data hunters strategy. Having got a vast array of base data you then apply various statistical techniques to find out what all this information is telling you. Your hope is that it is telling you something useful that can then be used to improve the development process.

One problem with this approach is that the very quantity and diversity of information can produce apparently meaningful results, information that in reality is telling you very little. For example, you may find that the number of comments per thousand lines of code is running at about 30%. Let us say that you are really clever and that you find the spread of this comment density ranges from 1 % to 75%. What conclusions can you draw from this? The answer is very little.

Now it can be argued that by relating comment density to error-fixing productivity you may be able to show that high comment density relates to high productivity. In some ways this may make sense if your engineers are using the code comments as the primary source of system documentation! Before throwing your hands up in horror, go and ask them if this is the case. You may be surprised. However, the very quantity of data that you will be collecting will probably mean that you cannot form that relationship from a statistically significant number of projects or observations within a project. Essentially, the data collection costs when working under the data hunting approach are simply too high because you do not know which base measures will be useful.

Alternatively, you may give the matter some thought and then go and measure everything that seems to make sense. Under Murphy's Law, the one thing you do not measure will be the thing you should have measured. I state this from bitter, personnel experience.

In one installation we measured every element of a system design that we felt could possibly be a valid predictor of software size, and hence development costs. The collection and analysis took quite a while and the results were interesting. In fact the best predictor seemed to be the number of pages in the functional specification. Try selling that one to senior management!

By the by, it is worth pointing out that we were using a relatively formalized, diagrammatic documentation technique so perhaps we should not be too surprised by this result. But the bottom line is that when we presented our results a member of the audience asked why we had not measured a particular design attribute. The answer was that we had not thought about that one, so we agreed that we would try it out. Of course that attribute, the number of data dictionary definitions, turned out to be the best predictor of all, at least for new systems which was our area of interest at that time.

Fortunately, there was also a need to get a size measure for enhancement work as well and the data dictionary model broke down when we tried to apply it to that sphere of engineering. We managed to save face to some degree!

Believe me, measuring everything in sight is fraught with difficulty and is not a cost effective method of deriving Software Metrics.

Fortunately there are other approaches that can be used. The most widely recognized and respected approach to metric derivation is something called the Goal/Question/Metric, or GQM, paradigm and this is discussed in some detail in Chapter 10 which describes the Component Design stage of the SMP development and implementation lifecycle.

The foundation of this approach, developed by the Software Engineering Laboratory in the USA, Rombach (1) , and is aimed at making practitioners ask not so much "what should I measure?" but "why am I measuring?" To be able to answer this question you need to be able to answer the additional question, "what business needs does the organization wish its measurement initiative to address?"

Capturing and specifying measurement requirements uses the same procedures as classic systems analysis, i.e., interviews, workshops, prototyping. And the foundation of good systems analysis is preparation.

Taking requirements from a number of sources you will probably identify certain similarities between requirements that will enable you to focus on those for which there is a general need. In this way you will be able to group requirements into particular classes. The benefit of doing this is that it reduces the amount of duplicated effort that may otherwise be expended. I offer the table below as one possible set of headings that can be used to group metrics requirements and these can be used to focus an interview or workshop session:

Table 9.3
Project Metrics
Business Entity Metric
Cost Estimation
Quality Assessment
(Applied Design Metrics) and Prediction
Project Control

I define Project Metrics as being any measure of a process or product attribute where the source data and the subsequent application is confined to a single project or product and the associated team. For example, consider field reliability of a product. The primary customer for such metrics is almost invariably the project manager.

Business Entity Metrics can be defined in the same way as project metrics except that the source data is drawn from a number of areas that, together, form a managerial unit. In terms of our example organization, a business entity could be the Test and Integration Group, the Product Division or the whole of the Systems Development area. An example of a business entity metric requirement would be of a product group manager wishing to monitor the general reliability levels of their products over time as a group rather than as a set of individual products. Another example would be where the management board wished to monitor return on investment for the whole of the IT function. In both cases the requirement could be met by aggregating information from measures taken against individual products or product groups respectively.

These areas have been discussed in Section One, project and business entity metrics coming under the general heading of Management Information.

Having obtained a set of requirements and grouped them into classes you may well see that you have two macro classes. The first will be requirements for information, the second will be for techniques that can be directly used to improve the development process in some form or another. Each macro class requires a slightly different approach.

Although these two approaches are really part of the Design stage of our metrics program I will briefly introduce them here. Requirements for information will mean that you need to adopt a model, either by using one that is generally well accepted and with which you feel comfortable, because there is no reason to reinvent the wheel, or if the wheel does not exist you will have to go through the modeling process yourself. The model will provide you with a composite metric which in turn will lead you to a set of base metrics. To define these two terms, composite and base metrics, a Composite Metric is a mathematical formula, the solution of which will provide you with quantitative value which in turn satisfies, through analysis, the initial requirement. Base Metrics are the items of raw data that need to be collected in order that the mathematical formula within the composite metrics can be applied.

If this sounds complicated the following may help.

Imagine that the product manager has asked you, as part of the metrics program, to supply information that will tell him or her how productive software development is within the revenue earning part of the organization. Obviously, the source data for this requirement will be drawn from more than one project or product area so we would seem to be talking about a Business Entity Metric.

Productivity is a relatively well understood economic measure that is widely used within most industries and it is generally defined to be:

Productivity = Work Product/Cost to Produce

To satisfy our current requirement this does require some additional definition. We may choose to define Work Product as "Function Points" or even as Lines of Code. In other words, the size of what we produce in terms of functionality or "volume" in the same sense as a car production line output measure may be defined as the number of cars produced during one shift.

Before anyone gets too upset at this let me stress that we need to be very careful in our definition of, say, lines of code, and the definition of other parameters in our model. We must also accept that productivity is not a magical, overall measure of process effectiveness. As you will have realized, we could produce lots of Lines of Code whose quality is lousy. However, productivity can be used as a high level measure of organizational performance and I will have more to say about how we derive valid measures later on.

Likewise, we could define our cost parameter as "effort" in terms of engineering days booked to a particular project. Now as this is a Business Entity Metric we could decide to sample a set of representative project/product teams or we could total across all teams. I will assume the latter case for this example.

We end up with a composite productivity metric defined as:

Productivity = (Total Project Lines of Code Delivered to Test and Integration during financial year x) / (Total project effort booked)

From this we can see that our base metrics, the raw data that must be collected to be able to derive our productivity metric, are:

Project Lines of Code delivered to Test and Integration
Project effort.

For now think about the terms used and the derivation method, not about the metric itself.

For the other common macro class of requirements, direct process improvement, the technique is slightly different. First we need to identify, at a conceptual level, what will be done, perhaps what we feel should be done. This is often based on examples taken from other organizations or on our own personal experience within the organization.

Conceptual models are a necessary step but they seldom provide the answer. The elements of the conceptual model need to be grafted onto the development process within the organization. In other words we need an Application model that will describe how we will do what is described within the conceptual model.

Again an example will help. Given a requirement to improve the effectiveness, i.e., the accuracy, of our cost estimates we may decide to use a proprietary cost estimation package. Conceptually we should realize that the package will need to be calibrated to our own environment before we can use it properly so our application model, based on that calibration, may say something like:

"At the point when requirements analysis has been completed, use the requirements specification to complete the estimation input form. Submit the form to the estimation team who will, in turn, supply you with a modified, package-based estimate and confidence level."

Again an oversimplified example but it illustrates that just deciding to use a cost estimation package is not enough. The application model addresses a great many issues that the conceptual model can ignore.

Of course, all of this assumes that you are able to obtain your requirements. This is not always as easy as it may seem so let us consider this area now in more detail.

Obtaining the set of specific requirements that will drive the Software Metrics program is not a task that can be easily defined because it involves interaction with people. What can be defined is the types of information that should be contained in such a requirement together with some approaches and techniques that can be used.

Essentially there are two techniques that can be used to define the specific requirements within the metrics program. The most obvious is to adopt a systems analysis approach by identifying specific, key players within the customer set. Structured interviews are then carried out involving these individuals.

My personal preference is a two pass approach. For each player, the first interview aims to identify their particular role within the organization as they see it; what processes they are personally involved in and what those processes entail; what responsibilities, in terms of deliverables, they have; and, most importantly, what they perceive to be the weaknesses in the processes they operate. This can be quite a tall order!

Do not minimize the importance of this activity. Talking to people takes time and to get the most out of the time you have available you will need to plan and structure your interview sessions. One plan which certainly appears to work goes as follows:

Start the session by briefly describing the purpose of the interview which is to ensure that the deliverables of the metrics project really address the personal needs of the interviewee and the business needs of the organization. Do not try to sell metrics too hard. You are there to listen and learn, not preach! By all means explain the high-level requirements of the program as this will help to set the focus of the interview session.
Ask the individual to explain his or her role within the organization in their own terms and be ready to spot "key terms" such as project, build or vagaries such as "size," "complexity" and "quality." You will see why this is important shortly. This part of the meeting is vitally important. You really need to listen to what is being said, and, perhaps more importantly, to what is not being said. Can you start to see areas that you think may benefit from measurement or from other metrics techniques you know of?
As the interview progresses, take notes. If you are unsure about what is being said, ask questions. Simple actions like this demonstrate interest on your part and provide a positive stroke to the other person's ego. Don't knock it! You need this person to work with you so you use everything you can to develop a rapport with him or her. One warning: a single question asked by you can drive the interview off at a tangent if you interrupt a persons flow of conversation so perhaps you should simply note a point and return to it later.
The interviewee will be involved in processes. As the interview progresses start to build a model of those processes on your note pad, perhaps using a Data Flow Diagram notation similar to that used in this book. Look for sub-processes that do not seem to be triggered or for information that is consumed but not used. Use points like these to drive the interview if it seems to be flagging.
Make sure that you have got it right. Play back your understanding of what has been said regarding the interviewees work. If you have got it wrong then do not argue, change your model. Remember, you are trying to find out how things are done. Do not try to impose text book solutions on your interviewee. If he says that they do not carry out any review of code designs then feel free to query it, but if that is the case then accept it.
As you talk, note the areas where there seems to be problems. If he is involved in estimation how do they do it? Can they supply information about the last project or product delivery in terms of how big a job it was, what the quality levels were, how effective the process was at getting rid of defects?
Now we get to the difficult part. You need to find out how Software Metrics can help the interviewee, so ask him. Now if you simply put the question you will probably be met with a blank look. You will need to steer this part of the interview by suggesting ideas to the interviewee. Would some assistance with estimation be appreciated, perhaps a centralized estimation group or some training in techniques? Are there some problems because of more senior managers asking for embarrassing information such as productivity levels—possibly embarrassing because the manager does not have the information? Do new tools need to be justified? Explore the possibilities with your partner, and I use the word deliberately because the interviewee should now start to feel part of the metrics project, a partner in it.
Finally, try to arrange another session with your new-found friend but also try to leave the date flexible. Why? The reason is that you will now have some serious work to do. First, you have to hold similar sessions with other individuals so that you can identify a macro set of requirements. You also need to check some of the points raised with other people. Is it really true that effort data is recorded accurately by all project team members as perhaps the manager believes or are the engineers making a wild stab at the figures at the end of each month? You will also find, almost certainly, that your first few interviewees raise even more fundamental questions that need to be answered. Be honest: if you do not have an answer then defer it by noting that it needs to be addressed and by telling the interviewee that various basic definitions and ground rules will be defined before the requirements are baselined. This will not be an alien concept to most people involved in software development.

Before we consider this last point, a few more words on initial requirements capture using this interview-based approach. How long should such an interview take? I seldom schedule more than two such sessions during a day even if I can only get the interviewee to agree to give me a hour. The reason is simply that people like to talk about themselves and it is amazing how quickly time passes in such a situation. In some cases, you may well find yourself sharing lunch and going on in the afternoon when you originally planned to finish by ten in the morning. Do not worry, this is time well spent.

Another point about scheduling. There is always a temptation to "start at the top." Don't! There are two reasons for my saying this. First, you will not be as well prepared as you should be when you do that first interview. In fact, it may take three or four sessions before you really start to get to grips with what you are doing so it is best to minimize any risk at this point. Second, the case when this relaxed, open format interview technique does not work is with the people with whom it would be most useful, senior managers. Senior managers, say the Systems Development manager, do not have the time to chew the fat with you, and even if they do they certainly do not want to seem as though they do.

Time with senior managers is difficult to get and is therefore valuable. To make the most of this time I can only suggest that you adopt the approach mentioned earlier with one modification: tell them what you think the problems are, what the options are and what you intend to do. The modification? Then ask them if they have any specific problem areas that they want addressed or if they want any of the identified action areas given a particularly high priority.

Of course, all senior managers have already identified the immediate business requirements and their priorities for the next five years as advised by all the management books. If you believe that, watch out for flying pigs! Even with senior managers you will need to control the interview but remember again, you are there to learn, not simply to make an impression; that comes later.

To be able to talk to senior managers sensibly using the approach outlined above means that you need to have an appreciation of the problems as perceived by the business, i.e., the very senior managers you will be talking to; you will need to have some idea what areas need to be addressed; and, you need to have a high-level plan even if this is then modified by the manager. Obviously, you cannot go and talk to the senior manager first before you have talked to anyone else.

One final point: do remember to consider business etiquette. Within your organization you may need to approach a senior manager before approaching any of his or her staff. From the previous stage, this permission may have been implied. It does no harm to politely get that permission again, explicitly.

Getting the requirements through individual interview is certainly one technique that can work effectively. There is another technique that can work more quickly in terms of elapsed time but, depending on the culture of the organization, it can involve a greater risk of getting it wrong, often because you do not get the key players involved.

Sometimes the approach is called brainstorming, or meta-planning or centers on a workshop. Essentially it involves a group of people simultaneously working out a set of requirements for the metrics program. It requires a fair amount of planning and coordination and also a strong meeting leader, who is more than a chairman, to control events as they occur.

On the positive side, it can give you a set of agreed requirements very quickly.

To use this approach within the type of organizational model I am using to illustrate the metrics program development and implementation lifecycle would mean that you need to prepare some material that could be used to introduce the topic of Software Metrics. This material needs to define what is meant by the term Software Metrics; it needs to introduce the principles of measurement and relate these to a software development environment; it should show examples of the effective use of Software Metrics; and, it should relate the information shown to the identified business needs of your own organization which, of course, form the basic, high level requirements of your own Software Metrics program. Avoid the use of mathematical formulae in this material as it can confuse the issue.

Once you have the material you will need to identify the key players that you intend to use to define your specific requirements. You will need to arrange some form of management sponsorship to ensure that you can get the people you want to the workshop. Ideally, the Systems Development director would agree to attend the meeting. You will have to plan out the structure of the session in some detail to ensure that you maintain control and get from it what you need.

On the day (and you will need at least a half day, ideally a whole day), be up front and honest. Explain that the purpose of the workshop is to derive a specific set of requirements that will be used to drive the metrics program. Present your introductory material in a strong way and be prepared to handle difficult questions from the participants in a realistic and sympathetic manner. Try not to be too defensive. After all you want their participation, so be prepared to modify your own ideas and concepts in the light of their contributions. Above all, do not come across as someone who claims to have the perfect answer to all their problems. First, they have heard it all before and are probably quite rightly cynical. Second, you do not have all the answers! What you do have is an approach that can help them, as managers of software development work, to help themselves.

If you have managed to get a full day, be very sure that you steer discussion towards the requirements definition by the start of the afternoon. It is very easy to get caught up in deep discussions about, perhaps, the true meaning of system size, and to find that it is five o'clock and you have half an hour left to sort out the requirements. If things start to get bogged down then try asking if they would like to try any of the approaches outlined in your examples within their own areas or you could even ask, 'what kind of information would you like that you don't get now?"

Prototyping is an approach I know has been used by one U.S.-based organization in the area of management information. The case was different from the usual situation in that the individual driving the program had a significant amount of experience in the application of Software Metrics, was well respected in the organization and had a strong supplier/client relationship with her customer. This approach worked for her and I could see it being very effective in phase two or three of a metrics initiative as a means of generating management commitment when you expand your program implementation. I have some doubts about applying it during the first stage when the metrics team will probably be learning about the topic themselves.

Whatever approach you use you should end up with a set of specific requirements that have come from the business and that the business wishes the Software Metrics program to address. Looking at these you may well find that they fall into a number of categories. First, you may see some that you feel you have a reasonable answer to and that will provide a quick payback for relatively little effort. You may also spot some that could be called research issues. Yes they are important but you know, from your earlier work, that there are no immediate answers just waiting to be put into place. You will also find some requirements that are obtuse. They are either very specialized or appear so convoluted that you could spend the next ten years just getting to grips with what is wanted. Avoid these like the plague.

In fact, you will almost certainly have too many requirements to address all at once so you will need to prioritize. One systematic approach to this is to look at the available data sources, together with the storage, analysis and feedback requirements. Address the areas where data is available already and where the storage, analysis and feedback requirements are clear first. How to identify what data is available is discussed shortly.

Very often, the Metrics Coordination Group are an ideal forum for determining priorities in terms of requirements to be addressed.

One other thing that you may well discover is that you need to establish some initial definitions.

< Day Day Up >