The Eclectic Approach to Integration and Testing: Balancing Art and Engineering

When I examined the methods that I use that work well when organizing and carrying out a test effort, I found that I was relying on communications skills that I learned as a dancer and ballet mistress rather than the formal approach that I learned as an engineer. When I am planning and reporting, I use tools and methods from engineering.

Dance is not communicated through written documents but by an older form of communication; dance is communicated by an oral tradition. This contrasts sharply with engineers and musicians, who communicate via written documents, but it correlates closely with the current requirements of software development. I mention musicians because there are a significant number of musicians in software development.

An oral tradition has the advantage that there is no time lost in recording instructions on another medium, and the disadvantage that there is no permanent record of what transpired and everyone must remain in close proximity to be kept informed. Large projects cannot be controlled using oral communications alone; neither can projects with team members who are spread out in disparate locations. Recording instructions and design specifics in writing takes time, but it allows ideas to be communicated to many people in remote locations.

Dancers absorb and memorize new material through a combination of oral, aural, and visual input. Virtually every movement that the dancer makes is set by the choreographer. Dancers rarely perform ad lib. Each movement is supposed to be performed on a specific beat in some precise location and spatial orientation on the stage. This is doubly true for members of the corpse de ballet, who must perform as a single unit, with each individual dancer precisely in synch with all the others. The dancers must execute complex coordinated movements precisely, time after time, on the beat, and in the correct place on the stage, down to the tilt of the head and the angle and alignment of their fingers. It is within this very demanding and precise framework that a dancer applies his or her art to make that particular rendition of the dance exceptional and memorable. A group of testers with these characteristics is definitely in a position to do an excellent job of testing a system.

Dancers are competent professionals when they are hired. When a new production is scheduled, everyone is given a part, usually one that is well suited to her or his abilities. The main difference between the ballet and the software business is that in ballet, everyone expects to be taught the part that they will perform in each new production. In business, the expectation is that somehow everyone will know his or her part. My friend Bill says, "Everyone wants to learn, but no one wants to be taught." In business this seems to be true. But it doesn't solve the problem of how to transfer knowledge to new players in a new production and get them working together.

I have had success coping with these difficulties in business by building teams. In a team environment, every member's work is visible; if someone is not keeping up, the situation can be examined and solutions can be proposed. The biggest problems arise when individuals become isolated and their work has low visibility. Today, these precepts are considered an integral part of managing a collaborative development effort.

The ballet mistress or master is responsible for rehearsing the company, spotting and correcting any problems, and tuning the whole company to give their best performance. If some part of the production is not working, it is the ballet mistress or master who takes the issue to the choreographer. To be able to do this job, the ballet mistress or master must attend every single choreography session and rehearsal. If any written notes exist for a production, it is usually the ballet mistress or master who creates and maintains them. She or he must be able to count every phrase of the music for the production from memory, preferably while singing the main themes. She or he must know, at least in general, each dancer's part and all their entrances and exits. In a pinch she or he must be able to dance the parts of dancers who are not present so the rest of the company is not disrupted by the absence. This may mean filling in for a chorus member or a soloist.

The ballet mistress or master must be able to spot and correct mistakes and be ready to arbitrate or break the tie in any disagreements among the dancers, like, "We are supposed to be in the air on seven." "No, we are supposed to be on the ground on seven." She or he must also be able to make compelling arguments to nondancers like lighting engineers and the musicians. For instance, the ballet mistress may need to persuade the conductor that the fouetté music must be played at a slower tempo so the prima ballerina does not fall down or spin off the stage like a runaway top. By the way, this is a validation on the part of the ballet mistress; it is based on her judgment. Verification would require that the argument be tested to verify what actually happened to the ballerina turning at the questionable tempo.

The ballet mistress or master is primarily a technician, not a manager; however, she or he must be able to manage the various egos and temperaments in the company to keep everyone happy-that is, ready, willing, and able to give a splendid performance.

The company takes correction from the ballet mistress or master for two reasons. The first is that only dancers with a long and broad experience base are chosen to perform this very demanding role, and their corrections are virtually guaranteed to improve the quality of the performance. The second is that the ruling of the ballet mistress or master can only be overturned by the choreographer.

So, while the experts, like the soloists, focus on giving the best performance possible in their part of the production, the ballet mistress or master is the integrator responsible for testing and fine-tuning the production as a whole. All these players, working together, are necessary to ensure a truly excellent performance.

We are human beings. Our endeavors include these roles because they are effective in managing human undertakings, particularly ones that require cooperation-teamwork. Each discipline has a key person who performs a role similar to the ballet mistress. In the symphony it is the conductor, in singing it is the choral director, in acting it is the director, and in sports it is the coach.

In safety-critical software, the testers are generally the most experienced people on a project, and their rulings are rarely challenged or overturned. It is not hard to imagine what happens to the entire process if this integrator's role in the test process does not exist, is filled by someone who is poorly informed, or is filled by someone who doesn't have a technical background.

Many case studies show that the software testers should be involved in the software design and development phases, yet the industry continues to treat software testing as an afterthought, and testers as unskilled temporary positions that can be filled by persons with no special training.

In keeping with my role as integrator, I go to great lengths to make sure I am invited to the design meetings and brainstorming sessions whether it is part of the plan or not. If that fails, I rely on the friends that I make among the designers and programmers to keep me informed. If all of these things fail, most especially if I am forbidden access to the developers, I cannot possibly succeed. When I do contract testing, I have clauses that allow my company to withdraw from the project in this situation. Even if I cannot withdraw, I am duty bound to report to my management, in writing, that the situation is untenable and that I cannot perform my job in a satisfactory manner.

Engineers communicate in writing. Except for clarifications, the field engineer gets all of her or his knowledge of the project and instructions from the plans. This means that the plans must be well thought out, precise, and up-to-date. When a situation arises that is not covered in the plans, the field engineer prescribes a solution, generally after consulting with a colleague and getting a second opinion to back up her or his own. Since I have already discussed the problems associated with trying to create and maintain paper documentation in the fast-paced software development industry, it should be clear that the engineering methods applied to software development often break down at this point. Workers do what the field engineer says because the engineer is the authority in charge. The field engineer's decision can generally only be overturned by direct challenge and an inquiry by other engineers at the behest of the owner or general contractor. Even if the field engineer's decision is demonstrated to be wrong, it may not be easy to overturn it.

Since the paper documents in the software development effort are rarely able to keep up with the actual developments, the integrator who tracks the evolution of the project firsthand can be more effective in managing a software test effort than the engineer relying on the written documentation. This advantage that the integrator has via word of mouth should diminish as we focus on implementing self-documenting strategies in electronic media to replace transcribed paper documentation, but these skills are always beneficial.

Traditionally, engineers are far removed from human factors concerns. A lack of concern for or understanding of the needs and priorities of people is a serious disadvantage for anyone trying to manage people today. The company loyalty that used to allow managers and engineers to rule by decree has been severely eroded. Recent studies have shown that the employees are more likely to feel loyal to their current project than to their company.

Over- and underplanning is one of the hottest issues being debated in the software journals in 2002. The Capability Maturity Model disciples require extensive planning in the best engineering tradition; meanwhile, the Agile developers prefer to produce and test small bits in immediate response to customer requirement or evolving thinking. In the theatre this is the difference between a Shakespearean play and an evening of improvisation. Like it or not, they both have their place.

In engineering, the finished project is rarely better than the plans. In art, the project owes little allegiance to a plan; it improves and evolves continuously. It is never finished, only released.

Note 

Two points for engineers in software to keep in mind:

1. You need to pick the right tool method or approach for the job at hand.

2. You need to remain flexible and plan for change.

Engineering provides methods for presenting and winning arguments through fact and measurement that are far superior to those used in the arts. The arts use the critique to make people aware of problems. Criticism is typically subjective and personal, as in "You made a mistake." Criticism is not easy to accept, especially for those not raised in a discipline where criticism is commonly used. The use of criticism leads to adversarial situations. Good engineering uses objective measurement and fact to make corrections. Facts and measurement are impersonal and much easier to accept, as in "There is a mistake that we must fix."

It has been said, "measure to discover," to which we should add, "not to lay blame unnecessarily." Testers who use engineering arguments, fact, and measurement are in a better position to be effective in assessing the actual importance of issues and win their points while avoiding adversarial encounters.

Sometimes neither the classical ballet mistress integrator nor the engineering approach is acceptable. Artists who are in the throes of the creative process with little or no supervision are not necessarily overjoyed to have an alternate authority introduced into their project-especially if they have succeeded in discrediting or dismantling the in-house test group.

An interesting outgrowth of this situation is the rise in popularity of the external review. An external review takes place when someone outside the project, or outside the company, tests the system and writes a bug report called a review. The review is submitted to management. Those writing the external review are disposable bad guys. Issues listed in the review are generally taken seriously by management because the source is external and uninvolved, and the arguments presented are generally unbiased. Many of the same issues may well have been logged by the in-house test group and denied by development.

Testers do not have to be experts in a system to test it well. They must be trained testers, using a systematic approach, sound reasoning, and measurements to make their case. They must also be well informed about the project and have good channels of communication to the experts. People skills are also a definite plus.

Like it or not, good testing also includes exploration. Good testers are the ones that dig in and explore the system. They want to know how it works. The ones that don't dig in aren't going to get the job done, and the ones that think they know how it works are no good either, because they are closed and biased.

The Top-Down Approach versus the Bottom-Up Approach

The approach to testing that was entrenched in the industry in 1987 when I began testing was the bottom-up approach. Basically the bottom-up approach to testing proceeds as follows: Each module or component is first tested alone; this is called unit testing. Next, the modules are combined a few at a time and tested. Simulators are used in place of components that are necessary but missing. More modules are added when the existing ones are stable until the entire system has been assembled. This very cautious approach is also from engineering; it is rigorous and thorough, but it is also very slow. It has one major drawback: Testers are testing the simulators, not the real system. I haven't seen a real system yet that behaved exactly like the simulator.

The bottom-up approach is a rigorous testing approach that comes from engineering. When applied to today's commercial development environment, bottom-up testing is like teaching each performing group their parts, then bringing them together, a few at a time, to rehearse. It is a very long and laborious process to assemble the entire cast, because the group dynamics change each time new cast members are added, creating a new set of issues at each step, none of which is relevant to the finished product. Such rehearsals are only of value if it is important to know in advance what the system behavior is likely to be if some of the members are not functioning. This type of testing may lead to a test effort that contains a huge amount of unproductive redundancy. However, sometimes it is the only way to accomplish the goal.

Top-down testing is like teaching each cast member their part individually, then getting as much of the system as possible together for a rehearsal as soon as possible. In the top-down approach to testing, each module is first unit-tested, then all the available modules are assembled,^[1] and the entire group is tested as a system from the highest possible point, usually the user interface, with as few simulators as possible.

In the top-down approach, testers begin with the integration phase. This requires that the code has been unit-tested before it is delivered to the testers. If the unit testing has not been successfully completed or if the units lack integrity, top-down testing will not succeed. If the new system is too unstable to test, the best that the tester can do is subdivide it and try again. Testing the whole system will not succeed if most of the test time is spent diagnosing buggy units.

^[1]This strategy for building a system from modules as they became available is also called incremental delivery.