Problems with Unit Testing

There are some known issues with unit testing that can affect its applicability to certain types of projects. In particular, unit testing is not well suited to multithreaded systems or systems that use asynchronous messaging (both system types are increasingly commonplace ”almost all modern GUI applications use multithreaded toolkits [e.g., Java Swing] and enterprise platforms such as J2EE make use of asynchronous messaging).

Tests for Asynchronous Messaging and Multithreaded Systems

There certainly are programming tasks that can t be driven solely by tests (or at least, not yet). Security software and concurrency, for example, are two topics where TDD is insufficient to mechanically demonstrate that the goals of the software have been met. ^[7]

One of the problems, for example, with the JUnit testing framework is that the test method needs to wait for a synchronous response so that it can do its assertEquals() on the result. You could get around this by maybe adding a callback method that waits for a preset time and then times out, but this is not ideal (not least because it could require modification of the code being tested ).

These days, most GUI programs are multithreaded to improve performance and usability. For example, in Java Swing, a dedicated thread (the event dispatch thread) handles processing of GUI events (such as notifying listeners when the user clicks a button or scrolls a window). If the listener code needs to perform a time-consuming operation (e.g., connect to a remote FTP server), it should create a separate thread to do this so that the GUI does not appear to lock up.

We provide a case study of an agile project targeting Swing in Chapter 15.

 

Multithreading is essential in GUI applications to keep the user experience a positive one. Does this then mean that GUI applications cannot be unit tested? Luckily, the problem is not insurmountable. Aggressive use of the Model-View-Controller (MVC) design pattern, for example, means that all the program logic can be kept out of the GUI- related code. The business logic can then be unit-tested independently of the GUI (i.e., the unit tests call the business logic classes directly, bypassing the GUI code).

You can use a similar testing method for non-GUI asynchronous code by moving as much of the logic as possible into synchronous methods that can be invoked directly from your unit tests. Inevitably, however, some code must remain asynchronous (e.g., event listeners).

The situation is improving as more community understanding is gained regarding how best to unit-test asynchronous systems (for example, the book Java Extreme Programming Cookbook ^[8] provides some useful tips on the subject) ”although asynchronous and multithreaded systems by their very nature will always be problematic . Therefore, refactoring of such systems ends up being more expensive and is more likely to introduce subtle concurrency-related bugs .

In fact, this category of system very much lends itself to detailed up-front design. Multitasking, concurrent transactions, and so on can be analyzed with design models such as state diagrams. Even with diagrams, it can still be difficult to prove that all the cases have been covered, so it is vital to follow a suitably thorough logical design process that does catch all the cases (or as many of them as possible) to minimize the amount of refactoring that will need to be done later.

VoXP
Voice of eXPerience: Unit Testing

by Robin Sharp

This short description from Robin of unit testing gone wrong is a sign of what can happen to an XP codebase that isn t being refactored. Although emergent design (without lots of up-front design) isn t exactly our favored approach, it s safe to say that if a team does use emergent design, team members should refactor ruthlessly as well. If they don t, then Fangs (the slippery -slope serpent ) strikes again. In this particular project, the managers were drawn to XP because they thought that not doing design up front would save them time and money.

The software was too complex to test fully. Our test suite took hours to run. In fact, the tests took so long (about 8 hours) that we often didn t have time to run them all. One of the issues was that tests are supposed to be self-standing. Tests often left the databases dirty. The side effects often caused errors in the code that we had no idea about. So we had to keep rebuilding the database. Continuous GUI testing is also very problematic, and you have to use a tool like WinRunner.

More from Robin in Chapter 9.

 

 

Other Problems

Some other downsides of unit tests are as follows :

• Unit tests catch only the bugs that you have anticipated. In XP, programmers are responsible for writing their own tests (which is a good thing), but there will inevitably be errors of omission.

• Not all code can be unit tested (at least not easily). It could, of course, be argued that in this case, the code should be simplified until it can be unit tested. As we mentioned in Chapter 3, some teams reach homeostasis, where they swing from doing too few tests to doing too many (and back again) until they reach the optimum level.

• Writing unit tests for every single class to test every aspect of every function is a tall order. By following XP to the letter, you ll likely end up writing one to two times as much test code as functional code. As the late, great Groucho Marx said (while interviewing a woman who had something like 15 kids ), I like my cigar, too, but I take it out of my mouth once in a while!

• What if there are bugs in the unit tests? Where there is code, there will be bugs. Usually, this turns out not to be a problem ”unless you re using a process that relies on 100% reliable automated tests. Luckily, XP has the answer:

  BugsInTheTests don t matter that much. ^[9]

• Unit tests are part of the codebase, so when the team refactors, it must refactor the test code too.

• Refactoring occasionally makes some tests obsolete. The team must make an effort to delete or update obsolete tests as the design changes.

• Even with such a large number of unit tests, is complete testing possible?

  If the objective of testing is to prove that a program is free of bugs, then not only would testing be practically impossible, but it would also be theoretically impossible . ^[10]

  This isn t an argument against unit tests, which are a valuable tool in the quest for the Holy Grail of complete testing. The problem is that in a software process such as XP, which embraces change, it s supposedly okay to go ahead and make changes to the system, because the unit tests will catch any resultant bugs. The tests are useful as a safety net, but they aren t sufficient to form an intrinsic part of the process ”in particular to remove the risks associated with Constant Refactoring After Programming. Unfortunately, the illusion exists that running, and rerunning, and rerunning the tests (over and over and over and over again) does remove those risks.

As we discussed earlier, test-first design isn t a replacement for up-front design. A major problem with applying unit testing to a tightly-wound process such as XP is that unit tests don t catch coding errors at a sufficient level of abstraction.

Unit Tests Are Easy “the Customer Gets to Write Those Nasty Acceptance Tests

One of the problems with test-first design (as a replacement for up-front design rather than a useful supplement) is that it doesn t make a concerted effort to anticipate error-prone usage paths. Unit tests are the easy part. Writing a unit test for a method is generally trivial. It passes or fails.

The nasty part of testing, where you have to deal with all the alternative courses of action (exceptional behavior, recovery, security, etc.) is really all defined in the acceptance tests. And guess what ”that s the customer s problem in XP. So Kent Beck s social contract has been changed in a way that the programmers loudly claim to be test driven, but in fact the hard part of testing has been delegated to the customer.

For more about Beck s plan to change the social contract of working, see the section That s the Customer s Problem in Chapter 5.

 

VoXP
Voice of eXPerience: Aspects of Testing

by David Van Der Klauw

More insight from David Van Der Klauw. Here he describes the pitfalls of turning the unit-test dial all the way up to ten. Software testing is an art and a science in itself. Software is sometimes easy to write but often hard to test and impossible to prove. What to do?

XP solves the problem of testing by simply insisting that everything be tested and with a test written before the code. If only life were so simple.

While acknowledging that testing is very important and something that CompanyXYZ [David s anonymous employer] has neglected, I believe that XP oversimplifies the difficulty of testing and misallocates testing resources.

Although test-first coding is an interesting concept, I believe that the reality is different from the XP theory. My experience is that many things that can be testfirst coded are so simple that a test is unnecessary. On the other hand, many things of moderate or greater complexity cannot be test-first coded. Only after the coding is finished is it possible to envisage the tests. I am not hard-line about this, and I may change my opinion over time.

As to degree of testing, I think XP makes the dangerous simplification of applying the same degree of testing to all areas. My concept of bug finding is something like this: A programmer should find 9/10 bugs before check-in, the automatic tests should find 9/10 remaining bugs before handover to QA, QA should find 9/10 remaining bugs to customers who find 9/10, and 1/10 remains undiscovered. Therefore, the customer sees only 1 in 1,000 bugs.

My belief is that this is good enough quality. Sure, it would be nice if the customer saw zero bugs, but is the customer prepared to wait years longer or pay twice as much for this level of quality? Probably not.

I understand that some people have gone overboard with the tests that they write. Tests have been written for some things (like constructors) that can never fail. Some testing of module internals has been so detailed and extensive that it greatly increases the cost of ever changing the internals of the modules in the future. This is not agile programming.

Rather than obsess about XP s testing simplifications , I suggest to instead do a lot of thinking about how best to test. Spend a reasonable percentage of total time on testing, and test as much as possible for the given time and resources. When a customer finds a problem, add new tests so that no customer will ever see that problem again.

Test what the customer sees first. As those tests fail, add more detailed internal-tests to pick up faults that are actually occurring.

 

VoXP
Voice of eXPerience: YAGNI 16 Tests

by David Van Der Klauw

One inevitable result of combining the two XP practices of Design After First Testing and Constant Refactoring After Programming ^[11] is that you can spend a lot of time writing tests for code that s subsequently thrown away. So you can spend time coding lots of tests that you don t need. It s not analysis paralysis, and some folks might even regard it as fun, but it s certainly an insidious way to pour programmer-hours down the drain.

This never happens on a real XP project, right? Well, David details his real-life experiences with this phenomenon in the following Voice of eXPerience segment, and then we turn the satire dial all the way up to ten with a piece that we call Camp Regretestskiy.

Our unit tests were made up of two functions per test. The first function called the second and outputted the name , description, and pass/fail result. The second function simply returned true or false.

The task of our pair was to add unit testing for a new menu condition that would affect eight menu items. My partner, a junior, claimed he knew the code well and jumped straight to some testing code with eight blocks of code like this:

 Function ShowTestMenuItemA  TestOutput(TestMenuItemA, description A, TestMenuItemA())  End 
 Function TestMenuItemA()  Return(GetMenuItem(ItemA).visible=true)  End 
 
 

I didn t like it and said that all eight checks should be done in the one function, but I was told that this was the simplest thing possible.

Now our task was to do the simplest thing possible, so my partner duplicated the 8 tests, edited all 16 to cover the new condition, green-screened it, and checked it in. Our highlight at the next stand-up meeting was that we have added eight new tests. Everyone clapped, but I was not impressed.

Later that week someone discovered that the function tested by our 16 tests was not actually called by the UI in determining menu item visibility. Therefore, the highlight at that stand-up meeting was that they were able to YAGNI 16 tests. Everyone clapped.

Somehow two XP highlights were a lowlight to me.

 

SATIRE WARNING
Camp Regretestskiy

Uncle Joe Steele sat at his desk and looked over his to-do list (which was, of course, written on a red index card). Rewrite history of C3, da! said Uncle Joe. C3 was glorious victory of the people. This has now been firmly established in books, magazines, and on newsgroups. Project was glorious success.

Next, buy borscht. I ll do it on the way home. Next, deal with big backlog of regression tests. Uncle Joe looked down at his code hound Jack, who was curled up in a corner of the office and said, This is a big problem, Jack, and we need to do something about it. All these dewelopers Constantly Refactoring After Programming and continuously integrating are changing our codebase so quickly that 8 hours is not enough time to run all of our tests, and the problem, of course, is made much worse when the dewelopers don t pair up, because there is twice as much new code to test.

Arf! agreed Jack. Uncle Joe pressed a button on his phone and summoned his assistant, Boris Codunov. Boris, a short man with a mustache who was wearing a black suit, walked in and said, Yes, Fearless Leader, how can I, your humble servant, subjugate myself to the interests of the project today?

Codunov, we have a problem, said Uncle Joe. Our glorious programming team has achieved such an incredible velocity in refactoring the codebase that we can t run our tests quickly enough. And, you know, some of these dewelopers just refuse to pair-program. This makes the problem even worse, as with pair programming there are only half as many streams of changes to integrate. As you know, Jack has been catching more and more dewelopers coding alone instead of pairing recently. But I have devised a brilliant solution that will solve both problems at once!

You see, Codunov, these disloyal dewelopers need to be taught a lesson. So, we will establish a gulag for regression testing. I have decided to call it Camp Regretestskiy. It will be located in Hackensnack, New Jersey. And you, Codunov, are going to run it for me, said Uncle Joe.

Hoo-boy, Boris said. I am honored to be in charge of Camp Regretestskiy. I ll get to work on it right away, Fearless Leader. Boris left the office, stopping to give Jack a pat on the head.

A few days later, JoJo and Loretta were refactoring the state tax computation for the thirty-fifth time when JoJo suddenly started sneezing. Achoo! Achoo! sneezed JoJo over and over again. Bless you, said Loretta after he stopped . Are you all right?

I m sorry, Loretta, said JoJo, but I seem to have picked up a nasty cold. I woke up this morning with a terrible hacking cough, and I ve been sneezing all day. I would ve stayed home but I didn t want you to get in trouble with Uncle Joe for coding alone. But I really don t want you to get sick too, so I think I d better go home early.

That s all right, JoJo, said Loretta, secretly relieved. Why don t you go home and get some rest? You can always refactor the tax computation again next week when you feel better.

Thanks, Loretta, said JoJo, packing up his briefcase.

Loretta, who liked JoJo but secretly preferred coding alone, got out a spray bottle of disinfectant and began to spray the keyboard and mouse that JoJo had been using. Even though she had been taking vitamin C++ every 15 minutes since JoJo arrived that morning, she could already begin to feel her eyes itch and water.

I d like to shoot the person that published the study claiming pair programming increased productivity, muttered Loretta under her breath . They forgot the chapter on ˜Pair Catching a Cold ”how s that for a productivity increase? Not only that, but I had this tax computation done a week ago and it passed all the tests 100%, but I seem to be the only one who minds refactoring around in a circle. Luckily, I kept a printout of my sequence diagram from last week. She wiped off the keyboard and settled down to start work.

A few minutes later in Uncle Joe s office, Jack woke up from his nap, barked softly, and lazily chased his tail in a circle three times. Uncle Joe looked up from his e-mail. Da, he said. Good boy, Jack. Let s go check if any of our dewelopers are being disloyal. I just got an e-mail from Codunov, and we ll have quite a surprise for them, won t we Jack?

Arf, said Jack as he wagged his tail. The coach and his code hound walked down the hall to the coding room.

Just as Jack was waking up from his nap, there was a loud argument underway in the coding room. I don t care what Uncle Joe says about coding alone, said Ralph. It says right here in this book that ˜there must be food is a core XP principle. And I . . . am turning the dial all the way up to 20 on this one!

From the size of your belt, Ralph, it looks like the dial is a bit past 20 ”maybe about 54, said Alice.

One of these days, Alice . . . Pow! Right in the kisser! Now I m going to the store to get donuts , and don t try to stop me! exclaimed Ralph.

Only the door can stop you, Ralph ”when you can t fit through it anymore! replied Alice.

Achoo! sneezed Loretta. Bless you, said Ralph over his shoulder as he stormed out of the coding room. Better take more vitamin C++, Loretta, said Alice as she settled down to continue refactoring the social security withholding calculation.

Just as the elevator doors closed behind Ralph, Uncle Joe and Jack came down the hall toward the coding room. As Jack walked into the room and saw both Loretta and Alice coding alone, the fur under his collar began to stand up. He bared his teeth and started growling and barking. Jack went over to Loretta s desk, grabbed the sequence diagram she was using with his teeth, and then tore it to shreds.

Help! cried Loretta. Please, Uncle Joe, make him stop! Y*b**nna mat! You have been a disloyal deweloper for the final time! Not only are you guilty of coding alone, but you have forgotten the most important rule that we have in this organization. And that is, Never Kode Vit Documentation! N! K! V! D! yelled Uncle Joe.

Five large men in red T-shirts with the letters NKVD emblazoned on the chest ran in. You called for us, Fearless Leader? asked Captain Medvedenko, who was an imposing 6 10 _ tall and weighed about 250 pounds .

Da, said Uncle Joe. Take this disloyal deweloper off to Codunov at Camp Regretestskiy!

Suddenly, Alice, who had been sitting in her chair terrified, sneezed, causing Jack to run over to her desk and start growling ferociously. Ralph, Ralph! barked Jack. Alice began to cry.

Uncle Joe turned around and looked at Alice and Ralph s empty chair next to her. Alice has been disloyal as well! Haven t I told you people that coding alone produces twice as much code and twice as many streams to integrate? Now you ll see what the testing backlog you create looks like! Take her too! At once!

Da, said Captain Medvedenko, and he and his squad bundled Loretta and Alice off to Hackensnack.

A few hours later, Loretta and Alice, accompanied by the enormous , hulking Captain Medvedenko, arrived in Hackensnack at the gates of Camp Regretestskiy. Loretta and Alice (who, being at the next desk, had also caught JoJo s cold) were both sneezing profusely.

I bring you new laborers, Comrade Codunov, Medvedenko said to Boris, who was accompanied by his girlfriend, Natasha. Boris handed Medvedenko a set of handcuffs.

Thank you, Captain, said Boris. Please chain their right wrists together for me, then we ll take them down to start coding up tests.

But Boris, darling, how are they going to code tests with their right hands chained together? asked Natasha.

Boris replied, Uncle Joe wants everybody at Camp R to learn to work in pairs . . . and don t forget, Natasha, we are leftists here at Camp R. We do everything with the left hand!

Medvedenko handcuffed poor Alice and Loretta s right wrists, and, accompanied by Boris and Natasha, walked them down several flights of stairs to a desk in a damp, dimly lit basement room with no windows .

Achoo! sneezed Alice, as they were walking down the stairs. The group approached a desk with a computer and a pile of index cards on top of it and two chairs in front of it.

Here you go! said Boris, with a grin. The story cards are waiting for you. Sit down and start coding those tests.

Alice and Loretta struggled to sit in the chairs and still reach the keyboard and mouse with their left hands. Achoo! sneezed Alice. Can you reach the mouse, Loretta?

I can type one-handed, but I can t type and move the mouse at the same time, said Loretta.

If I twist around like this . . . Achoo! I think I can just grab it, replied Alice. Boris, you are so evil, grinned Natasha, as she watched Loretta and Alice crane their necks trying to see the monitor.

You betcha, replied Boris. Uncle Joe didn t pick just anybody to run his gulag, you know. Boris and Natasha both broke into peals of gleeful laughter .

Slowly and painfully, Loretta and Alice began to write unit tests for the pile of stories. Just as they began to make a small dent in the pile, three guards wearing NKVD T-shirts walked in carrying big baskets full of index cards. They walked over to Loretta and Alice s desk and began dumping cards onto the pile.

Stop! screamed Alice. We ll never be able to code that many tests!

Comrade Nikita, the leader of the group, responded, We will bury you in index cards! He then took off his shoe and banged it on the table for emphasis. Alice and Loretta both began to cry.

As the months passed, Alice and Loretta, working left-handed , continued to churn out the unit tests. Slowly but surely, the stack of unit test printouts on their desk grew ever taller. Poor Alice and Loretta became thinner and thinner, due to the NKVD s you hack, we snack policy. They worked 18- hour shifts and longed for the days when they got to go home clean at 5:00 PM every day.

Finally, one day Boris and Natasha came to visit them. Well, disloyal dewelopers,-have you learned your lesson about coding alone? asked Boris.

Oh, yes , Mr. Codunov, said Loretta and Alice in unison . We promise we ll never think about coding alone again if you ll just send us back to Uncle Joe, they begged.

Boris looked at Natasha, then got out a key and removed the handcuffs from Alice and Loretta s right wrists. Call the fire department, Natasha, he said.

But Boris, darling, there isn t any fire, said Natasha. Just do it, Natasha ”there will be, grinned Boris, as he looked at the enormous pile of unit test printouts stacked up on the desk.

A few minutes later, Comrade Andrei Griffitsky of the Regretestskiy Fire Department (RFD) walked in, whistling a happy tune and carrying a flamethrower. Griffitsky was accompanied by a small group of men wearing RFD shirts. You sent for me, Comrade Codunov? asked Griffitsky.

Da! said Boris. Uncle Joe called me this morning and said that the customer has changed his mind about all the stories, so it s YAGNI for all these tests. You know what to do!

The RFD squad began to chant, YAGNI all the tests. YAGNI all the tests. YAGNI all the tests. YAGNI all the tests.

Griffitsky stopped whistling, grinned widely, took out his flamethrower, aimed it at the stack of unit test printouts, and incinerated them in a massive burst of flames. The RFD squad continued to chant, YAGNI all the tests. YAGNI all the tests. YAGNI all the tests. YAGNI all the tests.

This new flamethrower works even better than using a Molotov cocktail! said Griffitsky.

Loretta and Alice broke down and began to cry as they watched the results of all their left-handed unit-test coding go up in smoke.

You see, Natasha, they have learned their lesson. They re ready to go back to Uncle Joe, said Boris. And he handed the women back to Captain Medvedenko for the trip back home.

 

^[7] Kent Beck, Test-Driven Development: By Example (New York, NY: Addison-Wesley, 2002), p. xii.

^[8] Eric M. Burke and Brian M. Coyner, Java Extreme Programming Cookbook (Sebastopol, CA: O Reilly & Associates, 2003). This is one of the better XP books. It contains lots of practical, hands-on advice that s useful even for non-XPers.

^[9] Ron Jeffries posting to the C2 Wiki page Bugs In The Tests, http://c2.com/cgi/wiki?BugsInTheTests .

^[10] Boris Beizer, Software System Testing and Quality Assurance (International Thomson Publishing, 1984), p. 12.

^[11] See the section Emperor s New Code in the preface for the proper acronyms for these practices.