Customer Tests

Lets reflect further on the customer tests. I see these key lessons here:

First, the initial customer test was difficult to do. We didnt want to do it, we resisted it, and we had to get a bit creative to figure it out. Once we made the commitment to have the test, however, it took only a couple of hours to do it. This is almost always the way, in my experience. We think we dont need the test, we dont see how do to it, and then it isnt hard to do.

Second, the tests served us well, but not perfectly . They were pretty easy to write, but because they didnt give us good debugging information, we would sometimes skip them when we shouldnt have. So the customer test suite is not as strong as it should be. Of course, if we had a true independent customer, they would be focused on making those tests strong to be sure that the product was working as they wanted. If the customer tests had been stronger, they would have turned up more problems. Had they turned up more problems, we would have had the incentive to improve our ability to get useful information out of them.

Finally, as I learn anew on every project, end to end is further than we think it is. The most effective customer tests test the system from one edge to the other. In the case of the XML Notepad, that means from keyboard to display. We encountered a few cases where the tests ran but the GUI did not. We improved the tests in response to most of those, but my suspicion is that the tests are still a bit weak in some undiscovered areas. End to end is always further than we think it is.