Centuries ago, the critical constraining resource was land. Those who controlled the land controlled everything. At some point, the constraint became skills, and guilds and merchants created more wealth than did landowners. Later, the industrial revolution moved the constraint to capital, and power became financially driven. Today, the constraint is knowledge: technical knowledge, management knowledge, process knowledge, and market knowledge. Much of this knowledge is being expressed as software. Most of our current paradigms remain rooted in industrial/financial thinking and measurements. Those organizations that will dominate this century are those that shift to a focus on knowledge. Rally[1]
"We set out to create a tool for managing agile software development," said Ryan Martens, Chief Technical Officer of Rally Software Development.[2] "But much of our team's experience was driven by waterfall success, and we found ourselves doing waterfall in time-boxed increments." Ryan knew from firsthand experience that software companies could be quick and responsive at the beginning of their life, but after a while the code base always grew (that's the point, after all) and got increasingly difficult to change. He understood that agile development was supposed to be an antidote for these growing pains. What he hadn't quite gotten his arms around was that code is tolerant to change in the exact proportion that it is insulated from defects.
Anything that makes code difficult to change is technical debt. You can pay full price for code while you build it, or you can go into technical debt. Be warned, however, that technical debt extracts a very high interest rate, and the longer you are in debt the harder it will be to pay off. Ryan discovered that technical debt was Rally's biggest constraint: "We found that the only way to get ahead is to have a team, process, technical infrastructure, and guiding ideas that never let debt accumulate." "During the first year we had a lot of technical debt." Ryan says. "We released software in eight or ten week cycles, and two weeks of the cycle was spent 'hardening' the software. [Hardening is another way of saying "test-andfix."] We used JUnit for unit tests, of course, but a large part of our product is the user interface [UI]. We used HTTPUnit for testing the UI, but we found that it was not capable of testing page flows. Because of this, a lot of UI testing, along with all our acceptance testing, was manual. We had maybe 300 or 400 manual tests, and we could never run them all. Our technical debt increased every release. The testing load was the killer, and it just kept going up. "Early in the second year, we set out to tackle the root cause of the debt: Our page flow platform did not let us get inside and test it with application-level unit tests. At the same time, we generated statistics which showed that our page flow structure was wasting users' time with too many clicks. So we decided to move our page flow platform to Spring and AJAX and unit test the page flows with Fitnesse.[3] If we could use this architecture going forward, it would at least stop the manual testing debt from accumulating.
"We tried the new architecture for some new and relatively experimental pages to learn how it really worked. The few customers who had asked for features changes in this part of the application were thrilled and tolerated the occasional problem as we learned. After two releases of isolated parts of the application, we were confident enough to gradually retrofit our old pages with the new architecture. As an industry, we have been trained to think of new releases as driven by big bang architectural changes. Learning to think in small incremental and iterative chunks is a critical guiding idea of agile. We learned that incremental architectural changes work very well; you don't needor want the risk ofbig-bang architecture. "Spring proved to be easy to test. AJAX, however, needs direct execution through the browser. We brought in JIFFIE, which binds Java to Internet Explorer, so we could write unit tests in Java that test AJAX through the browser. Hardening went down from two weeks to one. But (there is always a but) we found that only a few test engineers had the background to write the new tests, and we were also accumulating automation debt. Automated testing became a bottleneck, so again our manual test coverage started going in the wrong direction. "As we moved into year three, we moved responsibility for maintaining the UI test harness to our developers, who were already responsible for writing the JUnit tests and FIT fixtures. Testers remained responsible for creating Fitnesse tables and JIFFIE tests. This has become increasingly important because we have introduced multiple hosted configurations, so we have to test many different configurations of the application. The testing infrastructure has become more complex and thus it needs to be more robust; it can only really be maintained by the team that knows the architecture of the user interfacethe developers. We are using Fitnesse to script all of our tests. We're putting a lot of effort in one button builds for the system testing harness that tests all our product configurations. "We hired a contract firm to help us, finally, pay down our remaining manual test dept by automating all of our remaining manual tests. We have finally replaced all of our original UI architecture, 14 months after making the decision to switch to Spring. Now that we are realizing the full benefit of running without debt, we can release every month with no hardening, which enables us to move to monthly updates and use quarterly releases to focus on major functionality enhancements. "We have invested a lot of effort in our testing framework because it is the key to growing the complexity of the application without growing the development team at the same pace. We have learned that acceptance-test-driven development is the way to goit's the only way to keep the technology agile as the application gets large. The key is to understand what creates debt, to slow downoccasionallyto catch up, and to realize that you are in it for the long haul. "We have figured this out through constant reflection and adaptation. After every iteration and every release, we discuss what worked and what didn't work, then make a list of what we will commit to changecommitment is essential. We tackle the list one at a time and experiment to see what works. "One of the things we tried turned out to be amazingly successful. When we reduced hardening from two weeks to one, we decided to use the free week to allow everyone to take a break from the high pressure of iterations. So instead of using the week to develop more code, we turned it into what we call "hack-a-thon." During hack-a-thon, which occurs immediately after a release, developers are free to experiment with anything they want. However, if customers find defects in the release, the developers have to stop hack-a-thon and create a patch. You can imagine that developers are extra careful not to let anything creep into the release that might cut into their hack-a-thon time. The biggest payoff from hack-a-thon, however, is that our most creative and novel ideas are hatched during this timeit is a time for building vision, design sets/continuums, and architectural spikes. It is undoubtedly the most productive time of all." Rally's focus on technical debt underscores how we have moved from the financial world, where debt is a financial term, to the knowledge world, where technical debt means that the expression of our knowledge is incomplete. Ryan's story also emphasizes the best way to create knowledge: Find the biggest problem that is getting in the way of your company's success, and focus everybody on figuring out how to solve that problem. Go at it one step at a time, check where that got you, and do it again and again and again. Make this the essence of everyone's job. What, Exactly, Is Your Problem?In 1950, Toyota almost went bankrupt and had to lay off one-third of its workforce. The Toyota Production System grew out of Toyota's commitment not to ever have to experience that painful situation again. The Toyota Production System was and remains fundamentally dedicated to reducing costs. The way Toyota figures it, the market determines what it will buy and pretty much sets the price. Profit is what's leftover after you subtract your costs from the market priceso the lower the costs, the higher the profits, and thus the better chance of staying in business and protecting jobs. Toyota didn't have a lot of money or tools when the Toyota Production System was developed. It had four simple goals:[4]
The Toyota Production System is Toyota's approach to meeting those goals. Rally perceived that its biggest threat to long-term profits was measured in units of time. The longer hardening took, the more debt it had. The more debt it had, the more linear the relationship between system complexity and development team size. And Rally knew that over time, a linear relationship of people to product size would be certain death. So the company attacked technical debt with the same vengeance that Toyota has always attacked cost. Art Smalley, is one of the handful of Americans who learned the Toyota Production System in the 1980s while working for Toyota in Japan. He is troubled by the tepid track record of lean initiatives and reflects on the root cause of the failure of most lean initiatives to gain traction. A couple of candidates he puts forward are 1) lean implementations seem to be focused on tools and practices rather than solving basic problems, and 2) lean training seems to be focused on staff specialists rather than on improving the problem-solving skills of natural work teams.[5]
Smalley believes that the first step in any improvement effort should be to ask two basic questions:
Lean initiatives must always start with a clear vision of how you make money and a sharp understanding of the most critical problem that is keeping you from doing so. The problem might be technical debt. It might be making sure each new video game is "fun." It might be system response time. It might be hardware and firmware integration. Whatever the problem is, becoming lean starts with a crisply defined, top-priority problem that is constraining business success. Once you have a clearly defined problem, the next step is to use a disciplined problem-solving method, informed by basic principles, to solve the problem. Train everyone how to do this with every aspect of their job. Create an environment in which work teams at all levels of the company continue to use a disciplined approach to solving their biggest current problem over many years. This is hard work and requires long-term thinking. Nevertheless, it is the path that Toyota has blazed to sustained success. A Scientific Way of ThinkingTom has a Ph.D. in physics. Our son, Dustin, has a Ph.D. in environmental engineering. Both of them spent several years under the direction of a major professor, using the scientific method to create and publish knowledge. As any scientist knows, the scientific method works like this:
The scientific method is the DNA of the Toyota Production System. Every Toyota worker is taught to use basic problem-solving techniques as the primary approach to doing their job. Toyota workers operate like a community of scientists, conducting ongoing experiments, constantly learning, and codifying new knowledge for the future in the same way that scientists have been doing for a long time. Figure 7.1 depicts Toyota's problem-solving method on the left and shows how it is similar to the plan, do, check, act (PDCA) cycle that Deming taught in Japan in the 1950s. Toyota embraced the scientific way of thinking, adding it to Sakichi Toyoda's concept of stop-the-line and Kiichiro Toyoda's vision of Justin-Time. The result is the Toyota Production Systemdisciplined problem solving at a detailed level on a consistent, companywide basis, focused on removing the current biggest constraint that is getting in the way of making a profit. Figure 7.1. The Toyota Production System problem-solving method and the PDCA cycle
The important thing to remember is that this problem-solving method is never used in a vacuum. It is used, consistently and repeatedly, to attack the current biggest problem facing the work team. As in any disciplined scientific environment, the work team is expected to keep track of all of their observations, their hypothesis, their experiments, and their results. There is at least as much knowledge created by experiments that fail as is created by finding a solution to the problem. Keeping Track of What You Know"You know the biggest problem with iterative development?" a product manager asked us. "I like the ability to keep trying new things, but I really can't keep track of everything I've tried. I worry that I'm going to have to go back over the same territory and learn the same lessons all over again." If learning is not captured in a useful manner, then people will indeed forget what they've learned. Even when individual people remember what they've learned, the organization will not benefit from the learning unless it is captured in a useful manner. Both Sakichi Toyoda and Kiichiro Toyoda were fundamentally inventors. They filed patents in countries around the world. Selling the patent rights to the automated loom provided seed money for Kiichiro to start an automotive company. Inventors who file patents have learned a lot about how to keep track of everything they learn, because the process of obtaining a patent depends heavily upon keeping detailed daily records of who knew what when, how they figured it out, and what the implications were at the time. Since inventors' notebooks are expected to be referenced regularly by inventor and colleague alike, they are concisely written and generously sprinkled with tables and graphs. Since they are legal records that may end up as evidence in a patent dispute, scientists make sure they are correct, complete, and contemporary.
In software development, the tests and code are often just the right combination of rigor and conciseness to document the knowledge embedded in the software. But experiments tried and options investigated on the way to making decisions about the product under development are easily forgotten, because they never make it into the software. Just writing something down does not necessarily turn it into knowledge, and information is just as easily lost in a sea of excess documentation as it is lost through lack of documentation. The Knowledge-Creating Company[6] by Ikujiro Nonaka and Hirotaka Takeuchi is a book about the mechanisms and processes by which knowledge is created in a company. They note that Western companies think of knowledge as something that is written down. Japanese companies think of written knowledge as only the tip of the iceberg; most knowledge is contained in subjective insights, intuitions, hunches, and mental models. This knowledge, tacit knowledge, does not come from studying, it comes from experience. It cannot be easily processed by a computer or stored in a database. It is hard to formalize and transmit to others.
Some think that if we just "document" what we learn through iterative developmentwrite down the decisions we made, why we made them, what we learnedwe will create organizational learning. But most likely, the pile of documentation we create will gather dust or take up space on a disk, quite useless as a learning tool. We tend to write diaries, unconsciously associating the depth of ideas with the thickness of the binder. But for those on the receiving end of documentation, the thickness of the binder is usually inversely proportional to its usefulness in transmitting knowledge. In order to create knowledge, we have to think not about the process of writing it down but about the people who will use what has been written. Technical writers do this for a living, so they might be asked play a key role in preserving team knowledge. The A3 ReportEarly in their careers, Toyota engineers learn the discipline of condensing complex thinking to a single A3 sheet of paper.[7] This forces people to filter and refine their thoughts so that anyone reading the A3 report will have all of their questions answered by reading a single sheet of paper. Different A3 reports have different purposes, but all of them capture critical knowledge in a way that is easy to store in a database, easy to post in a work area, easy to send to a manager, and is easy to incorporate into future experiments.
The Internet AgeNow that we are in the Internet age, we have some tremendously powerful ways to create and find knowledge. Powerful search engines change the way we think about indexing content. Blogs bring out the author in a large number of technical people. Wikipedia is a model for rapid and effectively collecting and arbitrating the content of very large of information libraries. At the same time, "knowledge management systems" didn't even show up in a recent survey of media used by knowledge workers.[8] Internet-age search and publishing tools do not excuse us from our obligation to concisely summarize our knowledge in a disciplined and useful way. But they will no doubt influence our approaches to keeping track of what we know.
|