Architectural Evolution and Maturation: Features versus Capabilities

Architectural Evolution and Maturation : Features versus Capabilities

Although much emphasis is placed on the initial creation and early versions of an architecture, the fact is that most of us will be spending the majority of our time working within an existing architecture. How a given architecture evolves can be more fascinating and interesting than simply creating its initial version. It is through evolution that we know where we have success, especially when the evolution is based on direct customer feedback. On a personal note, I've faced my greatest managerial challenges and had my most satisfying accomplishments when I've been able to work with software teams in modifying their existing architecture to more effectively meet the needs of the marketplace and position their product for greater success.

This process, which involves both evolution and maturation, is driven by actual use of the system by customers. Many companies claim to continually request customer feedback, but the reality is that customer feedback is most actively sought, and most thoroughly processed , when planning for the system's next release. The next release, in turn , is defined by its specified required functionality as expressed in the features marketed to customers. Whether or not these features can be created is dependent on the underlying architecture's capabilities. The interplay of requested or desired features and the underlying capabilities required to support them is how architectures evolve over time.

What makes this interplay so interesting is that it takes place within continual technological evolution. In other words, customer feedback isn't always based on some aspect of the existing system, but can be based on an announcement about some technology that could help the customer. In fact, the source of the announcement doesn't have to be a customer, but can quite often be the development team. Regardless of the source or context, it is useful to think of architectural evolution and maturation in terms of features and capabilities.

A feature, or function (I use the terms synonymously but prefer feature because features are more closely related to what you market to a customer) defines something that a product does or should do. The entire set of requested features defines the requirements of the product and can be elicited, documented/captured, and managed through any number of techniques. In case you're wondering what a feature is, here is a time- tested definition from Exploring Requirements: Quality Before Design [Weinberg and Gause, 1989]: "To test whether a requirement is actually a [feature], put the phrase 'We want the product to ' in front of it. Alternatively, you can say, 'The product should '" This approach shows that a wide variety of requirements qualify as features. Here are a few examples:

• Supported platforms. "We want the product to run on Solaris 2.8 and Windows XP."

• Use cases. "The product should allow registering a new user ."

• Performance. "We want the product to provide a dial tone within 100 ms of receiving the off-hook signal."

Note that, as descriptions of features, use cases have an advantage over other forms of documentation because they can put the desired feature into a specific context. The most useful context is based on the goals of the actors involved in the use casewhat they are trying to accomplish. Once you know this, and you know that the system can fulfill these goals, you have the data that product management needs to create the value proposition, which informs the business, licensing, and pricing models.

Features are most easily managed when clearly prioritized by marketing, and they are best implemented when the technical dependencies between them are made clear by the development team. This is because features are usually related, in that one often requires another for its operation. As stated earlier, because the system architecture determines how easy or hard it is to implement a given feature, good architectures are those in which it is considered easy to create the features desired.

Capability refers to the underlying architecture's ability to support a related set of features. The importance of a capability emerges when marketing is repeatedly told that a class of related features, or a set of features that appear to be unrelated on the surface but are related because of technical implementation, is difficult or impossible to implement within the given architecture. Examples in the sidebar on page 12 illustrate this point.

The interplay between architectural maturation and evolution is a function of time and release cycle. It is uncommon for the development team to implement most or all of the desired features in the very first release, especially if the product manager specifies requested features in a well-defined , disciplined manner. If all of the desired features aren't implemented in the first release, they often come along shortly thereafter. Because there aren't a lot of customers to provide feedback, the team is likely to continue working on the leftover features from the first release and complete them in the second. The focus of the team is on completing the total set of desired features, so there is usually little time or energy spent on changing the architecture. Instead, the initial architecture matures. Certainly some changes are made to it, but they are usually fairly tame.

After the system has been in continuous operation for three or more release cycles, or for two or more years , the initial features envisioned by its creators have typically been exhausted and product managers must begin to incorporate increasing amounts of direct customer feedback into the plans for future releases. This feedback, in the form of newly desired features or substantial modifications to existing features, is likely to mark the beginning of architectural evolution, as the development team creates the necessary capabilities that provide for these features.

Of course, not all architectural evolution is driven by customer demand. Companies that proactively manage their products will look for new technologies or techniques that can give them a competitive edge. Incorporating key new technologies into your evolving architecture can give you a sustained competitive advantage. Chapter 3 discusses this in greater detail, and Appendix B includes a pattern language that shows you one way to organize this process.

Architectural maturation and evolution are cyclical, with each phase of the cycle building on the previous one. New capabilities are rarely introduced in a mature state, as it is only through actual experience that can we know their true utility. Through feedback, these capabilities mature. In extremely well-tended architectures, capabilities are removed.

There is a special situation in which the maturation/evolution cycle must be broken and where capabilities dominate the development team's discussions. This is when you must undertake a complete redesign/rewrite of an existing system. Although many factors motivate a redesign/rewrite, one universal motivator is the fact that the existing system does not have the right capabilities and adding them is too costly, either in terms of time or in terms of development resources. In this situation, make certain your architect (or architecture team) clearly captures the missing capabilities so that everyone can be certain that you're going to start with a solid new foundation.

Architectural Capability or Feature Agility?

In one system I managed, users could search and collaboratively organize documents into folders. New documents also arrived from external sources on a regular basis. One set of requirements that emerged concerned storing and executing predefined queries against new documents. Another set comprised the notion that when one user modified the contents of a folder another user would receive an e-mail detailing the modification. Both feature sets required the architectural capability of a notification mechanism. Until this capability was implemented none of the desired features could be realized.

The primary target market of this system was Fortune 2000 companies, and the original business model was based on a combination of perpetual or annual licenses accessed by named or concurrent user. While these business models are standard in enterprise-class software, they prevented us from selling to law firms, which need to track and bill usage on a metered basis for cost recovery. Working with my product managers, we realized that we could address a new market segment if we provided support for a metered business model. Unfortunately, it was far easier to define the new business models than to implement them, as they required substantial changes to the underlying architectural capabilities. Instead of simply counting the number of named or concurrent users, the development team had to create several new capabilities, such as logging discrete events into secured log files for post processing by a billing system. Until this was done we were unable to address this newly defined market segment.

A different system I managed was responsible for creating trial versions of software. Trialware is a powerful marketing tool, created by applying special tools to software after it has been written to add persistent protection to it. Customers of this system asked for increased flexibility in setting trial parameters. Unfortunately, this seemingly simple request was not supported by the underlying architecture, and I had to initiate a major design effort to ensure that the right capabilities were added to the system.

A common example of the difference between features and capabilities is when product marketing requests that the user interface be localized in multiple languages. Each language can be captured as a separate feature and is often marketed as such, but unless the underlying architecture has the necessary capability for internationalization, supporting multiple languages can quickly become a nightmare for the entire organization. There are many other examples, especially in enterprise applications: workflow, flexible validation rules, increasing demands for customizations, to name but a few.

The motivation for redesign/rewrite (the lack of underlying capabilities) often finds its roots in the delivery of new features to a growing customer base as quickly as possible. Quite simply, success can breed failure when is not properly managed.

Architectural degradation begins simply enough. When market pressures for key features are high and the needed capabilities to implement them are missing, an otherwise sensible engineering manager may be tempted to coerce the development team into implementing the requested features without the requisite architectural capabilities. Usually many justifications are provided by everyone involved: Competitors will beat the team to an important customer, or an important customer won't upgrade to the next version, delaying critically needed revenue. Making these decisions is never easy.

The almost certain outcome of this well-intentioned act is that the future cost of maintaining or enhancing new features will be quite high. Additionally, because the underlying architectural capabilities have not been added to the system, any other new features depending on them will find themselves in the same predicament. Worse yet, the cost to morale will likely reduce the effectiveness of the team, further compounding the problemespecially if the alleged "market pressures" turn out not to be so pressing.

The result is that implementing new features results in the team taking on a technical "debt" that must be repaid in the future. ^[1] The principal of this debt is the cost associated with creating the right underlying capability. It simply won't go away. The interest is the additional burden of sustaining a feature in an architecture that can't support it. This interest increases with every release and increases again as customers continue to demand new features. If things get bad enoughand they can and dothe team might eventually have to scrap the entire architecture and start over. Servicing the debt has become too high.

^[1] Ward Cunningham is the first person I know of to refer to this as technical debt.

While sometimes there really is a critical market window and a shortcut must be taken, or you simply must implement a given feature for an important customer, more often this is just an illusion fostered by impatient and harried executives looking for a supposed quick win.

As people mature, they often became wary of taking on unnecessary debt. You cannot wish away debt but have to pay it. When a given set of desired functions require a significant new or modified architectural capability, avoid debt whenever you can. When it comes to implementing needed capabilities, the phrase "Pay me now or pay me later with interest and penalties" truly applies.

Entropy Reduction: Paying off Technical Debt after Every Release

In the heat of battle a development team will usually need to make some compromises on the "quality" of the code and/or its implementation to get the product finished. It doesn't matter if this team is "agile" or not. To get the product shipped you have to focus on shipping it. This usually means making compromises (hacks, ugly codeyou get the picture.).

No matter how they get into the code base, unless these compromises are removed the quality of the source base will degrade. After the product is shipped some reasonable amount of time needs to be devoted to improving the source code. I call this "entropy reduction" (ER). A development organization realizes several crucial benefits from the ER process, a few of which are itemized below in no particular order.

• It maintains a focus on source-level quality. Developers know the importance of a very fundamental level of source code quality. The company reduces risk associated with an unmaintainable code base.

• It reinforces good will in the development team. Great developers wince when they compromise the source code in order to ship the product. They'll do it, but they won't like it. If you don't let them go back and clean up, they will become "dead" to quality. If their sense of quality dies, so do your products.

• It substantially improves the maintainability and extensibility of the system.

• It allows the team to clearly understand the specific sets of behavior within the system and makes them aware of the possibilities of really changing it.

• It ensures that any inconsistencies relative to things like the coding standard are addressed.

Entropy reduction is not about making substantial changes to the architecture or adding lots of new features. The goal is to hold the external behavior of the system constant while improving its internal structure.

Entropy reduction is usually scheduled after a major release, roughly every 9 to 12 months for most products. Within these release cycles your team should be refactoring their source code as needed. Entropy reduction is often about handling refactorings that were postponed so that you could release your system or avoided because they were deemed too big or complex.

It is important to establish appropriate rules or guidelines when initiating an ER session. The most important is the requirement that no new features or capabilities be added during ER. To help enforce this rule, try to keep product management from becoming involved. Keep ER as a pure development activity.

Functional tests, if they exist, of course need to run. Do not ship the product after ER without a full and complete regression test.

Teams vary in their willingness to engage in ER. Those that embrace it are easy to manage. Those that enthusiastically embrace it may be trying to use the process to change the architecture. Be wary! Teams that have had their sense of quality and their desire to tend their architecture beaten out of them by poor management may have to be forced to do it ("I know that this ER thing sounds weird, but this is just the way that I do things, and since what we've done in the past isn't producing the results we want we're going to try something new").

Before engaging in ER the team should create and peer-review a plan that details the specific modules and/or functions they're going to improve. This is important because sometimes a team will propose changes that are a bit too big to accomplish in a one to two week period, which is the amount of time I normally allow for an ER session. To help ER run smoothly here are some best practices that have worked well for me.

• Use code tags. Code tags are a way of identifying sections of the source code that should be fixed in a future ER session. I've had teams use "TODO," "REDO," or even "HACK" to alert readers of the source that something should be fixed. Finding the tags is easy.

• Establish a rhythm. The rhythm of ongoing releases is very important to me. Successful teams develop a nice rhythm, working at a good pace during the majority of the development. It's like a brisk walksustainable but fun. Near the end, they work as needed, including 80- hour work weeks, to hit the release date. The sprint to the finish. After the release the team recovers. They rest or work on ER (which should not be mentally challenging).

• Time-box the ER activity. One to two weeks works best. I once tried three weeks, but the process was too hard to controlnew features were being added. If you need four weeks or more for productive ER, look in the mirror and say, "The architecture is dead."

• Don't ship the result. You could be making changes all over your code base. Just because your automated unit tests pass, you still can't ship without a full regression test.

• Choose your language carefully . I've used the terms "technical debt" and "entropy reduction" to mean the same basic thing. Different people hear these words in different ways, so consider if the phrase "entropy reduction" will work for your team before you use it. If it won't, consider something else, like "technical debt payback," or "architectural improvement."

• Use ER to reinforce other values you deem important. You can use ER to instill a whole host of values, including source code quality, the importance of finishing on time, responsible management ("incur a little technical debt now and you'll get the time to pay it back after the release"). You can associate cultural artifacts with the first ER session, like a "chaos" game or a toy (a symbol).

• Don't commit unless you can deliver. If you promise an ER session, make certain you can deliver. I almost lost all credibility in a turnaround situation when the CEO of my company told me that I couldn't do an ER session because he had promised a new feature to a key customer. It took some stringent negotiation, including a few heated meetings, but in the end we had the session before we attempted to deliver the promised feature. Avoid this pain and make certain you have appropriate senior executive buy-in for ER.