Challenges in Building Notification Applications

At first, building a notification application may seem like a trivial task: Gather information, apply some logic to match against subscriptions, and send notifications. When thought about like this, the need for a notification application platform such as SQL Server Notification Services may not be apparent. But, when the system must support thousands, perhaps millions, of subscribers (as commercial systems often must), when it is deployed in mission-critical business environments where reliability is of the utmost concern, and when it is hosted in a data center where security simply cannot be compromised, the seemingly trivial task becomes more challenging. This section explores some of the challenges in building notification applications in more detail. Later in this chapter, the section "SQL Server 2005 Notification Services" (p. 15) explains how these challenges are easily met when using SQL Server 2005 Notification Services as a foundation for your notification applications.

Scalability

The capability of supporting many users and large volumes of data is critical to the success of many notification applications. For example, consider a stockbroker application that notifies subscribers when their stocks cross price thresholds that they specify in their subscriptions. Large volumes of data flow into the application: stock feeds containing many thousands of updated stock prices arrive as often as every few minutes from the stock market. Depending on the size of the broker's client base, there may be tens (or hundreds) of thousands of subscribers, each with several subscriptions for their stock symbols of interest. For such an application to work effectively, it should be capable of handling this kind of volume without excessive hardware.

Even if your intended user base or anticipated data volume is small today, it's important to build your notification application in such a way that it can scale gracefully with growth. After a system is in place and starts being used, it becomes more difficult and more expensive to replace it, so anticipating future growth is vital to keeping IT costs down.

In notification applications, two factors usually affect scalability:

• The cost (in terms of computing resources) of subscription matching

• Capacity limitations in the input and output systems

These factors are described in more detail in the following sections.

The Cost of Subscription Matching

Subscriptions can vary in complexity. Revisiting the stockbroker example, the simplest kind of subscription might say, "Notify me when the price of XYZ stock rises above $10." A more complex subscription might say, "Notify me when the price of XYZ stock rises above $10, and the trading volume is greater than 100,000 shares, and XYZ corporation appears in one or more favorable news stories."

In general, more complex subscriptions allow users to be more specific about the notifications they want to receive. But, as subscriptions become richer and more complex, the task of matching them with a set of incoming data becomes more expensive. Consider the computing resources required to match the two subscriptions just described. The first simple subscription is no more than a string match on stock symbol and a simple numeric comparison on stock price. The second, more complex, subscription involves an addition numerical comparison, full-text searching on news stories, and probably some additional metadata querying to determine whether a given news story is favorable.

Obviously, more computing resources are required to evaluate subscriptions of the second kind. This limits the number of those subscriptions that can be supported on a given hardware configuration. Increasing the number of supported subscriptions by limiting their complexity is not really a solution. The richer the subscriptions supported, the better and more interesting the experience for the subscriber. What's needed is a highly scalable matching engine capable of matching complex subscriptions at high enough rates to support the subscriber and data volumes required. Building such an engine is certainly not easy.

Capacity Limitations in the Input and Output Systems

Capacity limitations in the input and output systems are usually outside the control of the notification application developer. Even if you were to build a notification application that could match subscriptions infinitely fast, you'd still be limited by the speed at which the incoming data could be brought in, the capacity of the outbound network from the machine generating the notifications, and the bandwidth limitations of the delivery systems that route the notifications to the end users.

Suppose you built a notification application capable of a subscription-matching rate that produced 500 notifications per second. But, let's say, the network card and protocol implementation on the host system could push out only 100 notifications per second. The easiest way to deal with this (short of getting better network hardware, which is usually expensive) is to split the matching and the delivery tasks and to add additional servers to the delivery end. In other words, one server could generate the 500 notifications per second, and the task of delivering these could be split across five servers, each capable of delivering 100 notifications per second.

This principle applies equally well on the input side. If the amount of incoming data that a single input machine can supply is limited, you can split the task of submitting the input data across several machines to increase the overall capacity of your system. In general, adding machines to increase scale is referred to as scaling out.

Scaling out requires the application to be architected to support a distributed deployment. Getting this right is often a substantial development task.

Reliability

Many notification applications are customer facing, and so they often come associated with certain quality-of-service requirements. Users simply expect to get their notifications, and if they don't, they quickly become dissatisfied.

To meet these customer expectations, a notification application must seldom (hopefully, never) crash, restart gracefully after downtime (planned or unplanned), and work with high-availability clustering systems.

In addition, notification applications must be capable of dealing with the reality of failures in the infrastructure around them. Incoming data feeds don't always arrive as expected or may contain erroneous information. Delivery systems sometimes fail. Notification applications should be capable of detecting (possibly even correcting) problems in incoming data feeds or alerting an administrator if feeds don't arrive as expected. Should notifications fail to be delivered, the application must be capable of retrying the delivery at a later time, using some intelligence to handle prolonged network downtime as well as random, intermittent failures.

Rich Application Features

When first thinking about building a notification application, developers usually think in terms of what the incoming data looks like, what kinds of subscriptions to support, what kind of logic it takes to match the two, and what the final notifications look like. Although these aspects are at the core of the application, they are usually not the parts that take the most time and effort to build, test, and debug.

When building a notification application without the help of a platform framework such as SQL Server Notification Services, the bulk of the development time is spent building features such as digesting (aggregating several notifications for the same subscriber into a single message), multicasting (sending the same notification to several subscribers), support for multiple languages, and scheduled subscriptions (subscriptions that fire at a specific time of day, rather than in response to any particular incoming information feed). Also, features that make the application manageable, such as diagnostic capabilities, reporting tools, and administration facilities, are often time-consuming to build.

Security and Privacy

Notification applications usually maintain personal data in the form of subscriptions that describe what information each user wants to receive, as well as the delivery addresses of the devices on which they receive it. Also, they have inbound and outbound connections that can be potentially dangerous if compromised. Therefore, security should be the first concern of anyone building a notification application.

Subscriber and subscription data must be kept secure and private. No unauthorized person should be able to read or modify this data. Furthermore, subscriptions should be isolated: No subscription should affect the results of any other.

Any inbound or outbound network connection should be treated as a potential security vulnerability. Regardless of what security measures are in place, a notification application must be designed for defense in depth, so that if a network connection were compromised and used by an attacker to gain entry into the system, his ability to do damage would be limited. A notification application should be designed to run under a low-privileged account so that it doesn't become a target for code-injection attacks.

Because notification applications, by nature, send messages, if compromised, they can be used to spam users or launch denial of service attacks. Auditing and throttling mechanisms need to be in place to prevent such malicious misuses should a security breach occur.

In addition, notification applications may be connected to one or more database servers on which data owned by other systems and applications is stored. Notification applications need to be designed so that no attacker could use them to gain unauthorized access to other data on the database servers.