Inside the Notification Generator

All rules in a SQL-NS application are invoked by the generator component of the SQL-NS engine. Before running the rules, the generator is responsible for setting up the runtime environment that the rules rely on, including the views of the event and subscription data, and the state required by the triggers on the notification views.

In preparation for learning about the debugging stored procedures used to run rules manually, it's important to understand the way the generator works internally. This section describes the generator's processing model and how it affects the way that rules execute.

The Quantum Clock

In the generator model, the passage of time is seen not as a continuum, but rather as a series of distinct, fixed-length intervals. In SQL-NS terminology, each interval is called a quantum. A quantum defines a discrete "slice" of time, as shown in Figure 11.1.

Figure 11.1. The generator divides time into distinct quantums.

If the length of a quantum is 1 minute, the generator divides any block of time into a series of 1-minute intervals. Consider a 3-minute period, starting at 12:00. The generator does not see this as a continuous stretch of time but instead as a sequence of three quantums: 12:00 to 12:01, 12:01 to 12:02, and 12:02 to 12:03.

Note

The correct plural form of the English word, quantum is really quanta. However, in the context of SQL-NS, the form quantums is commonly used instead. You'll see this usage in the names of several stored procedures, tables, and columns in SQL-NS databases, as well as in some documentation.

Although the usage is not always consistent, quantums seems to be the more accepted form. For this reason, I use quantums instead of quanta throughout this book.

The generator maintains two clocks: a real-time clock that keeps the actual time and a quantum clock that tracks the current quantum. When the generator first starts, the two clocks are synchronized, and the first quantum spans the interval from the current real clock time to one quantum interval in the past. For example, if the generator starts at 12:00 and the quantum interval is 1 minute, the first quantum is the time period from 11:59 to 12:00.

Quantum Processing

The generator performs its processing one quantum at a time. When processing a quantum, the generator runs the application's rules against only the event batches that were submitted during the time interval that the quantum spans. Continuing the previous example, during the processing of the first quantum, the generator would run the chronicle and match rules against only the event batches that arrived between 11:59 and 12:00. The data from all other event batches would not be visible in the event views used in the rules. Thus, the quantum serves as a scoping mechanism for the event data against which the rules run.

Note

The quantum being processed always describes a time period in the past. The generator never processes a quantum that defines a time period encompassing the present real clock time, or extending into the future.

In addition to scoping event data, the quantum also scopes the scheduled subscriptions against which the rules run. During the processing of a quantum, the only scheduled subscriptions visible to the rule logic are those for which next scheduled evaluation time falls between the start and end times of the quantum. The subscription views will contain only the data for these subscriptions.

Note

Although the end time of one quantum is the start time of the next quantum, quantums are treated as non-overlapping periods. An event batch that arrives exactly on a quantum boundary will be processed in the quantum starting at that boundary (not in the quantum ending at that boundary). For example, an event batch arriving at exactly 12:00 would be processed in the 12:00 to 12:01 quantum, not the 11:59 to 12:00 quantum. Similarly, scheduled subscriptions with schedules on quantum boundaries will be processed during the quantums starting at those boundaries.

When it finishes running the rules against the events and subscriptions in a given quantum, the generator sleeps until the start of the next quantum. This is the cause of the latency you may have seen when waiting for the generator to process the events you submitted to the music store application in previous chapters' examples. If the events arrived during a time when the generator was sleeping, they would not be processed until the start of the next quantum. The duration of the quantum interval determines how long the delay might be. The default quantum duration for SQL-NS applications is 1 minute, but we have used a shorter duration of 15 seconds in all the examples so far (the setting that changed the quantum duration in the music store application is shown in Listing 5.10, p. 156, in Chapter 5). Thus, you may have experienced wait times of up to 15 seconds before the generator began processing new events in the music store application.

The "Optimizing the Quantum Duration" section (p. 434) in Chapter 12, "Performance Tuning," explains more about altering quantum durations and the effect this has on application performance. In summary, specifying a shorter quantum duration makes the application more responsive (because each generator sleep is shortened) but increases the processing overhead. In your own applications, you will need to select an appropriate quantum duration that takes into account the frequency of event arrival and balances the responsiveness requirements against the overhead involved. Chapter 12 discusses these trade-offs in more detail.

Discrepancies Between the Quantum Clock and the Real-Time Clock

Ideally, the quantum clock and the real-time clock always remain synchronized, with the current quantum always running from one quantum interval in the past to the current real clock time, as shown in Figure 11.2.

Figure 11.2. The quantum clock and the real-time clock in sync.

For the clocks to remain synchronized, the work that the generator does for each quantum should take no longer than the quantum interval. However, processing the events and subscriptions in a quantum can sometimes take longer than the quantum interval when the system is under heavy load. Suppose, for example, that because of the volume of data, processing the events and subscriptions in the 12:00 to 12:01 quantum actually took 4 minutes to complete. The quantum processing would have started at 12:01 (in real clock time) and ended at 12:05. At this point, the generator would then process the next quantum, 12:01 to 12:02. In this state, we say that the quantum clock has fallen behind. If the quantum clock had been up-to-date, then, at 12:05, the generator would instead have begun processing the 12:04 to 12:05 quantum.

Even though the real-time clock is ahead of the quantum clock, the generator still processes one quantum at a time. Regardless of the real clock time, during the 12:01 to 12:02 quantum, the generator makes the events and subscriptions only from that quantum available to the rules. If additional events arrived after 12:02 or additional scheduled subscriptions were due for evaluation after 12:02, those would be processed during later quantums.

Although it always processes one quantum at a time, the generator does not sleep between quantums when the quantum clock falls behind the real-time clock. After finishing the processing of one quantum, the generator immediately starts the next one (without sleeping until a quantum boundary) until the quantum clock catches up to the real-time clock.

When configuring your own applications, always choose a quantum duration longer than the time required to process the events and subscriptions in a typical quantum. This allows the quantum clock to eventually catch up to the real-time clock, even if a few quantums occasionally require longer than the quantum duration to process. Figure 11.3 shows the quantum clock falling behind because of a long-running quantum and then catching up by evaluating the next few quantums more quickly.

Figure 11.3. The quantum clock falling behind and then catching up.

Apart from long-running quantums, another reason the quantum clock can fall behind the real-time clock is generator downtime. If the generator stops running, the real-time clock continues to advance, but the quantum clock stops. When the generator starts up again, the quantum clock resumes from the last quantum processed, even though the real-time clock may be significantly ahead. The generator continues to process quantums one at a time, until the quantum clock catches up with the real-time clock. This behavior is the same whether the generator was taken down intentionally or was terminated abnormally.

Note

You can control how far the quantum clock is allowed to fall behind the real-time clock through settings in the ADF. These settings specify when the generator should start skipping the processing of old quantums to allow the quantum clock to catch up to the realtime clock. The "Quantum Limits" section (p. 437) in Chapter 12 describes these settings in detail.

Activities Within a Quantum

The processing of a quantum consists of several steps. First, the generator selects the events and subscriptions against which the rules should execute and populates views of the data in these events and subscriptions. Next, it executes the rules by invoking their SQL statements, which operate against the event and subscription views. For those rules that insert into a notifications view, the generator sets up the state that the triggers on the views rely on. After all the rules have executed, the generator cleans up this state, as well as the views that it populated for the quantum, and then goes to sleep.

This section describes these stages of quantum processing in more detail. It's important to understand how quantum processing works in order to debug your rules.

Event and Subscription Selection

The first step in processing a quantum is selecting the events and subscriptions against which the rules will be invoked. To understand the way the generator selects events, look at an event batches table (for example, the NSSongAddedEventBatches table for the music store application's SongAdded event class).

Each row in the table represents one batch of events. The StartCollectionTime column contains the time at which the event provider began submitting the events in the batch, and the EndCollectionTime column contains the time at which the batch was finally committed. In a given quantum, the generator selects event batches where the EndCollectionTime falls within the quantum's boundaries. In other words, it selects batches where the EndCollectionTime is greater than or equal to the quantum start time and less than the quantum end time. The generator selects event batches that meet this criteria from all the event classes in the application.

Note

The terms quantum start time and quantum end time refer to the start and end of the time interval that the quantum represents. They do not refer to the times that the generator starts and ends the processing of the quantum. The quantum start and end times are expressed in terms of the quantum clock, whereas the processing start and end times are measured by the real-time clock.

The generator selects event-triggered subscriptions based on the event classes of the selected event batches. Recall from Chapter 5 that the declaration of an event rule in a subscription class specifies the name of the event class that it matches. For example, the event rule in the NewSongByArtist subscription class specifies that it matches subscriptions against SongAdded events. The generator determines which subscription classes have event rules associated with the event classes of the selected event batches and then selects all the event-triggered subscriptions of those subscription classes. The event rules executed in the quantum operate against these event-triggered subscriptions.

Finally, the generator selects scheduled subscriptions by evaluating their schedules and choosing the ones due to be evaluated between the quantum start and end times. Each scheduled subscription specifies a start time and recurrence pattern, and the generator uses an efficient algorithm to determine from these which subscriptions have next evaluation times that fall within the current quantum. Because scheduled rules are not associated with any particular event class, the generator considers scheduled subscriptions from all subscription classes, regardless of the event classes represented in the selected event batches.

View Materialization

After the generator selects the event batches and subscriptions for the quantum, it materializes views of the event and subscription data. For each event class for which there is at least one selected event batch, the generator populates an event view. Event views have the same names as the event classes they represent. Each view contains the event data from the selected batches of one event class.

Similarly, for each selected subscription class, the generator populates a subscription view with the selected subscription data. Subscription views have the same names as the subscription classes they represent. In the case of subscription classes with event rules, all subscriptions appear in the view. For subscription classes with scheduled rules, only those subscriptions scheduled to be evaluated during the current quantum are included.

Now that you understand how the views are defined, it's worth going back to the definition of a match rule and revisiting how the views are used. For example, the match rule for the NewSongByArtist subscription class is written against views of SongAdded events and NewSongByArtist subscriptions as follows:

 INSERT INTO [SongAlerts].[NewSong] SELECT ... FROM   [SongAlerts].[SongAdded] events JOIN   [Catalog].[SongDetails] songs     ON events.SongId = songs.SongId JOIN   [SongAlerts].[NewSongByArtist] subscriptions     ON subscriptions.ArtistName = songs.ArtistName 

The references to the views are shown in boldface font in the FROM clause: SongAdded is a view of events of the SongAdded event class, and NewSongByArtist is a view of the subscriptions of the NewSongByArtist subscription class. The generator is responsible for ensuring that these views contain the correct event and subscription data for the quantum before executing the rule. The rule itself just matches whatever data it sees in these views. This model allows you to write rules without having to think about the scoping of data. In the rule logic, you do not have to be concerned with distinguishing new events from old ones, the ordering of event batches, or evaluating subscription schedules.

The reliance on the event and subscription views is one of the reasons that it's difficult to run rules manually by cutting and pasting their text into a Management Studio query window. The views are populated by the generator only at runtime, so their contents are unpredictable at other times. Later in this chapter, you'll see how to set up these views using the SQL-NS debugging stored procedures so that you can execute rules manually.

Rule Firing

After establishing the event and subscription views, the generator executes the rules. This section describes two important aspects of rule execution: the generator ordering models and the handling of rule failures.

Ordering Models for Rule Execution

The generator offers two models for the ordering of the event batches and scheduled subscriptions processed by the rules: quantum ordering and strict event batch ordering. The choice of ordering model affects the way the rules are executed.

When using quantum ordering, the generator processes all event batches in the quantum together. After placing the event data for all the event batches in the views, it first executes the event chronicle rules, followed by the event-triggered match rules, and finally executes all scheduled rules. The generator does not make any guarantees about the order in which rules for different event classes will be executed. In fact, rules for different event classes may be executed in parallel.

When using strict event batch ordering, the generator strictly observes the order of events and subscriptions within a quantum. For this reason, this processing model is also sometimes referred to as subquantum ordering. Using this model, processing in a quantum proceeds according to the following sequence:

1.	The generator populates the views of the subscription classes that have event rules but leaves all other views empty.
2.	The generator processes all scheduled subscriptions scheduled to be evaluated before the arrival time of the first event batch in the quantum. It does this by placing these subscriptions in the scheduled subscription views and leaving the event views empty. It then executes the scheduled rules. After executing the scheduled rules, the generator empties the scheduled subscription views.
3.	The generator looks for the unprocessed event batch in the quantum with the earliest EndCollectionTime and places its event data in the event view. It then runs the event chronicle rules and event-triggered match rules (the event-triggered subscription views were established in step 1). After executing the rules, the generator removes the event data from the event views.
4.	The generator then looks for scheduled subscriptions in the quantum scheduled to fire before the EndCollectionTime of the next event batch in the quantum. If any such scheduled subscriptions exist, the generator places their data into the scheduled subscription views, runs the scheduled rules, and then empties the scheduled subscription views.
5.	The generator then repeats steps 3 and 4 until no more event batches or scheduled subscriptions remain unprocessed in the quantum.

This sequence of operations has the following implications:

• Event batches and scheduled subscriptions are evaluated in strict time order.

• Event batches are processed one at a time.

• During the processing of an event batch, only the event data from that event batch is visible in the events view. The data from other event batches in the quantum with earlier or later EndCollectionTime values is not visible.

• All scheduled subscriptions that occur between event batches are processed togetherthat is, individual scheduled subscriptions are not ordered with respect to one another.

Quantum ordering is the more efficient of the two models, but it does not offer stringent ordering guarantees. In contrast, as its name suggests, strict event batch ordering guarantees that all event batches and subscriptions in a quantum will be processed in order. But, making this guarantee reduces the generator performance. You will need to select the appropriate ordering model based on the required semantics for your application.

In most applications, the more relaxed ordering semantics of the quantum ordering model are sufficient. This is the default mode in which the generator operates. If the semantics of your application demand strict event batch ordering, you must select that model instead, but this will have an impact on the performance of your application.

To specify the generator ordering model, you use the optional <ProcessEventsInOrder> element in the ADF. The value of this element is a Boolean that indicates whether strict event batch ordering should be used. The default value is false, meaning that quantum ordering is used. <ProcessEventsInOrder> is a subelement of the <ApplicationExecutionSettings> element in the ADF, in which general executing settings for the application are specified. Listing 11.1 shows how the <ProcessEventsInOrder> element is used to specify strict event batch ordering for an application.

Listing 11.1. The <ProcessEventsInOrder> Element in the ADF

 <Application>   ...   <ApplicationExecutionSettings>     ...     <ProcessEventsInOrder>true</ProcessEventsInOrder>   </ApplicationExecutionSettings>   ... </Application> 

Handling Rule Failures

If a rule fails to execute, the generator logs an error in the Application event log (usually with the event ID, 4045). By looking at the event log using the Event Viewer, you can determine which rules failed. Usually, the event message provides some indication of the reason for the failure. In general, after a rule failure, the generator continues processing subsequent rules.

The exception to this is that if an event chronicle rule fails, the generator does not execute the event-triggered match rules for that event class. This maintains consistency within the application because, in many cases, the event-triggered match rules rely on state maintained by the chronicle rules. If the chronicle rule fails to update the state of the application, the event-triggered match rules may produce incorrect notifications.

Cleanup

After all rules have executed, the generator cleans up the state associated with the quantum. It removes the event and subscription views and clears out any temporary tables that it used.

If the generator shuts down abnormally during the processing of a quantum (due to a power failure, for example), the quantum cleanup will happen the next time the generator starts. In general, quantum state is always persisted in the database, so the generator can recover from abnormal termination gracefully.