Section 17.1. Latency

17.1. Latency

Network latency is the time it takes for a packet of information to travel across a network (or networks) from sender to receiver. On a high-speed local network, it may take only a millisecond (ms; one-thousandth of a second) for a single packet of information to travel from sender to receiver. An organization with network nodes spread over a few kilometers and a high-speed backbone should still provide latency of less than 10 ms. Organizations with a high-speed connection into an Internet service provider (ISP) may achieve latencies of 15 to 50 ms when reaching well-connected national sites, as well as some international sites. On the other hand, home users will often experience latencies in the 40 ms to 500 ms range when accessing the same sites on the Internet. Network congestion can increase latency dramatically. The routers in a congested network will drop packets they cannot handle. FlashCom uses the TCP protocol, which automatically resends dropped packets. If packets are repeatedly dropped and resent , latency can increase dramatically.

While everyone who has ever used the Internet has experienced network latency, the delay in delivering packets across the Internet is not the only cause of slow response times. A busy FlashCom application instance will queue remote method calls and deal with them one at a time. If there is a long delay between the time someone sends a message in a text chat and the time it takes her message to appear in the chat message area, it may be because of a slow network connection, a busy application instance, or both.

17.1.1. Measuring Latency

Latency is normally measured by sending information from the sender to the receiver and having the receiver immediately return it. The round-trip time is calculated when the information is returned and divided by 2 to get an approximation of the latency. The idea of bouncing a packet of information off the receiver and measuring the time it takes to return is somewhat analogous to sonar . So when Mike Muuss wrote a program that sent an Internet Control Message Protocol (ICMP) ECHO_REQUEST packet and measured the time until an IP/ICMP ECHO_REPLY packet was returned to measure latency, he named it ping . Ping is still a tremendously useful diagnostic tool, provided the system you are pinging is not on a network that refuses to forward ICMP packets. Ping gives a good point-in-time measure of basic network performance at the packet level. Most computers, including Windows machines, have a ping program you can run at the command prompt. Just type ping followed by a space and the hostname or IP address of the machine you want to ping. For example, pinging flash-communications.net as follows from my home machine, which has a DSL connection, typically reports latency of 65 ms:

 C:\>ping flash-communications.net Pinging flash-communications.net [209.29.148.179] with 32 bytes of data: Reply from 209.29.148.179: bytes=32 time=60ms TTL=118 Reply from 209.29.148.179: bytes=32 time=70ms TTL=118 Reply from 209.29.148.179: bytes=32 time=60ms TTL=118 Reply from 209.29.148.179: bytes=32 time=71ms TTL=118 Ping statistics for 209.29.148.179:     Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds:     Minimum = 60ms, Maximum =  71ms, Average =  65ms 

From my high-speed office connection, the same command produces dramatically faster results (latency below 10 ms is reported as 0 ms):

 C:\>ping flash-communications.net Pinging flash-communications.net [209.29.148.179] with 32 bytes of data: Reply from 209.29.148.179: bytes=32 time<10ms TTL=117 Reply from 209.29.148.179: bytes=32 time<10ms TTL=117 Reply from 209.29.148.179: bytes=32 time<10ms TTL=117 Reply from 209.29.148.179: bytes=32 time<10ms TTL=117 Ping statistics for 209.29.148.179:     Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds:     Minimum = 0ms, Maximum =  0ms, Average =  0ms 

There are two ways to construct a similar test using Flash and FlashCom to provide latency information to an application. One way is to have the client bounce a message off FlashCom or vice versa, by calling a remote method. For example, a Flash movie can get the current time, call a remote method on the server that does nothing but return, and then get the current time again after the remote method returns. The difference in time between when the remote method invocation was sent and when its onResult( ) handler is called is the round-trip time.

The other way to test latency is available only on the server. The Client.ping( ) method will send a message to the client, which the client will immediately return. Some time after the message is returned, the Client.getStats( ) method can be called to retrieve the round-trip time.

Example 17-1 is part of the definition of the client-side ConnectionTester component available from the book's web site. The abbreviated listing shows the pingResponder object that is created in the constructor, the set nc( ) method in which the component is passed an already connected NetConnection object, and the onEchoReply( ) method that is called when the onEchoReply( ) method being invoked on the server returns. For the full listing, see the ConnectionTester.as file in the examp1e17-1.zip archive.

Example 17-1. Partial listing of the client-side ConnectionTester class

 function ConnectionTester (  ) {   super( );   // Ping responder.   pingResponder = {owner:this};   pingResponder.onResult = function ( ) {     var receiveTime = new Date( );     this.owner.onEchoReply(this.sendTime, receiveTime);   };   pingResponder.onStatus = function (status) {     trace("ping responder.onStatus: " + status);   }; } public function set nc (nc:NetConnection) {   __nc = nc;   buffer = [];   pingResponder.sendTime = new Date( );   __nc.call(path + "echoRequest", pingResponder); } function onEchoReply (sendTime, receiveTime) {   var deltaTime = receiveTime - sendTime;   buffer.push(deltaTime);   pingCount++;   if (pingCount < totalPingCount) {     pingResponder.sendTime = new Date( );     __nc.call(path + "echoRequest", pingResponder);   }   else {     var sum = 0;     writeln("Pinged " + pingCount + " times.");     for (var i = 0; i < buffer.length; i++) {       writeln("Ping #" + i + " rtt: " + buffer[i]);       sum += buffer[i];     }     writeln("Average ping rtt: " + (sum/buffer.length));   } } 

If the full component path is pfcs/ConnectionTester/main , then the actual remote method being called is pfcs/ConnectionTester/main/echoRequest( ) . The pingResponder object is passed into the call( ) method. When the remote method returns, the pingResponder.onResult( ) method will be called. It will get the current time and then call the component's onEchoReply( ) method where the round-trip time is calculated.

The remote method is attached to the client on the server in the addClient( ) method of the server-side ConnectionTester class, as shown in Example 17-2.

Example 17-2. The addClient( ) method of the server-side ConnectionTester component

 ConnectionTester.prototype.addClient = function (client) {   var proxy = this.getProxy(client);   proxy.echoRequest = function ( ) {     return;   }; }; 

The getProxy( ) method returns the main object on the client as though it had been obtained this way:

 var proxy = client.pfcs.ConnectionTester.main; 

The net result is the creation of a method at:

 client.pfcs.ConnectionTester.main.echoRequest 

The echoRequest( ) method simply returns without performing any other action. Example 17-1 just reports the ping results in a TextArea.

Example 17-3 shows the pingClient( ) function of the server-side ConnectionTester class. It is meant to be called repeatedly using setInterval( ) ; it both gets the round-trip time for the last ping and sends a new ping to the client using client.ping( ) . The results are simply output using a trace( ) call.

Example 17-3. The server-side ConnectionTester.pingClient( ) method

 ConnectionTester.prototype.pingClient = function (client) {   var proxy = this.getProxy(client);   if (typeof proxy.pingCount == "undefined"  proxy.pingCount == 0) {     proxy.pingCount = 0;     proxy.buffer = [];   }   else {     proxy.pingCount++   }   if (proxy.pingCount > 0) {     var stats = client.getStats( );     proxy.buffer.push( stats.ping_rtt );   }   if (proxy.pingCount > 20) {     clearInterval(proxy.intervalID );     var sum = 0;     for (var i = 0; i < proxy.buffer.length; i++) {       sum += proxy.buffer[i];       trace("ping_rtt: " + proxy.buffer[i]);     }     trace("Average: " + (sum/proxy.buffer.length));   }   client.ping( ); }; 

The following statements can be executed within any method of the ConnectionTester class and will start pinging the client by repeatedly calling the component's pingClient( ) method:

 var proxy = this.getProxy(client); delete proxy.pingCount = 0; proxy.intervalID = setInterval(this, "pingClient", 500, client); 

I ran some tests in different scenarios to measure the round-trip time using various approaches. The results are shown in Table 17-1.

Table 17-1. Round-trip times for packet transmission

Test description	Min. round-trip	Max. round-trip	Avg. round-trip
Pinged a remote FlashCom test machine from home using the ping program	70 ms	81 ms	72 ms
Called a remote method on FlashCom from a Flash client	70 ms	90 ms	76 ms
Used FlashCom to call a remote method on the same Flash client	62 ms	125 ms	78 ms
Used client.ping( )	68 ms	132 ms	76 ms

The command-line ping program is doing its best to measure the round-trip time using ICMP packets, and the test program is doing the best it can to measure the round-trip time using TCP packets. The client.ping( ) method is also using TCP packets, but unlike the remote method test, its packets are sent by FlashCom to the client with the highest possible priority. The test program also incurs the extra overhead of calling a remote method on a FlashCom Server. During the test, I made sure the test FlashCom Server was not doing anything elsea highly unrealistic scenario.

In real applications, you will have to choose how you want to determine latency. Remote methods are queued for each application instance on the server. If an application instance is handling a lot of remote method calls, there may be a significant delay while other calls are invoked. So a remote method test is a good way to measure the response time for queued processes such as the call( ) method or NetStream.send( ) and SharedObject.send( ) . For something closer to the latency that will be experienced by streams, use a client.ping( ) test, as it should not suffer from message queuing delays.

If you run a large number of tests in which you compare the results of an ICMP packet-based ping test against a TCP-based remote method test, you may notice that you get slightly lower average times with a narrower range of individual round-trip times from the ICMP-based test. One reason for this is that ICMP is used to deliver messages about network status and is an unreliable protocol. "Unreliable" just means that not every packet must be delivered. TCP is a reliable protocol designed to carry data between hosts . TCP is reliable because any packets that are dropped in transit are resent. Messages that contain packets that have been resent will be delayed longer than messages composed entirely of packets that that did not have to be resent. So a TCP-based test will often contain a number of round-trip values that are much higher than the average.

There is a problem with trace output in the App Inspector ( app_inspector.swf ). The output may appear out of sequence. When debugging, output appearing out of sequence can be a real problem. Often it is necessary to include a millisecond timestamp in each message to see the real sequence of events. For example:

 trace((new Date()).getTime(  ) +             " client disconnected."); 

17.1.2. Clock Synchronization

Sometimes we want to do more than calculate an average latency. For example, in some computer games , the clocks of each client must be closely synchronized with the time on the server. Of course, Flash cannot reset the client's clock, but a test can be used to determine the difference in UTC time in milliseconds between the two clocks. Once the difference is known, the server time can always be estimated using the client's clock and the difference between the two clocks.

A test to determine the difference in time between the client's and server's clocks is similar to performing a latency test using remote methods. In a case in which Flash is used to call a remote method on FlashCom, the primary steps are:

1. Get the current time and save it.

2. Request the current time from the server by calling a remote method.

3. The server receives the remote method call, and gets and returns the current time.

4. When the client-side result handler is called, the client time is taken again and stored.

As a result, there are three times: sent time, server time, and received time. The latency is calculated as:

 latency = (receivedTime - sentTime)/2; 

Ideally, the four steps should be repeated five or more times with pauses of a few seconds between receiving a response from the server and sending another request. With five or more latency values, received times, and server times, a simple way to calculate the difference between clocks is to average all the times and then calculate:

 clientServerDelta = avgServerTime - avgReceivedTime + avgLatency; 

However, there is a problem with using simple averages, because there may be a small number of latency values that have been inflated by TCP packet resends. Exclude them from the averages with the following technique:

1. Sort all the latency values.

2. Select the median value.

3. Multiply the median value by 1.5 to determine an upper-limit value.

4. Remove all data with latencies higher than the upper limit from the data set.

As an alternative, the standard deviation of the latency values can be calculated, and all samples above one standard deviation from the median can be removed.

Once any out-of-range latency values have been discarded, the client-to-server delta can be calculated using the remaining averages. The ConnectionTester component contains an implementation of this technique that can be used when a client first connects. Even better would be to check latency periodically as long as the connection to FlashCom is open .

The algorithm is based on an article by Zachary Booth Simpson named "A Stream-based Time Synchronization Technique For Networked Computer Games," available at:

http://www.mine-control.com/zack/timesync/timesync.html

A related article, "Minimizing Latency in Real-Time Strategy Games" by Jim Greer and Zachary Booth Simpson, is available in Game Programming Gems 3 (Charles River Media).

In later sections of this chapter, I'll return to some techniques for minimizing some of the problems latency can cause.