Network Programming

[Previous] [Next]

Network programming refers to all operations that happen behind the scenes and involve cooperation between multiple applications running on multiple machines across a network. Although the user never sees these programs, they do the real work on the Internet and in other networked environments.

The server is the workhorse of network programming. A server is a program that provides services to other programs. A server waits for a request from another program, decodes that request, and sends a response. Because servers run unattended for days and weeks, they must be robust.

The program requesting a service from a server is a client. A client issues a request and waits for a response from the server.

A middleware application is a program that acts as a go-between for two other programs. Generally, a middleware program adds value to the transaction. An active content server that receives requests from a Web server and queries a database to fill the request is acting as middleware. It manages the communication between the Web server and the database server and adds value by adding HTML tagging to the database results.

A middleware application acts as both a client and a server. You might encounter multitier architectures in which multiple middleware applications broker and add value to communication across a network. You can easily program servers and middleware applications in OmniMark by using the connectivity libraries.

When embarking on a network-programming project, you will need to know a little bit about protocols. A protocol is simply an agreement between a client and a server about how they will communicate. If you use a common published protocol or publish your own protocol, you can enable any number of clients to communicate with your server. On the other hand, if you keep your protocol private (or even encrypted), you can help to secure your server against intrusion.

You need to know about two important types of protocol: transport protocols and application protocols.

Transport protocols are used to actually get messages safely across the network from one machine to another. TCP/IP is the transport protocol used on the Internet and is supported by OmniMark's network libraries and the TCPService and TCPConnection OMX components.

An application protocol is an agreement about what constitutes a message and what the message means. While disk files contain end-of-file markers, a network message is just a stream of bytes over an open connection. You have to look at the data itself to determine whether you have found the whole message. The OmniMark I/O Protocol library supports all the common methods of delimiting a message.

Once you have a complete message, you must decode it to see what it means and then generate and send the appropriate response. OmniMark is an ideal language for decoding network protocols. Its streaming features enable you to easily interpret a message and quickly formulate a response.

A Simple Server

The following program is a simple OmniMark server that returns the first line of a nursery rhyme when it receives a message naming the principal character of that rhyme.

 declare catch server-die include "omioprot.xin" include "omtcp.xin" include "builtins.xin" define switch function TCPservice-is-working read-only TCPService the-service as do when TCPServiceIsInError the-service local stream errorReport variable initial-size 0 TCPServiceGetStatusReport the-service into errorReport log-message "TCPService Error:" repeat over errorReport log-message errorReport again return false else return true done process local TCPService my-service set my-service to TCPServiceOpen at 5432 throw server-die unless TCPService-is-working my-service repeat local TCPConnection my-connection local stream my-response set my-connection to TCPServiceAcceptConnection my-service throw server-die unless TCPService-is-working my-service open my-response as TCPConnectionGetOutput my-connection using output as my-response submit TCPConnectionGetLine my-connection catch #program-error again catch server-die find "Mary%n" output "Mary had a little lamb%13#%10#" find "Tom%n" output "Tom, Tom, the piper's son.%13#%10#" find "Jack" | "Jill" "%n" output "Jack and Jill went up the hill.%13#%10#" find "die%n" output "Argh! Splat!%13#%10#" throw server-die 

A server operates rather like a telephone. First we place it in service by assigning it a telephone number. Then it must wait for a call. When a call comes, the server must answer the call, listen to the message, and make an appropriate response. The conversation can consist of a single exchange or multiple exchanges. When the conversation is over, the server hangs up and goes back to waiting for the next call.

We can break down the essential operation of a server into three phases:

• Start up Put the server in service
• Request loop Wait for calls, respond, and repeat
• Shut down Take the server out of service

Because a server runs for a long time and has to handle many requests, it has two overriding performance requirements:

• No matter what happens while servicing a request, the server must not crash. It must stay running.
• No matter what happens while servicing a request, the server must always return to a consistent ready state when the request is complete. (If the server were in a different state for each request, its responses would not be reliable.)

Let's look at how our sample server meets these requirements line by line.

 process local TCPService my-service set my-service to TCPServiceOpen at 5432 

This is the code that puts the server in service. It uses a TCPService OMX component to establish a service on port 5432 of the machine it is running on. The server's address will be the machine's network address combined with the port number. By using different ports, many different servers can run on the same machine concurrently.

 repeat local TCPConnection my-connection ... set my-connection to TCPServiceAcceptConnection my-service again 

This is the code that listens for an incoming call. TCPServiceAcceptConnection waits for a client to connect. When the server receives a connection, it returns a TCPConnection OMX component, which represents the connection to the client. The TCPConnection variable my-connection is declared inside the repeat loop so that the variable will go out of scope at the end of the loop, providing automatic closure and cleanup of the connection.

 repeat local TCPConnection my-connection ... set my-connection to TCPServiceAcceptConnection my-service ... submit TCPConnectionGetLine my-Conn again 

A TCPConnection OMX provides an OmniMark source so that the program can read data from the client. Reading data from a network connection, however, is different from reading from a file. While you can either read from or write to a file, a network connection is a two-way connection. OmniMark cannot detect the end of a message on a network connection the way it detects the end of a file. The connection stays open; there could always be more characters coming. For this reason, all network data communication requires a specific application protocol for determining the end of a message. OmniMark provides support for all the common application protocols used for this purpose through the I/O Protocol library. The TCP library uses the I/O Protocol library to support the common Internet protocols. This example uses a line-based protocol. In our request protocol, the end of a message is signaled by a line-end combination (ASCII 13, 10). The TCPConnectionGetLine function provides a source for line-end delimited data and closes that source when it detects a line end. We submit data from that source to our find rules, which will analyze the message and generate the appropriate response.

 repeat ... local stream my-response ... open my-response as TCPConnectionGetOutput my-connection using output as my-response submit TCPConnectionGetLine my-connection ... again 

Our TCPConnection OMX represents a two-way network connection. Not only must we get a source from it to read data, but we must also attach an output stream to it so that we can send data over the connection to the client. We do this with the TCPConnectionGetOutput function.

Once we prefix our submit with using output as my-response, our find rules read from and write to the network connection.

 find "Mary%n" output "Mary had a little lamb%13#%10#" find "Tom%n" output "Tom, Tom, the piper's son.%13#%10#" 

Ours is a line-based protocol, but line ends are different on different platforms (13, 10 on Windows, 10 on UNIX). OmniMark's %n is a normalized line end that will match either form. If you output %n, OmniMark will output the form appropriate to the platform the program is running on. Across a network—that includes machines from different platforms—we have to choose the appropriate form ourselves. Our protocol specifically requires 13, 10. But for matching purposes, we use %n so that even if the client forgets to send the appropriate line end sequence, we can still read the message. When we send, however, we explicitly send "%13#%10#" rather than "%n". This reflects an important maxim of network programming that reflects the so-called Netiquette guidelines: Be liberal in what you accept, conservative in what you send.

This is the find rule that detects the poison-pill message:

 find "die" output "Argh! Splat!%13#%10#" throw server-die 

To ensure an orderly shutdown, we provide a means of terminating our server by sending it a message to shut itself down. (In a production system, you might want to pick a slightly less obvious message for the poison pill.)

Shutting down the server is an exception to normal processing. We accomplish it by initiating a throw to a catch named server-die.

 process ... repeat ... again catch server-die 

We catch the throw to server-die after the end of the server loop. OmniMark cleans up local scopes on the way, ensuring a clean and orderly shutdown. Since we are at the end of the process rule, the program exits normally.

Error and Recovery

A server needs to continue running despite any errors that occur in servicing a particular request. However, a server should shut down if it cannot run reliably. The following code provides for both situations:

 define switch function TCPservice-is-working read-only TCPService the-service as do when TCPServiceIsInError the-service local stream errorReport variable initial-size 0 TCPServiceGetStatusReport the-service into errorReport log-message "TCPService Error:" repeat over errorReport log-message errorReport again return false else return true done process set my-service to TCPServiceOpen at 5432 throw server-die unless TCPService-is-working my-service repeat ... set my-connection to TCPServiceAcceptConnection my-service throw server-die unless TCPService-is-working my-service ... catch #program-error again 

If there is an error in processing a request, OmniMark initiates a throw to #program-error. We catch the throw at the end of the server loop, thereby providing for automatic cleanup of any resources used in servicing the request in progress, and assuring that the server returns to its stable ready state. (In this example, we've made no attempt to rescue the specific request in which the error occurred. In a production server you would want to provide such error recovery, but make sure you always have a fallback that aborts the current request and returns to a stable ready state.)

In the unlikely event that something goes wrong with the TCPService component, you can't do much except shut down the server. The current version of the TCP library does not support catch and throw, so you have to do an explicit test for errors in the service whenever you use the library. If you detect an error, log it and then throw to server-die to shut down the server.

This simple server program has everything you need for a robust and usable production server. You need to adapt the code to the protocol you are using, but apart from that, everything else is just regular OmniMark programming once input and output are bound to the connection.

A Simple Client

Any client program written in any language can use our server, as long as it knows the protocol. Here is a simple client written in OmniMark:

 process local TCPConnection my-connection local stream my-request set my-connection to TCPConnectionOpen on "localhost" at 5432 open my-request as TCPConnectionGetOutput my-connection using output as my-request output #args[1] || "%13#%10#" close my-request repeat output TCPConnectionGetCharacters my-Conn exit unless TCPConnectionIsConnected my-Conn again 

This client is called with the name of the nursery-rhyme character on the command line and prints out the line it receives from the server. Let's go through it line by line.

 set my-connection to TCPConnectionOpen on "localhost" at 5432 

Like the server program, the client uses a TCPConnection OMX component to create a connection. Unlike the server, the client does not require a TCPService component, as it is not establishing a service but simply making a connection to a service established elsewhere. The client takes a more active role than the server, however. While the server waits for a call, the client must take the initiative and make a call. It does this with the TCPConnectionOpen function. The TCPConnectionOpen function takes a network and port address for a server, and when the connection is made it returns a TCPConnection OMX, which we can write to and read from just as we did in the server program.

 repeat output TCPConnectionGetCharacters my-Conn exit when TCPConnectionIsInError my-Conn again 

When we read the data returned from the server we actually have two choices. Since our protocol is line-based, we could use TCPConnectionGetLine to read the response. But we also know that the server will drop the connection as soon as it finishes sending data. (This behavior is part of our protocol as well.) So we choose to keep reading data until the connection is dropped. This way we will get at least partial data even if something goes wrong and the server never sends the end of line. Be conservative in what you send and liberal in what you accept.

Clients for Common Servers

Most of the client programs you write in OmniMark will probably be for well-known servers such as HTTP, FTP, or SMTP/POP. OmniMark's connectivity libraries provide direct support for these and other common protocols, greatly simplifying the task of retrieving data from these servers.