Dispatching Communication Requests | Inside Server-Based Applications (DV-MPS General)

One decision you will need to make at the outset of any server-based development is what sort of dispatching mechanism you intend to use to handle client requests. This decision affects everything that you do. One method is to assign a single thread to handle each request, cradle to grave. This option has the advantage of not requiring any artificial state management. Your service will always know which client it is talking to for the life of the request. This is a good strategy in the following situations:

The length of time from the beginning to the end of the request is short. In this case, the overhead of managing the state so the task can be parceled out to worker threads is simply not worth the effort.

The number of simultaneous clients is known and relatively small. In this case, creating enough threads is not likely to be a problem. Of course, there is a limit to the number of threads that the operating system can reasonably support, but this number is not fixed and is likely to be higher than you think.

In other situations, you might be better off allowing each individual request for a given session to be handled by whatever thread happens to be available.

Managing Client State

One of the complications of assigning one thread per request is the need to keep track of state information apart from any single thread. When a single thread manages the entire client/server interaction, state information can be stored in stack variables within the thread. Sometimes, however, this is not possible.

One way to manage the state of each client is to encapsulate every transmission of data into a structure that contains what I call a server handle. You can use a server handle to associate any particular request with state information stored in the server. A common structure I use looks like this:

 typedefstruct_DATA  _  PACKET{ WORDServerHandle; WORDServiceCode; WORDReturnCode; DWORDDataLen; DWORDBufLen; charBuffer[MAX_DATA_SIZE]; }DATA_PACKET;

The client initially sets the ServerHandle member to 0. Upon receiving a packet with a ServerHandle of 0, the server initiates a new session. The handle can then be used as an offset into whatever state information the server needs to maintain. The next member, ServiceCode , is set by the client to tell the server exactly what service is required. For instance, in one server-based application I recently completed, one service code requested verification of serial number information and another requested version information.

ReturnCode is set by the server to indicate success or failure. The specifics of the return code system are unimportant as long as client and server agree. Both the client and the server set DataLen , indicating the useful length of the Buffer member. The client sets BufLen to the number of bytes allocated for Buffer .

When you talk with another person, extended silences that are not mutually agreed upon can indicate trouble. So it is with client/server communication. In addition to agreement on the data packet sent back and forth, both the client and the server must agree on the sequence of sending and receiving data. They must understand when to "speak" and when to "listen." Client and server must also agree on how long to wait for responses before presuming an error occurred.

Managing Failures

When applications run on a single machine, there is relatively little chance that two components on the machine will fail to communicate with each other. For instance, if someone reboots the machine, both components stop functioning. When the system is extended to multiple machines, there is a much greater chance of a failure that directly affects only one party. For instance, if the purpose of a client/server interaction is to update a record into a database, the following scenarios are possible:

The client submits the record. The server fails before performing the update or responding to the client.

The client submits the record. The server successfully performs the update but then fails before responding to the client.

In both cases, the client has initiated the operation but does not hear back. What choices does the client have? It could resend the update, hoping that it will work the second time, or it could simply log the error. Neither option is perfect. The correct solution probably depends upon the application. Modern Operating Systems by Andrew S. Tanenbaum (Prentice Hall, 1992) has a good discussion about the semantics of failure. Easy to implement are "at most once" semantics, meaning that an operation will be performed 0 times or 1 time. "At least once" semanticsmeaning an operation will be performed 1 or more timesare also easy to implement. The preferred "exactly once" semanticsan operation will be performed exactly onceare much more difficult to implement and will depend on the specifics of the system.

Dispatching Requests

One problem that is present regardless of the protocol used between client and server is dispatching requests from clients.

Scenario 1: Using Events

Event objects, which I discussed in Chapter 2, can have multiple threads waiting for the event object to be signaled. When set properly, with multiple threads waiting for the event object, one thread will be notified of the signaled event and the other threads will continue waiting. This can be used to our advantage.

The idea is this: Create some number of worker threads. The first task of each thread is to enter a loop controlled by WaitForSingleObject and wait for the event to be signaled. As client requests come in via a WinSock connection, the event is signaled. One of the threads is notified that the event has been signaled. This thread gets the socket, signals the dispatch thread that it has done so, and goes about its work. After the work for that client request is finished, the worker thread returns to the top of the loop and again waits for the event to become signaled. The worker threads wait without using a great deal of system-processing power.

This approach offers two advantages. First, the threads are created as the service begins, and they run throughout the life of the service. This saves the processing load caused by the constant creation and destruction of threads. Thread creation is less costly than process creation, but it is not free. Second, this approach allows us to control the number of threads. In some cases clients might need to wait while threads finish up other requests. This lag can be addressed by allowing the monitoring of thread usage and configuring thread usage to the level required to give acceptable performance while causing the least inconvenience to clients waiting for service.

The disadvantage of this approach is that as written, there is no allowance for a growing number of threads. This could be solved by controlling the maximum number of threads with a dynamically configurable value. While in practice adding threads would be easy, killing off running threads might be a bit more difficult.

CommService.h and CommService.cppshown in Listings 12-1 and 12-2, respectivelydefine a class that uses CppService as a base class and adds support for receiving and dispatching requests received via WinSock.

Listing 12-1

CommService.h

 #defineNUM_WORKERS4 classCCommService:publicCPPService { private: voidWSCommDispatcher(THREADPROCProc); voidSetLastSocket(SOCKETs); SOCKETGetLastSocket(); HANDLEm_hGotSocket; HANDLEm_hEvent; HANDLEm_hThreads[NUM_WORKERS]; DWORDm_dwThreadIDs[NUM_WORKERS]; public: staticDWORD__stdcallThread(LPVOIDt); virtualvoidDoWork(SOCKETs,WORDThreadNum); intGetServerPort(); voidSetServerPort(inttServerPort); intm_ServerPort; SOCKETLastSocket; CCommService(char*tsvcName,char*tdispName, DWORDtsvcStart=SERVICE_DEMAND_START, boolCreateMsgPump=true); ~CCommService(); virtualvoidRun(); virtualvoidOnInstall(); };

Listing 12-2

CommService.cpp

 //CommService.cpp:Communicationsservice,usingTCP/IP //andEvents // #include"stdafx.h" //------------------------------------------------------------------- //Examplesimplemainroutine intmain(intargc,char*argv[]) { CCommServiceMyCommService("CommService", "CommunicationsTestService"); //Wecoulddosomethinglikethefollowingifweneededto. //MyCommService.SetServerPort(5555); //Thisexampleusesthedefault. //Havewehandledtheargumentsanddonean //installoruninstall? if(MyCommService.ParseArguments(argc,argv)==false) { //Ifnot,starttheservicerunning. MyCommService.StartService(); } return0; } //------------------------------------------------------------------- //Theheartoftheclass,theWinSockdispatcher voidCCommService::WSCommDispatcher(THREADPROCProc) { //Declareandinitialize. BOOLfConnected=FALSE; DWORDdwThreadID=(DWORD)0; HANDLEhThread=INVALID_HANDLE_VALUE; intaddrLen; interror; structsockaddr_insrvSocketAddr; structsockaddr_incliSocketAddr; chardiag[255]; WSADATAwsaData; SOCKETsrvSocket; SOCKETcliSocket; error=WSAStartup(WS_VERSION_REQD,&wsaData); if(error!=0) { return; } if(LOBYTE(wsaData.wVersion)<2) { sprintf(diag,"\nVersion%d.%dnotsupported", LOBYTE(wsaData.wVersion), HIBYTE(wsaData.wVersion)); LogEvent(EVENTLOG_ERROR_TYPE,EVMSG_DEBUG,diag); return; } srvSocket=socket(AF_INET,SOCK_STREAM,0); if(srvSocket==INVALID_SOCKET) { sprintf(diag,"SERVER:WindowsSocketError%d-" "Couldn'tcreatesocket", WSAGetLastError()); LogEvent(EVENTLOG_ERROR_TYPE,EVMSG_DEBUG,diag); WSACleanup(); return; } srvSocketAddr.sin_family=AF_INET; srvSocketAddr.sin_addr.s_addr=INADDR_ANY; srvSocketAddr.sin_port=GetServerPort(); if(bind(srvSocket,(LPSOCKADDR)&srvSocketAddr, sizeof(srvSocketAddr))==SOCKET_ERROR) { sprintf(diag,"SERVER:WindowsSocketError%d-" "Couldn'tbindsocket", WSAGetLastError()); LogEvent(EVENTLOG_ERROR_TYPE,EVMSG_DEBUG,diag); WSACleanup(); return; } if(listen(srvSocket,SOMAXCONN)==SOCKET_ERROR) { sprintf(diag,"SERVER:WindowsSocketError%d-" "Couldn'tlistenonsocket", WSAGetLastError()); LogEvent(EVENTLOG_ERROR_TYPE,EVMSG_DEBUG,diag); WSACleanup(); return; } while((IsRunning())) { addrLen=(sizeof(cliSocketAddr)); cliSocket=accept(srvSocket, (LPSOCKADDR)&cliSocketAddr,&addrLen); if(cliSocket==INVALID_SOCKET) { sprintf(diag, "SERVER:WindowsSocketError%d-" "accept()gotinvalidsocket", WSAGetLastError()); LogEvent(EVENTLOG_ERROR_TYPE,EVMSG_DEBUG,diag); WSACleanup(); return; } SetLastSocket(cliSocket); SetEvent(m_hEvent); while(WaitForSingleObject(m_hGotSocket, 1000)==WAIT_TIMEOUT) { if(!IsRunning()) { WSACleanup(); sprintf(diag,"SERVER:Exiting"); LogEvent(EVENTLOG_INFORMATION_TYPE, EVMSG_DEBUG,diag); return; } } ResetEvent(m_hGotSocket); } WSACleanup(); sprintf(diag,"SERVER:Exiting"); LogEvent(EVENTLOG_INFORMATION_TYPE,EVMSG_DEBUG,diag); return; } //------------------------------------------------------------------- //Theconstructor CCommService::CCommService(char*tsvcName, char*tdispName, DWORDtsvcStart, boolCreateMsgPump) :CPPService(tsvcName,tdispName,tsvcStart,CreateMsgPump) { DWORDdwCreationFlags=0; //Createtheevent;defaultsecurity,NOTmanualReset, //NONsignaledoriginally,NOTnamed,sincejustusedby //thisprocess. m_hEvent=CreateEvent(NULL,FALSE,FALSE,NULL); if(m_hEvent==NULL) { LogEvent(EVENTLOG_ERROR_TYPE, EVMSG_DEBUG, "Can'tcreateevent!"); } //Createtheevent;defaultsecurity,ManualReset, //NONsignaledoriginally,NOTnamed,sincejust //usedbythisprocess. m_hGotSocket=CreateEvent(NULL,TRUE,FALSE,NULL); if(m_hEvent==NULL) { LogEvent(EVENTLOG_ERROR_TYPE, EVMSG_DEBUG, "Can'tcreateGotSocketevent!"); } m_ServerPort=5043;//adefaultport for(WORDloop=0;loop<NUM_WORKERS;loop++) { if((m_hThreads[loop]=CreateThread(NULL, 0,Thread,(LPVOID)loop, dwCreationFlags, &m_dwThreadIDs[loop]))==NULL) { LogEvent(EVENTLOG_ERROR_TYPE, EVMSG_DEBUG, "Can'tcreatethread!"); } } } CCommService::~CCommService() { if(m_hEvent!=NULL) { CloseHandle(m_hEvent); } if(m_hGotSocket!=NULL) { CloseHandle(m_hGotSocket); } for(WORDloop=0;loop<NUM_WORKERS;loop++) { if(m_hThreads[loop]!=NULL) { CloseHandle(m_hThreads[loop]); } } } //------------------------------------------------------------------- //Usedtopassasocketfromthedispatchertotheworkerthread SOCKETCCommService::GetLastSocket() { returnLastSocket; } //------------------------------------------------------------------- //Usedtopassasocketfromthedispatchertotheworkerthread voidCCommService::SetLastSocket(SOCKETs) { LastSocket=s; } //------------------------------------------------------------------- //Settheportnumbertheservershoulduse. voidCCommService::SetServerPort(inttServerPort) { m_ServerPort=tServerPort; } //------------------------------------------------------------------- //Gettheportnumbertheserverisusing. intCCommService::GetServerPort() { returnm_ServerPort; } //------------------------------------------------------------------- //Theroutinecalledtoruntheservice voidCCommService::Run() { WSCommDispatcher(Thread); m_isRunning=false; //Givethreadsachancetodie. Sleep(2000); return; } //------------------------------------------------------------------- //Starttheserviceuponinstallation. voidCCommService::OnInstall() { ::StartThisService(ServiceName()); } //------------------------------------------------------------------- //Thefunctionthatdoestheactualwork;thisshouldbeoverridden voidCCommService::DoWork(SOCKETs,WORDThreadNum) { chardiag[255]; chardata[1024]; BOOLfSuccess; intlen; staticintblksSent=0; intbytesRead; intmaxData=512; bytesRead=0L; do{ memset((void*)data,'\0',1024); if((len=recv(s, ((char*)data)+bytesRead, maxData,0))==SOCKET_ERROR) { fSuccess=FALSE; wsprintf(diag,"SOCKETERROR",data); LogEvent(EVENTLOG_ERROR_TYPE, EVMSG_DEBUG,diag); } else { if(len) { fSuccess=TRUE; bytesRead+=(DWORD)len; blksSent++; data[200]='\0';//Onlytrytologthefirst //200charactersorless. wsprintf(diag, "Clientsays:%stoThread%d", data,ThreadNum); LogEvent(EVENTLOG_INFORMATION_TYPE, EVMSG_DEBUG,diag); } } }while(IsRunning()&&len!=SOCKET_ERROR&&len!=0); //Thisisjusttosimulatedoingsomethingthattakes2seconds. Sleep(2000); closesocket(s); } //------------------------------------------------------------------- //Thethreadroutine;usedtocallDoWorkcorrectly DWORD__stdcallCCommService::Thread(LPVOIDt) { DWORDret; WORDThreadNum; SOCKETs; CCommService*p_this; p_this=(CCommService*)m_this; ThreadNum=(WORD)(t); do{ while((ret=WaitForSingleObject(p_this->m_hEvent,1000))==WAIT_TIMEOUT&& p_this->IsRunning()==TRUE) { ; } if(p_this->IsRunning()==FALSE) { break; } if(ret==WAIT_OBJECT_0) { s=p_this->GetLastSocket(); SetEvent(p_this->m_hGotSocket); p_this->DoWork(s,ThreadNum); } }while(ret==WAIT_TIMEOUTret==WAIT_OBJECT_0); return(0); }

This example required only minor changes to CppService. For convenience, I added a simple function, IsRunning . This is just an access function for the m_isRunning member variable implemented in CppService.h as follows. (The full source for this version of CppService is on the companion CD-ROM.)

 //AddedforChapter12 BOOLIsRunning() { if(m_CreateMsgPump) { MessagePump(&m_isRunning); } return(m_isRunning!=0); }

An overview of new class members Take a look at CommService.h in Listing 12-1. You'll see several new functions and data members. WSCommDispatcher is the function used to receive TCP/IP requests from clients and dispatch them to the worker threads. SetLastSocket and GetLastSocket work together to allow the newly dispatched thread to get the socket that it needs to handle the request. There are two event handles declared. The event handle m_gotSocket is used to signal that the worker thread has gotten the socket associated with the request, and the event handle m_hEvent is used to dispatch just a single thread. Two arrays, m_hThreads and m_dwThreadIDs , are dimensioned based on the number of worker threads to be used. These two arrays hold the thread handles and thread IDs, respectively.

Thread is a static member function that is the main routine for the worker threads. Recall that C++ class member functions always pass an implicit this pointer as the first unseen parameter. To allow the Thread function to be used as a thread procedure, we must declare it as a static function. DoWork is declared as virtual and is the function that actually performs the work of the server. This function accepts as a parameter the socket that resulted in the thread being dispatched and the thread number that called the function. The thread number is used in this case to permit the DoWork function to place the thread number in the event log.

The CCommService constructor The constructor for CCommService first creates the two event handles. The first event handle, m_hEvent , is created with default security and automatic reset. It is originally nonsignaled and not named. The automatic reset is important. If multiple threads are waiting on the event handle and automatic reset is enabled, only one thread will be notified when the event is signaled. The second event handle, m_hGotService , is created with the same parameters, except that it will be manually reset.

Next a default server port is set. This is an arbitrary value above 5000; in our example that value is 5043. Port numbers below 5000 are reserved for other uses. Finally, the constructor calls CreateThread in a loop until NUM_WORKERS threads are started. The destructor simply closes the handles opened in the constructor.

Other class member functions Run is overridden to actually control what the service does. In this case what Run does is quite simple:

 voidCCommService::Run() { WSCommDispatcher(Thread); m_isRunning=false; //Givethreadsachancetodie. Sleep(2000); return; }

The main task of Run is to call WSCommDispatcher . WSCommDispatcher is responsible for virtually all of the WinSock functionality. The following series of steps is required for acting as a WinSock server:

WinSock must be initialized by calling WSAStartup .

The server socket must be created using socket .

The server socket must be bound, calling bind .

The server must call listen to initiate listening on the port.

The server must call accept , returning the socket to be used for sending and receiving data.

WSCommDispatcher first calls WSAStartup :

 intWSAStartup(WORDwVersionRequested, LPWSADATAlpWSAData);

This function initiates the WinSock library. It also allows you to specify the highest version of WinSock that the program can accept in the first parameter, wVersionRequested . You should place the minor version in the high-order byte of wVersionRequested and the major version in the low-order byte. The second parameter is a pointer to a WSADATA structure:

 typedefstructWSAData{ WORDwVersion; WORDwHighVersion; charszDescription[WSADESCRIPTION_LEN+1]; charszSystemStatus[WSASYS_STATUS_LEN+1]; unsignedshortiMaxSockets; unsignedshortiMaxUdpDg; charFAR*lpVendorInfo; }WSADATA;

Generally, you can ignore most of this information. The first and second data members are the WinSock version that the WinSock library expects the caller to use and the highest version supported. These are often the same value. The character arrays can be filled with somewhat useful information, though I was at one point surprised to find an implementation that reported "Running (Duh!)" in szSystemStatus . After that episode, I generally avoided doing anything with these values. You should ignore the last three members of this structure for WinSock 2 and greater; these members are not used for Windows 2000 or Windows NT 4. WSCommDispatcher checks the version of WinSock and verifies that it is version 2 or later.

Next the server socket is obtained through a call to s ocket :

 SOCKETsocket(intaf, inttype, intprotocol);

The first parameter, af , is the address family. This is normally the constant AF_INET. WinSock 2 supports other address families, such as AF_IPX and AF_ APPLETALK. The second parameter is type , which can be SOCK_STREAM or SOCK_ DGRAM. SOCK_STREAM provides a sequenced, reliable communication medium between the client and server. SOCK_DGRAM supports connectionless, unreliable communication. In most cases SOCK_STREAM is appropriate. The final parameter, protocol , is set to 0 in CCommService to use the default protocol. The protocol selected by default depends on the value passed in type .

NOTE
Calling one type of communication reliable and the other unreliable is not a value judgment. Reliable means that the packets are sequenced and that there is a mechanism for retrying failed communication. Unreliable communication does not provide this, and is appropriate for brief, stateless communication.

WSCommDispatcher next calls the WinSock function bind.

 intbind(SOCKETs, conststructsockaddrFAR*name, intnamelen);

The first parameter to bind is the socket obtained with the call to socket . In the case of a TCP/IP socket, the second parameter is a pointer to a sockaddr_in structure:

 structsockaddr_in{ shortsin_family; u_shortsin_port; structin_addrsin_addr; charsin_zero[8]; };

WSCommDispatcher sets sin_family to AF_INET, sin_port to the return from the C CommService member function GetServerPort , and sin_addr to INADDR_ANY, meaning that any address on the host can be used, in the case of a multihomed host (a host with more than one address). The last parameter to bind is namelen , which is set to the size of the sockaddr_in structure.

If bind succeeds, the next step is for WSCommDispatcher to call listen to place the server socket in a state to allow it to listen for incoming connections.

 intlisten(SOCKETs, intbacklog);

The first parameter is the server socket previously obtained through a call to socket . The second parameter, backlog , specifies the maximum length of the queue for incoming connections. In Listing 12-2, backlog is set to SOMAXCONN to allow the maximum number of backlogged connections to be set by the underlying service provider. If the number specified for backlog is out of range, the valid number nearest the requested value is used.

Once the socket has been set to listen for connections, the WSCommDispatcher function enters the loop shown in the following code fragment:

 while((IsRunning())) { addrLen=(sizeof(cliSocketAddr)); cliSocket=accept(srvSocket, (LPSOCKADDR)&cliSocketAddr,&addrLen); if(cliSocket==INVALID_SOCKET) { sprintf(diag,"SERVER:WindowsSocketError%d-" "accept()gotinvalidsocket", WSAGetLastError()); LogEvent(EVENTLOG_ERROR_TYPE,EVMSG_DEBUG,diag); WSACleanup(); return; } SetLastSocket(cliSocket); SetEvent(m_hEvent); while(WaitForSingleObject(m_hGotSocket, 1000)==WAIT_TIMEOUT) { if(!IsRunning()) { return; } } ResetEvent(m_hGotSocket); }

This loop is the controlling loop for the WinSock monitoring program. The loop is controlled by the return from IsRunning , a Boolean function that returns false if the service is stopped. At the top of the loop, accept is called:

 SOCKETaccept(SOCKETs, structsockaddrFAR*addr, intFAR*addrlen);

This call is similar to bind . The second parameter is a pointer to a buffer that is filled with address information about the client. This is the socket that will be used for further communication with the client.

At this point, the problem is this: How do we activate a waiting worker thread and make that thread aware of the socket? Recall that the dispatch mechanism that we use has all worker threads initially wait on m_hEvent , the event handle set for automatic reset. When we signal m_hEvent , one of the threads waiting on that object will be activated. Thus, rather than polling for an available thread, the thread is assigned by virtue of being one of the threads waiting on this event. We still need to get the socket to the worker thread. Since we do not know which worker thread might be activated, we need to have a way to pass the socket to whatever thread becomes active.

To do this, we call SetLastSocket , a method of CCommService . This method quickly sets a member of the class to the socket passed and sets the m_hEvent event. This is the signal for one of the worker threads to be fired. The worker thread is sitting in the loop in the following code fragment:

 do{ while((ret=WaitForSingleObject(p_this->m_hEvent, 1000))==WAIT_TIMEOUT&& p_this->IsRunning()==TRUE) { ; } if(p_this->IsRunning()==FALSE) { break; } if(ret==WAIT_OBJECT_0) { s=p_this->GetLastSocket(); SetEvent(p_this->m_hGotSocket); p_this->DoWork(s,ThreadNum); } }while(ret==WAIT_TIMEOUTret==WAIT_OBJECT_0);

Once m_hEvent is signaled, the thread tests several conditions. If the return from WaitForSingleObject indicates the event was signaled, the thread gets the last socket by calling the GetLastSocket method. The thread then sets the m_hGotSocket event, which acts as a signal for the loop in WSCommDispatcher to reset m_hGotSocket and continue trying to accept connections.

At the bottom of the loop inside the Thread function in Listing 12-2 is a call to DoWork , the function that actually does the processing. In this simple example, all that DoWork does is read from the socket passed from Thread and log the text in the event log. To read from the socket, the WinSock function recv is called.

 intrecv(SOCKETs, charFAR*buf, intlen, intflags);

The first parameter is the socket to receive data on. The data is written to buf for a maximum length of len bytes. The final parameter, flags , can be MSG_PEEK to read the incoming data but not remove it, or MSG_OOB, meaning that recv should process out-of-band data.

Possible enhancements to CCommService While CCommService is a workable solution, we can enhance it in several ways. First, if WSCommDispatcher signals m_hEvent and no worker threads are available, the processing in that method will stop, possibly losing incoming connections. A solution that would also solve another problemthe need to grow the worker thread poolwould be to create a semaphore that would be incremented each time a worker thread starts operation and decremented as the thread begins to await another task. If WSCommDispatcher notices that an incoming connection arrives and no thread is waiting (determined by checking the semaphore), it could create an additional thread.

Another optimization would be to further isolate the protocol within the class. Currently two methods of the class use the WinSock functions. By abstracting the communication in a class (as we will do shortly for client-side communication), all WinSock-specific functionality could be isolated within the external class, thus allowing a single class to support (for example) both WinSock and named pipes.

Scenario 2: Using I/O Completion Ports

I/O Completion Ports (IOCPs) are an alternative to the approach used in Listing 12-2. IOCPs have a well-deserved reputation for being difficult to implement. Part of the complexity comes from the fact that the preparation of an operation and the handling of the results occur in two different places. Adding to this complexity are the obscure techniques required to mix synchronous I/O and IOCPs and ensure that the threads handling I/O can get the proper context.

Listings 12-3 and 12-4 contain NPService.h and NPService.cpp, respectively. The service created by compiling these modules communicates with a client by using named pipes and IOCPs. The purpose of the communication in this case is to change the duration, timing, and frequency of a beep that is part of the main processing loop for the service. What's most important about this example is that it demonstrates a way to allow the operating parameters of a service to be modified either on the same machine or on any machine that is accessible across the network.

Listing 12-3

NPService.h

 #defineREQ_DURATION1 #defineREQ_FREQ2 #defineREQ_PAUSE3 typedefstruct_DATA_PACKET{ WORDServerHandle; WORDServiceCode; WORDReturnCode; DWORDDataLen; DWORDBufLen; charBuffer[1024]; }DATA_PACKET; structOVERLAPPED_PLUS{ OVERLAPPEDoverlapped; DATA_PACKETdataPacket; }; classCNPService:publicCPPService { private: CRITICAL_SECTIONm_cs; DWORDm_dwPause; DWORDm_dwDuration; DWORDm_dwFreq; charm_szPipeName[128]; public: HANDLEInitializeThreads(void); staticDWORD__stdcallNPCommDispatcher(LPVOIDProc); staticDWORD__stdcallThread(LPVOIDh); virtualvoidDoWork(HANDLEhPipe,WORDThreadNum); LPSTRGetPipeName(LPSTRszBuffer) { strcpy(szBuffer,m_szPipeName); return(szBuffer); } voidSetPipeName(LPSTRszNewPipeName) { strncpy(m_szPipeName,szNewPipeName,128); } CNPService(char*tsvcName,char*tdispName, DWORDtsvcStart=SERVICE_DEMAND_START, boolCreateMsgPump=true); ~CNPService(); virtualvoidRun(); virtualvoidOnInstall(); };

Listing 12-3 first defines some constants used by the client to specify the service requested. In this case, setting the duration of the beep, the frequency of the beep, and the length of the pause are each separate operations. It is important to minimize round trips to the service, especially when communicating across a network. In a production application, all variables would be changed in a single operation.

Next is a definition of a structure named DATA_PACKET. This structure contains information that allows us to establish a client's context on the server, as well as allowing us to establish a service code for a particular request. OVERLAPPED_PLUS requires some explanation.

When we perform asynchronous I/O operations, we use a structure named OVERLAPPED to allow the operating system to complete the operation asynchronously.

 typedefstruct_OVERLAPPED{ DWORDInternal; DWORDInternalHigh; DWORDOffset; DWORDOffsetHigh; HANDLEhEvent; }OVERLAPPED;

The first two members are used internally and should not be touched. The third and fourth members, Offset and OffsetHigh , specify where in a file the operation should take place. These offsets are ignored when the OVERLAPPED structure is used for named pipes or completion ports. The final member, hEvent , must be set to an event object created with CreateEvent . The event object must be a manual-reset event.

The OVERLAPPED_PLUS structure contains an OVERLAPPED structure plus a DATA_PACKET structure. This is a fairly common trick to allow asynchronous operations to keep track of a context. Often, only a pointer to a context object is placed after the OVERLAPPED structure. I decided for illustrative purposes to include the entire structure.

The class declaration for the CNPService class contains some new elements over and above those in the CPPService class. The first is m_cs , a critical section object. This critical section will be used to serialize access to the next three data members, m_dwPause , m_dwDuration , and m_dwFreq . These data members are used to control the portion of the Run method that performs the simple task of looping around a Beep function call. The pipe name is stored in m_szPipeName . Several new methods are also added to the class declaration. These are fleshed out in Listing 12-4.

Listing 12-4

NPService.cpp

 //NPService.cpp:Definestheentrypointfortheconsoleapplication // #include"stdafx.h" #defineMAX_CLIENTS100 #defineMAX_DATA_SIZE4096 intmain(intargc,char*argv[]) { CNPServiceMyNPService("NPService","NamedPipeService"); MyNPService.SetPipeName("NPServiceTest"); MyNPService.m_isRunning=true; //Havewehandledtheargumentsanddonean //installoruninstall? if(MyNPService.ParseArguments(argc,argv)==false) { //Ifnot,starttheservicerunning. MyNPService.StartService(); } return0; } CNPService::CNPService(char*tsvcName,char*tdispName, DWORDtsvcStart,boolCreateMsgPump):CPPService(tsvcName, tdispName,tsvcStart,CreateMsgPump) { strcpy(m_szPipeName,"NPServiceTest"); InitializeCriticalSection(&m_cs); m_dwFreq=2000; m_dwDuration=200; m_dwPause=1000; } CNPService::~CNPService() { DeleteCriticalSection(&m_cs); } //------------------------------------------------------------------- //Theroutinethatcausesrequeststobedispatched DWORDCNPService::NPCommDispatcher(LPVOIDNotUsed) { charszPipeName[255]; boolfConnected=false; BOOLfSuccess; CNPService*p_this; HANDLEhPipe; OVERLAPPED_PLUS*overlapped; p_this=(CNPService*)m_this; SECURITY_ATTRIBUTESsa;//Securityattributes. PSECURITY_DESCRIPTORpSD;//PointertoSD. //Allocatememoryforthesecuritydescriptor. pSD=(PSECURITY_DESCRIPTOR)LocalAlloc(LPTR, SECURITY_DESCRIPTOR_MIN_LENGTH); //Initializethenewsecuritydescriptor. InitializeSecurityDescriptor(pSD,SECURITY_DESCRIPTOR_REVISION); //AddaNULLdescriptorACLtothesecuritydescriptor. //SeeMSKBarticlesQ126645andQ102798. //NotethattheNULLDACLmeansthatALLrights //areallowed. SetSecurityDescriptorDacl(pSD,TRUE,(PACL)NULL,FALSE); sa.nLength=sizeof(sa); sa.lpSecurityDescriptor=pSD; sa.bInheritHandle=TRUE; HANDLEhCompletionPort=p_this->InitializeThreads(); while(p_this->m_isRunning) { if(p_this->m_isPaused) { Sleep(1000); } else { strcpy(szPipeName,"\\\\.\\pipe\\"); strcat(szPipeName,"NPServiceTest"); hPipe=CreateNamedPipe(szPipeName,//Nameofthepipe PIPE_ACCESS_DUPLEX//Pipeisbidirectional FILE_FLAG_OVERLAPPED, PIPE_TYPE_MESSAGE//Eachwrite=message PIPE_READMODE_MESSAGE PIPE_WAIT, MAX_CLIENTS,//Maximum100instances MAX_DATA_SIZE, MAX_DATA_SIZE, NMPWAIT_USE_DEFAULT_WAIT, &sa); hCompletionPort=CreateIoCompletionPort(hPipe, hCompletionPort, (DWORD)hPipe,0); overlapped=newOVERLAPPED_PLUS; memset((void*)overlapped,'\0', sizeof(OVERLAPPED_PLUS)); overlapped->overlapped.hEvent= CreateEvent(NULL,TRUE,FALSE,NULL); fSuccess=ConnectNamedPipe(hPipe, (LPOVERLAPPED)overlapped); while((WaitForSingleObject(overlapped->overlapped.hEvent, 1000))==WAIT_TIMEOUT) { ; } } } return0; } //------------------------------------------------------------------- //Theroutinecalledtoruntheservice voidCNPService::Run() { DWORDtPause; DWORDtFreq; DWORDtDuration; DWORDdwThreadId; HANDLEhThreadHandle=CreateThread(NULL, 0, NPCommDispatcher, 0, 0, &dwThreadId); //Closehandle,threadwillcontinue. CloseHandle(hThreadHandle); //ThiswillcallMessagePumpifneeded. while(IsRunning()) { //Getaconsistentsetofvalues... EnterCriticalSection(&m_cs); tPause=m_dwPause; tFreq=m_dwFreq; tDuration=m_dwDuration; LeaveCriticalSection(&m_cs); //...thenusethemOUTSIDEthecriticalsection. Beep(tFreq,tDuration); Sleep(m_dwPause); } m_isRunning=false; //Givethreadsachancetodie. Sleep(2000); return; } //------------------------------------------------------------------- //Starttheserviceuponinstallation. voidCNPService::OnInstall() { charszKey[256]; charszBuffer[255]; HKEYhKey=NULL; BYTEdata[4096]; DWORDcData=4096; DWORDloop; BOOLfoundIt=FALSE; longret; DWORDtype=REG_BINARY; chardiag[255]; strcpy(diag,"HandlingNamedPipeInstallationIssues"); LogEvent(EVENTLOG_INFORMATION_TYPE,EVMSG_DEBUG,diag); //Makeregistryentriestosupportnamedpipeswithout //requiringausersession.SeeMSKBarticlesQ126645and //Q102798. strcpy(szKey, "SYSTEM\\CurrentControlSet\\Services\\" "LanmanServer\\Parameters\\"); ret=RegOpenKeyEx(HKEY_LOCAL_MACHINE, szKey,0,KEY_ALL_ACCESS,&hKey); strcpy(szKey,"NullSessionPipes"); if((ret=RegQueryValueEx(hKey,szKey, NULL,&type,&data[0],&cData))==ERROR_SUCCESS) { for(loop=0,foundIt=FALSE;loop<cData;) { if(strlen((char*)&data[loop])&& !(stricmp((constchar*)&data[loop], GetPipeName(szBuffer)))) { foundIt=TRUE; } loop+=strlen((char*)&data[loop])+1; } //Ifwefoundit,thiswaspreviouslyinstalled. //Wecanwalkaway. if(foundIt==FALSE) { if(cData<4080) { strcpy((char*)&data[cData-1], GetPipeName(szBuffer)); cData+=strlen(szBuffer); data[cData]='\0'; data[cData+1]='\0'; strcpy(szKey,"NullSessionPipes"); SetLastError(0); ret=RegSetValueEx(hKey,//Handletokeytosetvaluefor szKey,//Addressofsubkeyname 0,//Reserved type,//Typeofvalue //Addressofvaluedata (constunsignedchar*)data, cData+1);//Sizeofvaluedata if(ret!=ERROR_SUCCESS) { printf("\nGetLastErrorreturns%don" "registrysave...", GetLastError()); } } } } RegCloseKey(hKey); strcpy(diag,"Abouttostartserviceininstaller..."); LogEvent(EVENTLOG_INFORMATION_TYPE,EVMSG_DEBUG,diag); printf("\nAbouttostartservice..."); ::StartThisService(ServiceName()); return; } //------------------------------------------------------------------- //Thefunctionthatinitializesthreadsandcompletionport HANDLECNPService::InitializeThreads() { DWORDloop; HANDLEhCompletionPort; HANDLEhThreadHandle; DWORDdwThreadId; SYSTEM_INFOsystemInfo; boolfConnected=false; DWORDdwCompletionKey=0; //Createcompletionporttobeassociatedwithpipes. hCompletionPort=CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL,0,0); if(hCompletionPort==NULL) { returnNULL; } //Getnumberofprocessors. GetSystemInfo(&systemInfo); //Createtwothreadsperprocessor. for(loop=0; loop<systemInfo.dwNumberOfProcessors*2; loop++) { hThreadHandle=CreateThread(NULL, 0, Thread, hCompletionPort, 0, &dwThreadId); if(hThreadHandle==NULL){ CloseHandle(hCompletionPort); returnNULL; } //Closethehandle,thethreadwillcontinue. CloseHandle(hThreadHandle); } //Returnsuccessfully. returnhCompletionPort; }//InitializeThreads //------------------------------------------------------------------- //Thethreadroutine DWORD__stdcallCNPService::Thread(LPVOIDContext) { HANDLEhCompletionPort=Context; BOOLbSuccess; DWORDdwIoSize; OVERLAPPED_PLUS*lpOverlapped; DWORDcbRead; HANDLEhPipe; CNPService*p_this; p_this=(CNPService*)m_this; while(p_this->m_isRunning) { bSuccess=GetQueuedCompletionStatus(hCompletionPort, &dwIoSize, (LPDWORD)&hPipe, (LPOVERLAPPED*)&lpOverlapped, (DWORD)1000); if(!bSuccess){ CloseHandle(hPipe); continue; } //Thisoddbehaviorisrequiredtomake //thehandleactassynchronous. lpOverlapped->overlapped.hEvent= ((HANDLE)((DWORD)lpOverlapped->overlapped.hEvent0x1)); printf("\nSuccess!"); ReadFile(hPipe,&lpOverlapped->dataPacket, (sizeof(DATA_PACKET)),&cbRead, (LPOVERLAPPED)lpOverlapped); DWORDdwNewValue= *((DWORD*)&lpOverlapped->dataPacket.Buffer[0]); switch(lpOverlapped->dataPacket.ServiceCode) { caseREQ_DURATION: EnterCriticalSection(&p_this->m_cs); p_this->m_dwDuration=dwNewValue; LeaveCriticalSection(&p_this->m_cs); break; caseREQ_PAUSE: EnterCriticalSection(&p_this->m_cs); p_this->m_dwPause=dwNewValue; LeaveCriticalSection(&p_this->m_cs); break; caseREQ_FREQ: EnterCriticalSection(&p_this->m_cs); p_this->m_dwFreq=dwNewValue; LeaveCriticalSection(&p_this->m_cs); break; } strcpy(lpOverlapped->dataPacket.Buffer, "MessageReceived!"); lpOverlapped->dataPacket.DataLen= strlen(lpOverlapped->dataPacket.Buffer); lpOverlapped->dataPacket.BufLen= sizeof(lpOverlapped->dataPacket.Buffer); while(WriteFile(hPipe,&lpOverlapped->dataPacket, (sizeof(DATA_PACKET)), &cbRead, (LPOVERLAPPED)lpOverlapped)==FALSE&& GetLastError()==ERROR_IO_PENDING) { Sleep(1); } DisconnectNamedPipe(hPipe); CloseHandle(hPipe); hPipe=NULL; CloseHandle(lpOverlapped->overlapped.hEvent); deletelpOverlapped; } return(0); }

Installation issues OnInstall solves a problem that occurs because named pipes is an authenticated protocol. When initially testing a named pipes server converted from a console mode application to a service, I discovered that I was unable to connect to any pipe; I received access-denied errors every time I tried to make the connection. The service was starting under the security of the System account, and therefore should have had rights to do what it pleased, as long as it was not accessing resources off the server. As I often do when I encounter access-denied errors on a service, I changed the security context of the service and had it start under my user ID and password (my account was in the domain admins group). Not surprisingly, the service correctly connected to the client when I started it this way.

This is not an ideal situation. When distributing a service application, requiring a service to run under a user account can be a pain. I understood the issue: a named pipes server could impersonate the named pipes client, possibly allowing an errant service to do widespread damage. I was confident that the service in question would not do any damage and in fact was not using impersonation.

I found the solution in two Microsoft Knowledgebase articles, Q102798 and Q126645, available on the companion CD-ROM. The first article suggested that creating a security descriptor would resolve the problem. This step is required and included in the method NPCommDispatcher , but it is not sufficient to allow the service to connect to named pipes clients. The second article correctly described a registry change required to enable the service to connect to a named pipes client. A registry entry, NullSessionPipes (located at HKLM\System\CurrentControlset\Services\LanManServer\Parameters ), is a double-null terminated binary entry. Each service that needs to connect to a null session pipe must be listed in this entry, separated by NULLs. The final entry is followed by two NULLs. Figure 12-1 shows the end of the NullSessionPipes entry with NPService as the last entry.

Figure 12-1 The registry entry that allows a service running under the System account to connect to named pipes clients.

The operation of the service Once the service is installed it can be run. The Run method first creates a thread to run the NPCommDispatcher method. This method is declared static so that it can be used as a thread procedure. NPCommDispatcher first creates a security descriptor to be used when creating named pipes to allow the service to connect using the System account.

Next the method InitializeThreads is called. This function creates a completion port using the Win32 function CreateIoCompletionPort :

 HANDLECreateIoCompletionPort(HANDLEFileHandle,//Filehandletoassociatewith //theI/Ocompletionport HANDLEExistingCompletionPort,//HandletotheI/Ocompletion //port DWORDCompletionKey,//Per-filecompletionkeyforI/O //completionpackets DWORDNumberOfConcurrentThreads//Numberofthreadsallowedto //executeconcurrently)

The first parameter, FileHandle , is the file handle to associate with the IOCP or the value INVALID_HANDLE_VALUE if the port should be created without associating it with a handle. This handle can be a file or pipe handle or a WinSock socket. In this case, this parameter is passed as INVALID_HANDLE_VALUE because no pipes are currently active. ExistingCompletionPort associates a new instance of the completion port with an existing completion port. NPService passes NULL for the ExistingCompletionPort value in InitializeThreads because there is no existing IOCP. CompletionKey is a context value that is often used to send the I/O handle to the routine that will service the IOCP. In this case, it is passed as 0. The final parameter, NumberOfConcurrentThreads , is the number of I/O threads the system allows to run. Passing a value of 0as in the NPService exampletells the system to allow one I/O thread per processor on the system. Whenever a thread processing an IOCP enters a wait state, that thread is no longer considered in the thread count. Therefore when that previously waiting thread begins processing again, momentarily more threads than specified will be running. The number will be reduced as other IOCP threads terminate.

InitializeThreads next calls GetSystemInfo to obtain the number of processors on the system. This number is used to create two threads to monitor IOCPs for each processor. (I'll discuss the Thread method shortly.) Once the threads are created, InitializeThreads returns the handle to the completion port.

Once InitializeThreads returns, NPCommDispatcher enters a loop controlled by the value of m_isRunning . This is to allow this thread to properly shut down when the service is stopped. If the service is not paused, a named pipe is created using the Win32 CreateNamedPipe function. CreateNamedPipe is a server-side only function:

 HANDLECreateNamedPipe(LPCTSTRlpName,//Pointertopipename DWORDdwOpenMode,//Pipeopenmode DWORDdwPipeMode,//Pipe-specificmodes DWORDnMaxInstances,//Maximumnumberofinstances DWORDnOutBufferSize,//Outputbuffersize,inbytes DWORDnInBufferSize,//Inputbuffersize,inbytes DWORDnDefaultTimeOut,//Time-outtime,inmilliseconds //Pointertosecurityattributes LPSECURITY_ATTRIBUTESlpSecurityAttributes);

The first parameter, lpName , is a pointer to the name of the pipe. This is a UNC name, in the form \\.\pipe\PipeName . The dot (.) indicates the pipe is on the current machine. The pipe exists on the logical share pipe. The final portion of the name is used to uniquely identify the pipe to clients and when the server creates additional instances of the pipe.

The open mode, passed as dwOpenMode , accepts three different classes of flags. The first type refers to the type of access the pipe will allow:

PIPE_ACCESS_DUPLEX means the pipe allows bidirectional communication.

PIPE_ACCESS_INBOUND means the pipe will allow only incoming data.

PIPE_ACCESS_OUTBOUND means that from the server's perspective, this is a write-only pipe.

A second type of flag that can be passed to dwOpenMode has two values:

FILE_FLAG_WRITE_THROUGH causes writes to be completed across a network before a write operation returns. This flag is only valid for byte mode pipes (which I'll discuss shortly).

FILE_FLAG_OVERLAPPED enables overlapped, or asynchronous, operation mode. This flag allows the operation of the pipe using IOCPs.

A third type of flag that can be passed to dwOpenMode relates to security:

WRITE_DAC means the caller will have write access to the named pipe's Discretionary Access Control List (DACL).

WRITE_OWNER means the caller will have write access to the named pipe's owner.

ACCESS_SYSTEM_SECURITY means the caller will have rights to the named pipe's System Access Control List (SACL).

In Listing 12-4, CreateNamedPipe is called with PIPE_ACCESS_DUPLEX to create a named pipe for reading and writing and FILE_FLAG_OVERLAPPED to allow the pipe to be used with IOCPs.

The next parameter, dwPipeMode , also has three general types of flags. The first type controls the way data is accessed from the pipe:

PIPE_TYPE_BYTE creates a pipe in which data is written as a stream of bytes and is the default.

PIPE_TYPE_MESSAGE creates a pipe in which data written to the pipe in a single operation is considered a message. The same value must be passed for each instance of the named pipe.

Another pair of flags controls how data is read from the pipe:

PIPE_READMODE_BYTE is the default and creates a pipe in which data is read as a stream of bytes.

PIPE_READMODE_MESSAGE creates a pipe that reads one message at a time. The message mode for writing and reading is preferred unless you want to have your program manually terminate messages so that the byte stream can be parsed as it is read.

The final pair of flags that can be passed as part of dwPipeMode is the most misunderstood:

PIPE_WAIT sets the blocking mode for the pipe being created.

PIPE_NOWAIT sets the nonblocking mode for the pipe being created.

In a situation in which you might need to test for other conditions while waiting, your natural inclination is to use PIPE_NOWAIT, as with a server-based application. This is not the casePIPE_NOWAIT is provided only for compatibility with LAN Manager version 2. Using PIPE_WAIT in a combination with dwOpenMode that includes FILE_FLAG_OVERLAPPED provides the correct mix for use with IOCPs.

The nMaxInstances parameter passed to CreateNamedPipe controls the maximum number of instances of the named pipe that will be allowed. In Listing 12-4, nMaxInstances is set to a constant that evaluates to 100. In the real world, the number of concurrently active instances of a named pipe implemented as this application is likely to be lower. PIPE_UNLIMITED_INSTANCES allows the number of pipes to be limited only by the amount of memory present.

The next two parameters, nOutBufferSize and nInBufferSize , are advisory in nature. The actual size of the buffers can be the system-defined minimum, the system-defined maximum, or the specified size. The default timeout is specified in nDefaultTimeout , which specifies the timeout in milliseconds if WaitNamedPipe is called using NMPWAIT_USE_DEFAULT_WAIT. The final parameter is the security attribute, set in accordance with the requirements in Microsoft Knowledge Base article Q102798.

Next an IOCP is created through a call to CreateIoCompletionPort . In Listing 124, unlike the call in InitializeThreads , the pipe handle is passed and the existing completion port handle returns from InitializeThreads . The pipe handle, hPipe , is cast as a DWORD and sent in the CompletionKey parameter to CreateIoCompletionPort. When we look at the Thread method of CNPService , we'll see that this is passed in to GetQueuedCompletionStatus .

An OVERLAPPED_PLUS structure is then allocated and filled in, as in the following code fragment:

 overlapped=newOVERLAPPED_PLUS; memset((void*)overlapped,'\0', sizeof(OVERLAPPED_PLUS)); overlapped->overlapped.hEvent= CreateEvent(NULL,TRUE,FALSE,NULL); fSuccess=ConnectNamedPipe(hPipe, (LPOVERLAPPED)overlapped); while((WaitForSingleObject(overlapped->overlapped.hEvent, 1000))==WAIT_TIMEOUT) { if(!(p_this->m_isRunning)) { break; } }

The OVERLAPPED_PLUS structure is allocated with new. The hEvent member of the embedded OVERLAPPED structure is set to a handle created with CreateEvent . ConnectNamedPipe is called with the pipe handle and a pointer to the OVERLAPPED_PLUS structure cast as an OVERLAPPED. Next we wait on the event handle. When this is signaled, the pipe will be connected. While we wait, every second we break from WaitForSingleObject and test the Boolean flag m_isRunning. We break out of the loop if the flag is TRUE.

Interestingly, the pipe has been created, we know the connection has been made, but no reading or processing has been done. That is done in the Thread member function. Two instances of the thread per processor start in InitializeThreads . The beginning of the Thread method is shown in the following code fragment:

 p_this=(CNPService*)m_this; while(p_this->m_isRunning) { bSuccess=GetQueuedCompletionStatus(hCompletionPort, &dwIoSize, (LPDWORD)&hPipe, (LPOVERLAPPED*)&lpOverlapped, (DWORD)1000); if(!bSuccess){ continue; } }

Recall that Thread is a static member function; it can only directly access static members of CNPService . As with a static member function of the base class, we use a pointer to the classheld in m_this to get to nonstatic members. After a check at the top of the while loop to confirm the service is still running, we call GetQueuedCompletionStatus . This is the function that acts as a gateway to the IOCP. If we run the service without any clients attaching, this function will wait the amount of time specified in the timeout parameter and return indicating a failure. The prototype for GetQueuedCompletionStatus is as follows:

 BOOLGetQueuedCompletionStatus(HANDLECompletionPort,//TheI/Ocompletionport //ofinterest LPDWORDlpNumberOfBytesTransferred, //Toreceivenumberofbytes //transferredduringI/O LPDWORDlpCompletionKey,//Toreceivefile'scompletionkey LPOVERLAPPED*lpOverlapped,//ToreceivepointertoOVERLAPPED //structure DWORDdwMilliseconds//Optionaltimeoutvalue);

The first parameter is the handle of the completion port. The number of bytes transferred is passed in lpNumberOfBytesTransferred . The third parameter is lpCompletionKey . Recall that this parameter contains the pipe handle that we set when we created the IOCP in NPCommDispatcher . This will be used for further operation on the pipe. The fourth parameter is lpOverlapped , a pointer to an OVERLAPPED structure. Recall that when connecting to the pipe using ConnectNamedPipe , we passed an OVERLAPPED_PLUS structure rather than a simple OVERLAPPED structure. This OVERLAPPED_PLUS structure can be used for additional context, if needed. One example of this is a server that uses IOCPs for all I/O operations. When GetQueuedCompletionStatus returns, the current thread might need to know whether the operation that completed was a read or a write operation. Information about the operation that triggered the queued completion status can allow the server to act appropriately.

When GetQueuedCompletionStatus returns indicating success, the next step is to actually read some data. The queued completion status that the program has been notified of is the connection to a named pipes client. ReadFile must be used to actually get the data.

There is a complication here. What happens if we read the data in the pipe using ReadFile ? This function would return immediately and fire another completion when done. This is not what we're looking for. Further, we know that future requests from the client and writes to the client will be short in duration and will come as fast as the client can process them. In this case, we'd like to use the normal overlapped operations rather than the IOCP mechanism. The method to make this happen is somewhat odd and depends on the fact that Win32 treats the low 2 bits of a handled value specially. These bits are not set on normal handles. ReadFile and WriteFile are designed to see the low bit of a handle as a magic flag that says normal IOCP mechanisms should not be used. This is the reason for the following odd bit of code:

 lpOverlapped->overlapped.hEvent= ((HANDLE)((DWORD)lpOverlapped->overlapped.hEvent0x1));

Through all the casts, we simply set that magic bit of the handle so that IOCP notifications are not used. The call to ReadFile allows us to read information about the reason for the call in the ServiceCode member of DATA_PACKET. The call to ReadFile also allows us to read information from the buffer. We cast the first element in the buffer to a DWORD and use this value as a parameter to set the duration, pause, or frequency used to allow the system to control the service's beep.

The code to set the m_dwDuration , m_dwPause , and m_dwFreq variables is straightforward, including a call to EnterCriticalSection and LeaveCriticalSection to control access to these variables as they are set. Finally, for demonstration purposes only, a message, "Message Received!", is sent back to the client. Once the message is sent, DisconnectNamedPipe is called with the pipe handle and the handle is closed with a call to CloseHandle . The event handle previously created and sent in the OVERLAPPED structure is also closed.

The Run method is the last part of Listing 12-4 left to describe. It is similar to other examples, but in this case the calls to Beep and Sleep use the members of the class that control the duration, pause, and frequency.

A New Option: QueueUserWorkItem

Several new functions in Windows 2000 allow work items to be queued for processing. QueueUserWorkItem allows a unit of work to be queued:

 BOOLQueueUserWorkItem(LPTHREAD_START_ROUTINEFunction, PVOIDContext, ULONGFlags);

The first parameter is Function , the function that will be executed. Context is the 4-byte value that will be passed to Function when it is called. Flags is a bit-flag that can be set to the values described in Table 12-1.

Table 12-1 Values for the Flags Parameter of QueueUserWorkItem

Value	Meaning
WT_EXECUTEDEFAULT	The function is queued to a non-I/O thread.
WT_EXECUTEINIOTHREAD	The function is executed in an I/O worker thread. This flag should be used to ensure that the function will be executed in a thread that will not terminate with pending asynchronous I/O requests.
WT_EXECUTEINPERSISTENTIOTHREAD	The documentation indicates that this flag causes the function to run in a thread that never terminates. The documentation also says that no worker threads are persistent.
WT_EXECUTELONGFUNCTION	This is a hint to the system that the function being called could wait for a long time. The system uses this information to decide if new threads should be created. This flag can be used only with WT_EXECUTEINIOTHREAD.

An alternative to explicitly calling QueueUserWorkItem is to use the Win32 API function BindIoCompletionCallback , also new to Windows 2000:

 BOOLBindIoCompletionCallback(HANDLEFileHandle, LPOVERLAPPED_COMPLETION_ROUTINEFunction, ULONGFlags);

This function greatly simplifies the use of I/O completion ports, a complex operation in prior Windows versions. FileHandle is the handle to a file opened with the FILE_FLAG_OVERLAPPED flag. Function is a pointer to a routine to be called when the I/O on the file is complete. The last parameter, Flags , is unused and must be 0. The prototype for Function is:

 VOIDCALLBACKFileIOCompletionRoutine(DWORDdwErrorCode,//Completioncode DWORDdwNumberOfBytesTransfered,//Numberofbytestransferred LPOVERLAPPEDlpOverlapped//Pointertostructurewith //I/Oinformation);

The first parameter, dwErrorCode , is the completion code, which will be 0 if the I/O was successful or ERROR_HANDLE_EOF if the operation fails. The second parameter, dwNumberOfBytesTransfered , specifies the number of bytes transferred or returns 0 if an error occurs. The last parameter, lpOverlapped , is a pointer to an OVERLAPPED structure.

Most of the overlapped structure can be ignored when we use it for named pipe access and completion callbacks. The last member, hEvent , is often used as a way to pass context to the FileIOCompletionRoutine . If we were using a file handle rather than a named pipe handle, we could simply allocate additional memory after the OVERLAPPED structure and place context information there, as we did in Listing 12-4.

The advantage of QueueUserWorkItem and BindIoCompletionCallback should be clear: much of the work is done for you. The disadvantage is that you as a developer have no real control over the number of threads created. Thus, it is possible that one server-based application on a shared machine can monopolize the processor.

WinSock vs. Named Pipes

Both WinSock and named pipes have their advantages. If you need to create a server-like application that must run on a Windows 9x machine, WinSock is all that you really can use, since Windows 9x can be a named pipes client but not a named pipes server. Note that many of the functions used for the WinSock service in Listings 12-1 and 12-2 and for the named pipes service in Listings 12-3 and 12-4 use functions that are available only on Windows 2000 and Windows NT.

If you need to impersonate a client in order to enforce security, named pipes wins hands down. ImpersonateNamedPipeClient is a relatively easy way to allow your server to act under the authority of the connected client. If you need to connect over wide-area networks or the Internet, WinSock is a better choice.

Events vs. I/O Completion Ports vs. QueueUserWorkItem

Choosing the overall structure of a server-based application is perhaps a bigger decision than choosing a communication protocol. In both the WinSock example and the named pipes example, the details of the one protocol could be easily ripped out and replaced with those of the other protocol. It is much more difficult to replace one architecture for handling client requests with another.

The WinSock example using an event handle to control access to a set of worker threads was in many ways easier to create than the IOCP example or the named pipes example. The structure seems clear and it seems to offer reasonable control over the number of threads and over how work is performed. IOCPs have the advantage of being standardized with operating system support. Once you have created one IOCP-based service, you will more easily be able to create a second. IOCPs also ensure that the most recently used worker thread will be used first, giving the best chance that its memory will not have been paged out to disk. There are no guarantees when using the automatic reset event handle with multiple threads running. Both IOCPs and named pipes hold up under reasonable loads with a small number of clients simulating a larger number of clients.

QueueUserWorkItem offers great promise, though with too little developer control for my comfort. In theory, BindIoCompletionCallback seems like another step in the right direction for creating high performance servers to handle large numbers of clients.

In practice, a number of the server-based applications I have created have used the simpler one-thread-per-client model. Of course, I've limited these applications to small numbers of clients with each interaction comprised of a short query and response, but for some simpler requirements this model can suffice.