The WebRequest Class: More Than Meets the Eye

I l @ ve RuBoard

The WebRequest Class: More Than Meets the Eye

The WebRequest class is designed to fulfill a network request to a particular URI. The network request consists of the information you'll be sending to server; an instance of a WebResponse class will be produced that consists of the response data stream from the server. To illustrate how these two classes work together, look at the following example, which simply reads a page and dumps the output to the console:

 ImportsSystem ImportsSystem.Net ImportsSystem.Text ImportsSystem.IO ModuleHelloInternet SubMain() DimreqAsWebRequest=_ WebRequest.Create("http://www.gotdotnet.com/") DimresAsWebResponse=req.GetResponse() DimsrAsStreamReader=NewStreamReader(res.GetResponseStream()) DimresChar(256)AsChar DimcharCountAsInteger=sr.Read(resChar,0,resChar.Length) DoWhilecharCount>0 DimstrAsString=NewString(resChar,0,charCount) Console.Write(str) charCount=sr.Read(resChar,0,resChar.Length) Loop res.Close() EndSub EndModule 

The sample starts off with a call to WebRequest.Create , a static method of the WebRequest class, which creates a WebRequest instance of this class using the webRequestModule defined for the specified URI ”in this case, the HTTP protocol. What's really happening under the covers is that the WebRequest.Create method looks at the protocol defined in the specific URI. It then checks the machine.config file's defined webRequestModules for the specified protocol. (See the following excerpt from the machine.config file.) In this case, the HttpRequestCreator is defined for the HTTP protocol, so the Create method uses the HttpRequestCreator class to create an instance of an HttpRequest class. Because HttpRequest inherits from WebRequest , you do not need to treat it any differently.

Note

The HttpWebRequest and HttpWebResponse classes support most of the HTTP 1.1 protocol's feature set.

 <webRequestModules> <addprefix="http" type="System.Net.HttpRequestCreator" /> ... </webRequestModules> 

Once we have an instance of our WebRequest class, we call the GetResponse method, which returns an instance of a WebResponse class. This class contains the response data in a Stream , which can be obtained by calling the GetResponseStream method. To easily read the data in the stream, we create a new StreamReader class, passing the response stream to the constructor. We then read characters from the stream until we reach the end of the stream. As a last act before we're done, we close the StreamReader instance. Easy.

Note

GetResponse is a synchronous method: the calling thread is blocked until the response is returned by the server. In the real world, where response times can be slow, it's not a good idea to block the thread ” especially if the thread is a user interface thread, which can result in a unresponsive user experience during the time it takes to make the call to GetResponse . To avoid this problem, you can use the BeginGetResponse and EndGetResponse methods instead.

Supporting Client Authentication

You often need to provide some client authentication when communicating with a Web server. You can provide this by using the Credentials property of the WebRequest class. The simplest case requires you to create an instance of a NetworkCredential class and then assign it to the Credentials property. This mechanism supports Basic, Digest, NTLM, and Kerberos authentication. It also stores appropriate information ( name , password, domain, and so forth). This property can contain an instance of the NetworkCredentials class (if there is only one credential) or an instance of the CredentialCache class (if there are multiple sets of credentials tied to specific sites). The following sample demonstrates how you might use the Credentials property:

 DimwrAsWebRequest ... 'Option1:createacredential,thenassignit DimncAsNewNetworkCredential("tony", "wong") wr.Credentials=nc 'Option2:Createasetofcredentials,addthemtothecache,then 'assignthecachetotheCredentialsproperty DimcacheAsNewCredentialCache() cache.Add(NewUri("www.microsoft.com"), "Basic",nc) cache.Add(NewUri("msdn.microsoft.com"), "Basic",nc) wr.Credentials=cache 

Managing Your Connections

Resource management is an important part of all well-designed applications. Network connections, like any other physical resource, should be tightly controlled and considered a relatively scarce resource. Thankfully, most of the connection management work for the WebRequest class is handled for you. You can customize a number of supported connections through the App.Config or Machine.Config files. The following excerpt demonstrates how this works:

 <connectionManagement> <addaddress="*" maxconnection="2" /> </connectionManagement> 

This section of the configuration file defines the maximum number of connections to a server or group of servers. The default entry means there can be a maximum of two simultaneous connections in your application to each server. If you exceed this limit, you'll start encountering exceptions. By adjusting this setting in the application configuration file, you can override the default and also add server-specific settings, as shown here:

 <connectionManagement> <addaddress="*" maxconnection="5" /> <addaddress="www.microsoft.com" maxconnection="10" /> </connectionManagement> 

In this example, I've overridden the default maximum connections and set it to 5; the http://www.microsoft.com server has been allowed up to 10 simultaneous connections.

Note

Connection limits are per host or server or URI. They are protocol- and domain name “specific. That is, it doesn't matter whether the resource you're using is on the same server. (You can exceed the number of connections to a single machine if you're using different protocols.)

System.Net also provides the ServicePoint and ServicePointManager classes for programmatically managing connection limits. If you don't want to rely on the configuration file for setting your connection limits, use the ConnectionLimit property of the ServicePoint class for a particular host. The following example demonstrates how you can customize the connection limits for a specific host dynamically at runtime:

 DimuriAsNewUri("http://www.microsoft.com") DimspAsServicePoint=ServicePointManager.FindServicePoint() sp.ConnectionLimit=20 

To change the setting programmatically for all hosts , change the DefaultPersistentConnectionLimit property of the ServicePointManager class ( DefaultNonPersistentConnectionLimit when you use HTTP 1.0). You usually don't need be bothered with these classes unless you're hitting the connection limit and want to change it right away.

Note

The default value of the KeepAlive property of HttpWebRequest is true , which means that all the connections will remain open and will be reused for subsequent requests to same server unless a timeout occurs (specified by the MaxIdleTime property of the ServicePoint ).

Extending this concept, the WebRequest class's ConnectionGroupName property lets you group connections. It is mainly relevant for middle- tier components that want to reuse existing connections to a server.

Note

Reusing existing connections is a recommended best practice for increasing network performance. A certain amount of overhead is associated with establishing a network connection. If you can make the connection only once and reuse the open connection, you'll improve the efficiency of your application. For example, The SqlClient ADO.NET provider uses connection pooling for all connections to SQL servers to achieve greater responsiveness and application throughput.

Creating Custom WebRequestModules

This section is really an extension of our earlier discussion of pluggable protocols. The WebRequest/WebResponse model allows for the creation of custom protocol handlers and even overriding existing default handlers. For example, consider a design for creating an FTP protocol handler (ignoring the actual protocol details):

1. Create a FileWebRequestCreator class that inherits from the IWebRequestCreate class. In the Create method, return an object of FtpWebResponse (mentioned below).

2. Register this class using the RegisterPrefix method of the WebRequest class, as follows . By registering the class, you are making it known that there is an object called ftp that handles all ftp: requests.

3. Create FtpWebRequest , which inherits from the WebRequest class and overrides all needed members : the ConnectionGroupName , Content ­Length , ContentType , Credentials , Headers , Method , PreAuthenticate , Proxy , RequestUri , and Timeout properties; and the Abort , BeginGet ­RequestStream , EndGetRequestStream , BeginGetResponse , EndGetResponse , GetRequestStream and GetResponse methods.

4. Create an FtpWebResponse class that inherits from the WebResponse class and overrides all necessary members: the ContentLength , ContentType , Headers , and ResponseUri properties; and the Close and GetResponseStream methods.

Note

It's a good idea to program in terms of WebRequest and WebResponse classes wherever possible so that any new protocols that are added by the .NET Framework or new protocols that you add will continue to be used with minimal (if any) changes to the code.

Here's what the skeleton code for a new WebRequest handler might look like:

 ImportsSystem.Net PublicClassFtpWebRequestCreator ImplementsIWebRequestCreate FunctionCreate(ByValuriAsUri)AsWebRequest_ ImplementsIWebRequestCreate.Create ReturnNewFtpWebRequest(uri) EndFunction EndClass PublicClassFtpWebRequest InheritsWebRequest Privatem_UriAsUri FriendSubNew(ByValuriAsUri) MyBase.New() m_Uri=uri EndSub EndClass PublicClassFtpWebResponse InheritsWebResponse FriendSubNew() MyBase.New() EndSub EndClass 

Advanced WebRequest Features

Now that we've covered the basics of WebRequest and WebResponse , let's look at the some of the more advanced features. As you've seen, the Create method of the WebRequest determines which protocol will be used, and any new protocols can be registered via the RegisterPrefix method of WebRequest . Here's how you would typically use the WebRequest and WebResponse classes:

1. Create a request using WebRequest.Create . Depending on the protocol being used, you can typecast this into an appropriate protocol-specific class.

2. Fill in Credentials property of the WebRequest . You can use CredentialCache.DefaultCredentials for NTLM, negotiate, and Kerberos-based authentication or use NetworkCredential to specify a specific username/password.

3. If you're using a proxy that requires authentication, create and set Credentials for the WebProxy class.

4. Fill in protocol-specific properties (such as the user agent and client certificates for HTTP requests).

5. Call the GetResponse method of the WebRequest class. This will cause a blocking request. If you need asynchronous calls, use BeginGetResponse and EndGetResponse instead (as noted below).

6. Typecast the WebReponse returned to appropriate protocol class and access any protocol-specific methods. Get the stream using the GetResponseStream method. Close the stream and the Response class.

The following sample puts all of the previous steps into practice:

 ImportsSystem ImportsSystem.Net ImportsSystem.IO ImportsSystem.Text ImportsSystem.Security.Cryptography.X509Certificates ModuleWebRequestSample SubMain() DimsbAsNewSystem.Text.StringBuilder() Try DimwreqAsWebRequest=_ WebRequest.Create("https://SomeServer/SomePage.aspx") wreq.Credentials=CredentialCache.DefaultCredentials DimmyProxyAsNewWebProxy("ProxyServer:Port") myProxy.Credentials=NewNetworkCredential("UserID",_  "Password",_  "Domain") CType(wreq,HttpWebRequest).ClientCertificates.Add(_ X509Certificate.CreateFromCertFile("")_) DimwrepAsWebResponse=wreq.GetResponse() DimsAsStream=wrep.GetResponseStream() Dimbuffer()AsByte=NewByte(1024){} DimcountAsInteger=0 While(count<s.Read(buffer,0,buffer.Length)) sb.Append(Encoding.Default.GetString(buffer,0,count)) EndWhile s.Close() wrep.Close() CatchweAsException sb.Append(we.ToString()) EndTry Console.WriteLine(sb.ToString()) EndSub EndModule 

That's a more complete implementation using WebRequest . Next, we'll look at another important feature: asynchronous operations with WebRequest .

Asynchronous Operations

As I mentioned earlier, you can use the BeginGetResponse and EndGetResponse methods of the WebRequest to initiate and complete requests in an asynchronous manner. An important point to keep in mind is to not mix asynchronous and synchronous operations. The WebRequest classes were not designed to support both types of communication simultaneously , and you might end up with strange results if you try it. Do it one way or the other, but never mixed. This includes dealing with the Stream data returned by the WebResponse class. In other words, if you use the asynchronous BeginGetResponse method, you should also use the BeginRead / EndRead asynchronous methods of the resulting Stream object.

Now let's take a closer look at the BeginGetResponse method of the WebRequest class. The BeginGetResponse has following prototype:

 PublicOverridableFunctionBeginGetResponse(_ ByValcallbackAsAsyncCallback,_ ByValstateAsObject_)AsIAsyncResult 

Note

If you implement your own pluggable protocol, you must override the BeginGetResponse method.

The first parameter is an AsyncCallback delegate. The callback method that implements it will use the returned IAsyncResult interface to obtain the status of this asynchronous request and complete the request by calling the EndGetResponse method. This function returns an IAsyncResult that uniquely identifies the asynchronous request. The second parameter represents the state of the asynchronous request. When you perform asynchronous operations, you must pass and preserve data across asynchronous calls. This state parameter can be any object, but it often makes sense to create a specific class to help maintain the state information of the Request and eventual stream so that the class can be used with the Stream reader callback functions as well. Our sample defines the following State class:

 PublicClassState ConstBufferSizeAsInteger=4096'defaultbuffersize PublicRequestAsWebRequest'StoretheWebRequestclass 'forcallingEndGetRequest PublicResponseAsWebResponse'Storeforclosingout 'Responsewhendone PublicResponseStreamAsStream'ForcallingBeingRead/EndRead PublicDataAsStringBuilder'Actualdataofinterest PublicBuffer()AsByte'arraytoreadbytesfromResponse PublicSubNew() Request=Nothing Response=Nothing ResponseStream=Nothing Data=NewStringBuilder() Buffer=NewByte(BufferSize){} EndSub EndClass 

Now that we've discussed the implementation of asynchronous requests, it's time to see it in action. Building on the earlier examples, following is an example of asynchronously downloading a file using the WebRequest class. (See also the AsyncWebRequest sample project.)

Note

This example uses a ManualResetEvent to prevent the application from exiting prematurely.

 ImportsSystem ImportsSystem.Net ImportsSystem.IO ImportsSystem.Text ImportsSystem.Threading ModuleAsyncWebRequest PublicClassState ConstBufferSizeAsInteger=4096'defaultbuffersize 'StoretheWebRequestclassforcallingEndGetRequest PublicRequestAsWebRequest 'StoreforclosingoutResponsewhendone PublicResponseAsWebResponse PublicResponseStreamAsStream'ForcallingBeingRead/EndRead PublicDataAsNewStringBuilder()'Actualdataofinterest PublicBuffer()AsByte'arraytoreadbytesfromResponse PublicSubNew() Buffer=NewByte(BufferSize){} EndSub EndClass PrivateallDoneAsNewManualResetEvent(False) SubMain() DimrequestStateAsNewState() requestState.Request=WebRequest.Create("http://www.microsoft.com") DimrAsIAsyncResult=_ CType(requestState.Request.BeginGetResponse(_ NewAsyncCallback(AddressOfRequestCallback),_ requestState),IAsyncResult) 'Don'texittillManualResetEventissetinCallback allDone.WaitOne() EndSub SubRequestCallback(ByValarAsIAsyncResult) Try DimrequestStateAsState=CType(ar.AsyncState,State) requestState.Response=requestState.Request.EndGetResponse(ar) requestState.ResponseStream=_ requestState.Response.GetResponseStream() DimiarReadAsIAsyncResult=_ requestState.ResponseStream.BeginRead(_ requestState.Buffer,_ 0,_ requestState.Buffer.Length,_ NewAsyncCallback(AddressOfStreamCallback),_ requestState) CatchexAsException Console.Error.WriteLine(ex.ToString()) EndTry EndSub SubStreamCallback(ByValarAsIAsyncResult) Try DimrequestStateAsState=CType(ar.AsyncState,State) DimbytesReadAsInteger=requestState.ResponseStream.EndRead(ar) If(bytesRead>0)Then requestState.Data.Append(_ Encoding.Default.GetString(requestState.Buffer,_ 0,_ bytesRead)) requestState.ResponseStream.BeginRead(requestState.Buffer,_ 0,_ requestState.Buffer.Length,_ NewAsyncCallback(AddressOfStreamCallback), requestState) Else requestState.ResponseStream.Close() requestState.Response.Close() Console.WriteLine(requestState.Data.ToString()) allDone.Set() EndIf CatchexAsException Console.Error.WriteLine(ex.ToString()) EndTry EndSub EndModule 

Note

The synchronous WebRequest.GetResponse method actually uses the asynchronous methods ( BeginGetResponse and EndGetResponse ) internally.

That wraps up our discussion of the WebRequest and WebResponse classes. Now on to a related class: WebClient .

I l @ ve RuBoard