Web Class Basics
There are four classes of interest for accessing Web resources: HttpWebRequest , HttpWebResponse , WebRequest , and WebResponse . The first two are used specifically for HTTP communication, while the second two offer an extensible interface to a variety of application protocols, including HTTP.
The Web classes in this chapter are all request-response transactions. The client makes a request to a server to perform some action. The server receives the request and generates a response that is returned to the client. Following the response, the requested entity or content is transmitted.
The HTTP protocol defines several actions a client can request of the server. These actions are listed in Table 10-1. The most common method, GET , requests to retrieve a URI resource. For example if a GET request is made for http://www.microsoft.com/ an HTTP request is generated and sent to www.microsoft.com . The server then examines the request, generates a response indicating success, and sends the entity, which is the default HTML page for the Web site. The second column of the table indicates which version of the HTTP protocol introduced the method.
| Method | Minimum Version | Description |
|---|---|---|
| GET | 0.9 | Retrieves a URI resource |
| HEAD | 1 | Retrieves only the metadata of a URI resource |
| POST | 1 | Submits a form or other data (such as a SOAP message) |
| PUT | 1.1 | Uploads a file |
| DELETE | 1.1 | Deletes a URI resource |
| TRACE | 1.1 | Traces proxy chains |
| OPTIONS | 1.1 | Queries HTTP server options |
Additionally, the HTTP protocol defines a series of header options that affect the behavior of the response. These options are simple string name /value pairs. The HTTP protocol defines numerous header options. The majority of these HTTP headers are exposed as properties of the HttpWebRequest , HttpWebResponse , WebRequest , and WebResponse classes. The ContentLength property that maps to the Content-Length: header is an example. Notice that some headers are valid only for requests, others for response, some for both request and response, and others for entities only. Table 10-2 lists the HTTP headers exposed as properties.
In addition to many of the headers being exposed as properties, the property Headers returns a WebHeaderCollection object that contains the entire set of headers associated with the request or response. The Add method of the WebHeaderCollection class can be used to set additional HTTP headers in the collection. Notice that the headers exposed directly as properties cannot be set using the WebHeaderCollection . Instead, they should be accessed through their strongly typed properties.
| Property | HTTP Header | Valid Method | Description |
|---|---|---|---|
| Accept | Accept: | Request | Specifies acceptable media types (e.g., text/html, image/ jpeg, etc.) |
| Connection | Connection: | Request, Response | Controls how connections are handled (e.g., disable persistent connections) |
| ContentLength | Content-length: | Entity | Specifies the length of the returned entity in bytes |
| ContentType | Content-type: | Entity | Specifies the media type of the returned entity (e.g., text\html, image\jpeg, etc.) |
| Expect | Expect: | Request | Indicates expected behavior of client by the server |
| IfModifiedSince | If-Modified-Since: | Request | Used to verify cache is up-to- date |
| Referer | Referer: | Request | Specifies the URL of the document containing the reference to the requested URL |
| TransferEncoding | Transfer-encoding: | Request, Response | Specifies any transformations applied to content (e.g., chunked) |
| UserAgent | User -Agent: | Request | Specifies the client software issuing the request |
Only the most relevant headers are covered here; for a full listing of the HTTP headers and their corresponding properties, consult the .NET Framework SDK and RFC 2616.
HttpWebRequest and HttpWebResponse
The HttpWebRequest and HttpWebResponse provide HTTP-specific implementations of the WebRequest and WebResponse classes (which are discussed later) and allow access to URI-based HTTP resources. These classes offer a complete implementation of the HTTP 1 and 1.1 protocols.
As we mentioned earlier, HTTP traffic is based on request-response, and the first step of accessing HTTP-based Web resources is to create an instance of an HttpWebRequest object that indicates a Web resource. This is done by calling the Create method of the WebRequest class as in the following sample:
C#
HttpWebRequesthttpRequest; httpRequest=(HttpWebRequest)WebRequest.Create(http://www.winisp.net/goodrich/default.htm");
Visual Basic .NET
DimhttpRequestasHttpWebRequest httpRequest=WebRequest.Create(_ http://www.winisp.net/goodrich/default.htm _)
The example creates an HttpWebRequest object that points to the URI http://www.winisp.net/goodrich/default.htm . Notice that the constructor for the HttpWebRequest class should not be called ”only the WebRequest.Create method. See the WebRequest and WebResponse section of this chapter for more details on the requirements of calling the WebRequest.Create method. Once an instance of the Web object is created, the HTTP headers and properties can be modified. Also, by default the HttpWebRequest method is GET . At this point, nothing has hit the network because an object needs to be created in order to set properties before the request is initiated.
The next step is to issue the request and receive the response from the destination. This is done by calling the GetResponse method, which returns a WebResponse object that can be cast to an HttpWebResponse object if you need to access the HTTP-specific properties:
C#
HttpWebResponsehttpResponse; httpResponse=(HttpWebResponse)httpRequest.GetResponse();
Visual Basic .NET
DimhttpResponseAsHttpWebResponse httpResponse=httpRequest.GetResponse()
The returned HttpWebResponse object will indicate the response code of the request via the StatusCode property, as well as a text description of the result in StatusDescription . If an entity is to be sent following the request, the next step is to call GetResponseStream , which returns a Stream object that is then used to receive the entity data. Refer to Chapter 2 for information on manipulating Stream objects. If no response stream is associated with the HttpWebResponse , a ProtocolViolationException is thrown when GetResponseStream is called.
Finally, once all data is received from the response stream, the stream should be closed either by calling the Stream.Close method or by calling HttpWebResponse.Close . Note that both methods can be called, but at least one must be invoked to free the connection. The HttpResponse.Close method essentially calls the Close method of the stream and is provided as a convenience. If neither is called, the application will leak resources and run out of allowed connections. Notice that since HTTP 1.1 is implemented, keepalives are enabled by default.
Web Exceptions
As shown, setting up an HTTP Web request and receiving the response is easy and straightforward; however, the previous example didn t handle any exceptions. Exceptions are most commonly encountered when retrieving the response for a Web request and these exceptions take the form of an instance of a WebException class. Once a WebException occurs, the Status property indicates the type of failure. Common failures include underlying protocol failure, request timeout, etc. The Status property is WebExceptionStatus -enumerated type. Table 10-3 lists typical failures and their description, but consult the .NET Framework SDK for a complete description of each of the enumerated error types.
| Error | Description |
|---|---|
| ConnectFailure | An error occurred at the transport (e.g., TCP) level. |
| ConnectionClosed | The connection was unexpectedly closed. |
| Pending | The asynchronous request is pending. |
| ProtocolError | A response was received but a protocol-level error occurred (see Table 10-4 for HTTP protocol-level errors). |
| ReceiveFailure | An error occurred receiving the response. |
| RequestCanceled | The request was canceled by the Abort method. This error is used for any error not classified by the WebExceptionStatus enumeration. |
| SendFailure | An error occurred sending the request to the server. |
| Success | The operation completed successfully. |
| Timeout | No response was received during the specified timeout period. |
| TrustFailure | The server certificate could not be validated . |
If the indicated exception status is WebExceptionStatus.ProtocolError ” which means a response from the server was retrieved but a protocol error was detected ”then an HttpWebResponse object has been created and can be accessed to obtain more information about the failure. In the case of the HTTP protocol, the HTTP-specific error code can be retrieved from the HttpWebResponse class as the HttpStatusCode property. Table 10-4 lists the common HTTP return codes and their descriptions. Again, consult the .NET Framework SDK for a complete list of the errors and meanings.
| Status Code | HTTP Error Code | Description |
|---|---|---|
| Accepted | 202 | The request has been accepted for further processing. |
| BadRequest | 400 | The server could not understand the request. This error is also used for any errors not classified by the HttpStatusCode enumeration. |
| Continue | 100 | The client can continue with the request. |
| Forbidden | 403 | The server has refused to respond to the request. |
| GatewayTimeout | 504 | An intermediate proxy server timed out while waiting for a response. |
| InternalServerError | 500 | A generic error occurred on the server while responding to the request. |
| NotFound | 404 | The requested resource does not exist on the server. |
| OK | 200 | The request was successful and the requested information was sent. |
| ProxyAuthenticationRequired | 407 | The proxy server requires an authentication header. |
| RequestTimeout | 408 | The client did not send the request in the time expected by the server. |
| ServiceUnavailable | 503 | The server cannot handle the request due to high load or because it is down. |
| Unauthorized | 401 | The requested resource requires authentication to access. |
The following example verifies in the catch block whether a response was returned and, if so, prints the HTTP status code information:
C#
HttpWebRequesthttpRequest; HttpWebResponsehttpResponse; try { httpRequest=(HttpWebRequest)WebRequest.Create("http://www.winisp.net/goodrich"); httpResponse=(HttpWebResponse)httpRequest.GetResponse(); } catch(WebExceptionwex) { if(wex.Status==WebExceptionStatus.ProtocolError) { httpResponse=(HttpWebResponse)wex.Response; Console.WriteLine("HTTPResponseCode:{0}", httpResponse.StatusCode.ToString()); httpResponse.Close(); } } Visual Basic .NET
DimhttpRequestAsHttpWebRequest DimhttpResponseAsHttpWebResponse Try httpRequest=WebRequest.Create(http://www.winisp.net/goodrich) httpResponse=httpRequest.GetResponse() CatchwexAsWebException Ifwex.Status=WebExceptionStatus.ProtocolErrorThen httpResponse=wex.Response Console.WriteLine(HTTPResponseCode:{0}",_ httpResponse.StatusCode.ToString()) httpResponse.Close() EndIf EndTry Common Scenarios
Now that we ve covered the basics for HTTP Web requests and responses, we ll move on to discuss some common scenarios in more detail. In this section, we present detailed samples for making HTTP requests with the GET and POST methods.
GET
GET retrieves content from Web servers and is the default method for most Web transactions. In the previous examples we showed how to construct instances of the HttpWebRequest class and how to retrieve the response header, but not how to retrieve the entity. The basic steps for issuing a GET request and handling the response are:
-
Create an HttpWebRequest object with the URL to retrieve
-
Retrieve the HttpWebResponse object for the request
-
Retrieve the stream to receive the entity data
-
Receive the data until the end of the stream is reached
The following code sample illustrates these steps. Once it obtains the HttpWebResponse object, it retrieves the stream handle and reads the returned data until the end. If the retrieved entity is text, a StreamReader is created to retrieve the data; otherwise , if the entity is an image, a BinaryReader is used.
C#
HttpWebRequesthttpRequest=null; HttpWebResponsehttpResponse=null; BinaryReaderbinReader=null; StreamReaderstreamReader=null; try { //CreatetheHTTPrequestobject httpRequest=(HttpWebRequest)WebRequest.Create("http://www.winisp.net/goodrich"); //SetsomeHTTPspecificheaders httpRequest.UserAgent= "MyUserAgent/1.0"; //Gettheresponseobject httpResponse=(HttpWebResponse)httpRequest.GetResponse(); if(httpResponse.ContentType.StartsWith(@"image")) { //Forimageentities,usethebinaryreader binReader=newBinaryReader(httpResponse.GetResponseStream()); byte[]responseBytes; //Readtheresponsein4KBchunks while(true) { responseBytes=binReader.ReadBytes(4096); if(responseBytes.Length==0) break; //Dosomethingwiththedata } } elseif(httpResponse.ContentType.StartsWith(@"text")) { //Fortextentities,usethetextreader. streamReader=newStreamReader(httpResponse.GetResponseStream(),Encoding.UTF8); stringhttpContent=streamReader.ReadToEnd(); //Dosomethingwiththedata } } catch(WebExceptionwex) { Console.WriteLine("Exceptionoccurredonrequest:{0}", wex.Message); if(wex.Status==WebExceptionStatus.ProtocolError) httpResponse=wex.Response; } finally { if(httpResponse!=null) httpResponse.Close(); if(binReader!=null) binReader.Close(); if(streamReader!=null) streamReader.Close(); } Visual Basic .NET
DimhttpRequestAsHttpWebRequest=Nothing DimhttpResponseAsHttpWebResponse=Nothing DimbinReaderAsBinaryReader=Nothing DimstreamReaderAsStreamReader=Nothing Try CreatetheHTTPrequestobject httpRequest=WebRequest.Create(_ "http://www.winisp.net/goodrich") SetsomeHTTPspecificheaders httpRequest.UserAgent= "MyUserAgent/1.0" Gettheresponseobject httpResponse=httpRequest.GetResponse() IfhttpResponse.ContentType.StartsWith("image")Then Forimageentities,usethebinaryreader binReader=NewBinaryReader(_ httpResponse.GetResponseStream()) DimresponseBytes()AsByte Readtheresponsein4KBchunks While(True) responseBytes=binReader.ReadBytes(4096) IfresponseBytes.Length=0Then GoToAfterLoop EndIf Dosomethingwiththedata EndWhile AfterLoop: ElseIfhttpResponse.ContentType.StartsWith("text")Then Fortextentities,usethetextreader. streamReader=NewStreamReader(_ httpResponse.GetResponseStream(),Encoding.UTF8) DimhttpContentAsString=streamReader.ReadToEnd() Dosomethingwiththedata EndIf CatchwexAsWebException Console.WriteLine("Exceptionoccurredonrequest:{0}",_ wex.Message) Ifwex.Status=WebExceptionStatus.ProtocolErrorThen httpResponse=wex.Response EndIf Finally IfNothttpResponseIsNothingThen httpResponse.Close() EndIf IfNotbinReaderIsNothingThen binReader.Close() EndIf IfNotstreamReaderIsNothingThen streamReader.Close() EndIf EndTry The Chap10\HttpGetRequest code sample illustrates retrieving a URI, as well as linked documents and images in that URI to a local directory, and provides a progress indicator for each file as it downloads.
POST
Issuing a POST request is similar to issuing an HTTP GET request. One difference is that the request stream is retrieved from the HttpWebRequest object to send the post data. The general steps for issuing a POST request are:
-
Create an HttpWebRequest object with the URL to post to
-
Change the Method property to POST and the ContentType property to application/x-www-form-urlencoded
-
Retrieve the request stream from the HttpWebRequest object
-
Send the post data on the stream
-
Retrieve the HttpWebResponse object
-
Retrieve the response stream from HttpWebResponse and read the response
The following code sample illustrates issuing a POST request and receiving the response:
C#
HttpWebRequesthttpRequest; HttpWebResponsehttpResponse; BinaryReaderhttpResponseStream; try { //CreateHTTPWebrequest httpRequest=(HttpWebRequest)WebRequest.Create("http://www.microsoft.com"); //Changemethodfromthedefault "GET" to "POST" httpRequest.Method= "POST"; //Postedformsneedtobeencodedsochangethecontenttype httpRequest.ContentType= "application/x-www-form-urlencoded"; //DatatoPOST stringpostData= "sl=foo&s2=bar"; //Retrieveabytearrayrepresentationofthedata byte[]postBytes=Encoding.UTF8.GetBytes(postData.ToString()); //Setthecontentlength httpRequest.ContentLength=postBytes.Length; //RetrievetherequeststreamsowecanwritethePOSTdata StreamhttpPostStream=httpRequest.GetRequestStream(); //WritethePOSTrequest httpPostStream.Write(postBytes,0,postBytes.Length); httpPostStream.Close(); //Retrievetheresponse httpResponse=(HttpWebResponse)httpRequest.GetResponse(); //Retrievetheresponsestream httpResponseStream=newBinaryReader(httpResponse.GetResponseStream(), Encoding.UTF8); byte[]readData; while(true) { readData=httpResponseStream.ReadBytes(4096); if(readData.Length==0) break; //Processresponse } httpResponseStream.Close(); httpResponse.Close(); } catch(WebExceptionwex) { Console.WriteLine("Exceptionoccurred:{0}",wex.ToString()); httpResponse=(HttpWebResponse)wex.Response; httpResponse.Close(); } Visual Basic .NET
DimhttpRequestAsHttpWebRequest DimhttpResponseAsHttpWebResponse DimhttpResponseStreamAsBinaryReader Try CreateHTTPWebrequest httpRequest=WebRequest.Create("http://www.microsoft.com/") Changemethodfromthedefault "GET" to "POST" httpRequest.Method= "POST" Postedformsneedtobeencodedsochangethecontenttype httpRequest.ContentType= "application/x-www-form-urlencoded" DatatoPOST DimpostDataAsString= "sl=foo&s2=bar" Retrieveabytearrayrepresentationofthedata DimpostBytes()AsByte=Encoding.UTF8.GetBytes(_ postData.ToString()) Setthecontentlength httpRequest.ContentLength=postBytes.Length RetrievetherequeststreamsowecanwritethePOSTdata DimhttpPostStreamAsStream=httpRequest.GetRequestStream() WritethePOSTrequest httpPostStream.Write(postBytes,0,postBytes.Length) httpPostStream.Close() Retrievetheresponse httpResponse=httpRequest.GetResponse() Retrievetheresponsestream httpResponseStream=NewBinaryReader(_ httpResponse.GetResponseStream(),Encoding.UTF8()) DimreadData()AsByte While(True) readData=httpResponseStream.ReadBytes(4096) IfreadData.Length=0Then GoToafterwhile EndIf Processresponse EndWhile afterwhile: httpResponseStream.Close() httpResponse.Close() CatchwexAsWebException Console.WriteLine("Exceptionoccurred:{0}",wex.ToString()) httpResponse=wex.Response httpResponse.Close() EndTry The Chap10\HttpPostRequest code sample illustrates posting a form to a URI and receiving the response.
Web Proxies
Corporate environments often incorporate a firewall that requires a proxy server to be specified for Web traffic to travel beyond the corporate network. If a request is made for a resource outside the local proxied network, the request will fail and a WebException is thrown. For Web requests to traverse the proxy, the Proxy property of the HttpWebRequest class must be set. To set the proxy, an instance of the WebProxy class is created with the string name of the proxy server. The following code illustrates creating a HttpWebRequest and setting the Proxy property. Notice that the property needs to be set before invoking the request with the GetResponse method.
C#
WebProxyproxyServer=newWebProxy("http://corpproxy/"); HttpWebRequesthttpRequest=(HttpWebRequest)WebRequest.Create("http://www.winisp.net/goodrich"); httpRequest.Proxy=proxyServer; HttpWebResponsehttpResponse=(HttpWebResponse)httpRequest.GetResponse(); Visual Basic .NET
DimproxyServerAsWebProxy=NewWebProxy(http://corpproxy:80) DimhttpRequestAsHttpWebRequest=WebRequest.Create(_ http://www.winisp.net/goodrich) httpRequest.Proxy=proxyServer HttpWebResponsehttpResponse=httpRequest.GetResponse()
In the preceding example, a specific proxy server is set for the request by specifying the proxy server name in the WebProxy constructor. If Microsoft Internet Explorer is configured with a static proxy server, the static WebProxy method GetDefaultProxy will return the currently configured proxy server as a WebProxy object, as shown by the following statement:
WebProxyproxyServer=WebProxy.GetDefaultProxy();
If the proxy server requires authentication, the necessary credentials to access the server must be specified in the Credentials property of the WebProxy class. Creating and assigning network credentials is covered in detail later on in this chapter.
| Note | The next version of the .NET Framework is likely to add automatic proxy discovery ”that is, support Web Proxy Auto Discovery (WPAD). |
In addition to the ability to set the proxy server information on each request, the .NET Framework maintains a global proxy setting for use as the default proxy for all Web requests if a proxy setting is not explicitly provided. This is done through the GlobalProxySelection class. The property of interest is Select , which is assigned a WebProxy object that will be the default proxy server for subsequent requests when a proxy server is not explicitly assigned. The GlobalProxySelection.GetEmptyWebProxy method is of interest because it can be assigned to GlobalProxySelection.Select to turn off all use of a proxy server across the entire application.
Finally, a default proxy setting for System.Net can be specified in the machine.config file located in the .NET Framework runtime installation directory usually found under <drive>:\<windir>\Microsoft.Net\Framework\<version>\
config. The default setting should appear as follows :
<system.net> <defaultProxy> <proxyusesystemdefault="true"/> </defaultProxy> </system.net>
This setting indicates that System.Net will use the computer s proxy settings, as specified in the Internet Options control panel applet. The system default settings for proxy servers will generally suffice and applications do not have to set a specific proxy on the request.
WebRequest and WebResponse
The WebRequest and WebResponse classes offer protocol agnostic abstract interfaces to request-response communication. Currently, the .NET Framework offers handlers for file and HTTP transfers, and it is possible for other protocols to be plugged into the same harness by inheriting from the WebRequest and WebResponse classes.
We ve seen how the WebRequest class plays a role in issuing an HTTP request. Earlier, when we created an HTTP request, we called the WebRequest.Create method with the URI of the resource being requested. Because the URI begins with the protocol http , the WebRequest class is able to determine that the HTTP handler should be called to service this request. Likewise, if the URI passed to Create was file://c:\files\myfile.txt , then the file handler FileWebRequest is called to service the request.
The advantage of using the WebRequest and WebResponse classes, rather than casting the returned object to its specific type, such as HttpWebRequest , is that all operations for issuing the request and handling the response are available in these base classes. That is, a Web page can be retrieved solely by creating a WebRequest object and retrieving the WebResponse or by using WebClient , which builds on WebRequest and WebResponse . Notice that for C#, it is necessary to cast the Web object to its specific type to access properties specific to that protocol (for example, the Connection property of the HttpWebRequest class).
As mentioned earlier, the sequence of calls for handling a request- response operation with WebRequest is the same as it is for the HttpWebRequest class. The following code illustrates this point:
C#
//Createtherequestobject WebRequestrequest=WebRequest.Create(address); //Issuetherequest WebResponseresponse=request.GetResponse(); //Readthecontent StreamReaderreader=newStreamReader(response.GetResponseStream(), Encoding.ASCII); stringcontent=reader.ReadToEnd(); //Displaythecontenttotheconsole Console.WriteLine(content); //Closetheresponse response.Close()
Visual Basic .NET
EAN: 2147483647
Pages: 121