Accessing Other Web Sites with HTTP

The .NET Framework contains a rich set of classes for accessing other Web sites across the Internet. You can use these classes to perform such actions as retrieving Web pages from other Web sites, posting form data to other Web sites, and resolving DNS host names and IP addresses.

Why would you want to do this? One common problem that Web sites face is how to share information across the Internet. Suppose, for example, that you are a book distributor and you want to automatically send a list of top-selling books to all your affiliate Web sites. In this situation, you could create a Web page on the main distribution Web site with a list of top-selling books. Your affiliate Web sites could then use the .NET classes described in this section to automatically retrieve this Web page and display the list of top-selling books.

NOTE

A better alternative for distributing information across the Internet is XML Web Services. XML Web Services are described in detail in Part VI of this book, "Building ASP.NET Web Services."

The .NET classes for Internet access can be roughly divided into three groups:

• The WebClient class ” Contains easy-to-use methods for uploading and downloading data from a remote Internet server. Behind the scenes, this class uses the WebRequest and WebResponse classes.

• The WebRequest and WebResponse classes ” Use descendents of these classes for fine-grained control over Internet requests and responses. The HttpWebRequest and HttpWebResponse classes provide support for both synchronous and asynchronous requests using the HTTP protocol. Behind the scenes, these classes rely on the Sockets classes.

• The Sockets classes ” Implements the Windows Socket Interface (Winsock32). Includes a Socket , TcpClient , TcpListener , and UdpClient class.

All the Internet access classes can be found in the System.Net and System.Net.Sockets namespaces.

Using the WebClient Class

The easiest way to grab a page off another Web server is to use the WebClient class. The WebClient class has three methods for downloading data:

• DownloadData ” Returns a byte array from an Internet address.

• DownloadFile ” Saves a file to the file system retrieved from an Internet address.

• OpenRead ” Returns a stream from an Internet address.

For example, the page in Listing 26.18 retrieves the home page from the Amazon.com Web site, replaces any image tags with the string [image] , and displays the altered page (see Figure 26.6).

Listing 26.18 StripImage.aspx

 <%@ Import Namespace="System.Net" %> <Script Runat="Server"> Sub Page_Load   Dim wcMicrosoft As WebClient   Dim objUTF8Encoding As UTF8Encoding   Dim strResponse As String   wcMicrosoft = New WebClient   objUTF8Encoding = New UTF8Encoding   strResponse = objUTF8Encoding.GetString( _     wcMicrosoft.DownloadData( "http://www.amazon.com" ) )   strResponse = RegEx.Replace( strResponse, "(<img.*?>)", "[image]", RegExOptions.SingleLine )   Response.Write( strResponse ) End Sub </Script> 

The C# version of this code can be found on the CD-ROM.

Figure 26.6. Stripping images from Amazon.com.

In Listing 26.18, an instance of the WebClient class is created. Next, the DownloadData() method is called with the address of the Amazon.com Web site.

The DownloadData() method returns a byte array. Before you can do anything interesting with the data returned, you need to convert the byte array into a string. Use the GetString() method of the UTF8Encoding class to accomplish this.

Finally, the Replace() method of the RegEx class is used to replace every <img> tag with the string [image] . Calling Response.Write() renders the downloaded page to the browser.

Using the HttpWebRequest Class

Instead of using the WebClient class, you can use the HttpWebRequest and HttpWebResponse classes to grab a Web page from a Web server. These classes are more difficult to use, but they are also more flexible. For example, you can use the HttpWebRequest and HttpWebResponse classes to request a page both synchronously and asynchronously. Furthermore, you can use these classes to simulate HTML form posts.

The page in Listing 26.19 uses the HttpWebRequest and HttpWebResponse classes to simulate a form post. The page posts whatever data you enter into a text box to another page named Post.aspx and displays the results.

Listing 26.19 HttpWebRequest.aspx

 <%@ Import Namespace="System.Net" %> <%@ Import Namespace="System.IO" %> <Script Runat="Server"> Sub Button_Click( s As Object, e As EventArgs )   Dim objRequest As HttpWebRequest   Dim strRequest As String   Dim arrRequest As Byte()   Dim objUTF8Encoding As UTF8Encoding   Dim strmRequest As Stream   Dim objResponse As HttpWebResponse   Dim srResponse As StreamReader   ' Initialize request object   objRequest = _     CType( WebRequest.Create( "http://localhost/Post.aspx" ), HttpWebRequest )   objRequest.Method = "POST"   objRequest.ContentType = "application/x-www-form-urlencoded"   ' Create request body   strRequest = "Message=" & Server.UrlEncode( txtMessage.Text )   objUTF8Encoding = New UTF8Encoding   arrRequest = objUTF8Encoding.GetBytes( strRequest )   ' Add body to request   objRequest.ContentLength = arrRequest.Length   strmRequest = objRequest.GetRequestStream()   strmRequest.Write( arrRequest, 0, arrRequest.Length )   strmRequest.Close()   ' Get response   objResponse = objRequest.GetResponse()   srResponse = New StreamReader( objResponse.GetResponseStream(), Encoding.ASCII )   lblResponse.Text =  srResponse.ReadToEnd()   srResponse.Close() End Sub </Script> <html> <head><title>HttpWebRequest.aspx</title></head> <body> <form runat="Server"> <h3>Enter a message:</h3> <asp:TextBox   id="txtMessage"   Runat="Server" /> <asp:Button   Text="Post!"   OnClick="Button_Click"   Runat="Server" /> <h3>Response:</h3> <asp:Label   id="lblResponse"   EnableViewState="False"   Runat="Server" /> </form> </body> </html> 

The C# version of this code can be found on the CD-ROM

NOTE

Before you use the page in Listing 26.19, you might need to change the path passed to the WebRequest.Create() method from http://localhost/post._aspx to the location of the Post.aspx page on your server.

The contents of the Post.aspx page is contained in Listing 26.20.

Listing 26.20 Post.aspx

 <Script runat="Server"> Sub Page_Load   lblMessage.Text = Request.Form( "Message" ) End Sub </Script> <html> <head><title>Post.aspx</title></head> <body> You posted the following message: <br> <asp:Label   id="lblMessage"   Runat="Server" /> </body> </html> 

The C# version of this code can be found on the CD-ROM.

The Post.aspx page simply echoes the value of the Message form field.

Resolving Domain Names

Included in the System.Net namespace is a Dns class. You can use the Dns class to translate between IP addresses and domain names.

The Dns class supports the following methods:

• GetHostByAddress ” Returns an IPHostEntry from an IP address.

• GetHostByName ” Returns an IPHostEntry from a domain name .

• Resolve ” Returns an IPHostEntry from either an IP address or domain name.

For example, the page in Listing 26.21 contains a form with a text box for an IP address. If you submit the form, the domain name corresponding to the IP address is displayed (see Figure 26.7).

Listing 26.21 Dns.aspx

 <%@ Import Namespace="System.Net" %> <Script runat="Server"> Sub Button_Click( s As Object, e As EventArgs )   Dim objIPHostEntry As IPHostEntry   objIPHostEntry = Dns.GetHostByAddress( txtIP.Text )   lblHostName.Text = objIPHostEntry.HostName End Sub </Script> <html> <head><title>Dns.aspx</title></head> <form runat="Server"> <h2>Enter an IP Address:</h2> <asp:TextBox   id="txtIP"   Runat="Server" /> <asp:Button   Text="Submit!"   OnClick="Button_Click"   Runat="Server" /> <p> <asp:Label   id="lblHostName"   EnableViewState="False"   Runat="Server" /> </form> </body> </html> 

The C# version of this code can be found on the CD-ROM.

Figure 26.7. Resolving an IP Address.