Hack 39. Use XMLHttpRequest to Scrape an Energy Price from a Web Page | Ajax Hacks: Tips & Tools for Creating Responsive Web Sites

Allow the user to choose an energy fuel type and generate the current price without refreshing the page.

The Web includes lots of different places for getting the latest energy prices, such as for a barrel of crude oil or the average cost of U.S. residential propane. These web sources usually involve loading into the browser a particular business-oriented page over and over again (such as from cnnfn.com) or visiting multiple web sites in search of various prices. This hack offers an alternative: it automatically grabs an energy price based on the user's choice in a select list element. The hack doesn't involve any page rebuilding, so the feature can be built into a broader application for handling energy prices.

Getting in a Scrape

The source of the energy price is a public-domain site managed by the U.S. Energy Information Agency (U.S. EIA). You can also use a commercial web service to access instantaneous energy prices, which avoids having to scrape or harvest the price from the HTMLa better solution from an application-design standpoint but not free of charge. The EIA site suits our purpose, however, because it illustrates how to the access multiple data pieces from third-party sources, then displays of the data value without rebuilding the entire page. The sequence for this hack's behavior goes like this:

The user chooses a fuel type in the select list.
This choice triggers the select's onchange event handler.
The event handler uses XMLHttpRequest to send a request to a Java JSP page.
The JSP uses a predefined component to scrape the energy price from the U.S. EIA page, then sends the price as text back to the web page.
The web page shows a "Fetching energy price..." message, then displays the latest price in a colorful font.

Figure 4-18 shows the hack's web page.

Figure 4-18. Fetching a live energy price

When the user makes a select-list choice, Figure 4-19 shows what the result looks like.

Figure 4-19. Checking out the price

The JavaScript code the page uses is rather simple:

window.onload=function(){     var _url=     "http://www.parkerriver.com/energy.jsp?priceTyp=";     if($("fuelType")){         $("fuelType").onchange=function(){             try{                 showQuote($("msg"),                  "Fetching energy price...");                 httpRequest("GET",_url+$F("fuelType"),                 true,handlePrice);             } catch(errv) {                 alert("Sorry, but we failed to get the energy price "+                       "because "+                       "of this error: "+errv.message);             }         };     } } function showQuote(_id,txt){     if(_id && txt){_id.innerHTML=txt;} } function handlePrice(){     try{         if(request.readyState == 4){             if(request.status == 200){                 var resp =  request.responseText;                 if(resp != null && resp.length > 0){                     showQuote($("msg"),                     "The requested price is: "+resp);                 }   else {                     showQuote($("msg"),                     "The price is not available at this time.");                 }             } else {                 //request.status is 503 if the application                 // isn't available;                 //500 if the application has a bug                 alert(                 "A problem occurred with communicating between"+                 " the XMLHttpRequest object and the server program.");             }         }//end outer if     } catch (err)   {         alert("It does not appear that the server "+               "is available for this application. Please"+               " try again very soon. \\nError: "+err.message);     } }

window.onload sets up the onchange event handler for our lone select element containing the choices of fuel types. onchange points to a function that the browser will call each time the user chooses a different option in the select list. The hack then takes the fuel choice and sends it to the JavaServer Pages component using the request object.

See "Use Your Own Library for XMLHttpRequest" [Hack #3] for an explanation of the http_request.js library this hack uses for handling XMLHttpRequest.

The JavaScript uses a couple of handy shortcuts from the Prototype library (see Chapter 6). The select element's id value is "fuelType", as in <select />. $("fuelType") is the equivalent of document.getEle-mentById("fuelType"), and $F("fuelType") provides the current value of the select element.

The hack finally inserts the return value, an energy price that the EIA refreshes about once per week, inside a span element on the web page. This final step occurs inside handlePrice(), which is the readystate handler for the request object.

var resp =  request.responseText; if(resp != null && resp.length > 0){   showQuote($("msg"),   "The latest price is: "+resp);}

For Java Jocks

In case you're interested in the "scraping" code, here's the OilpriceCallback Java class that fetches a crude-oil price:

package com.eeviewpoint; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.net.URL; import javax.swing.text.html.HTMLEditorKit.ParserCallback; import javax.swing.text.MutableAttributeSet; import javax.swing.text.html.parser.ParserDelegator; import javax.swing.text.html.HTML; public class OilpriceCallback extends ParserCallback       implements Scraper{     private boolean crudeOilCrumb=false;     private boolean foundCurrPrice=false;     private String urlSource=      "http://tonto.eia.doe.gov/dnav/pet/pet_pri_fut_s1_d.htm";     private String result = "";     public String getUrlSource() {         return urlSource;     }     public void setUrlSource(String urlSource) {         this.urlSource = urlSource;     }     public String getResult() {         return result;     }     public void setResult(String result) {         this.result = result;     }     public void handleText(char[] chars, int i) {         String str = new String(chars);         if(str.indexOf("Crude Oil") != -1){             crudeOilCrumb=true;         }         if(this.crudeOilCrumb && this.foundCurrPrice &&                 getResult().length() == 0){             setResult(str.trim());         }     }     public void handleStartTag(HTML.Tag tag,         MutableAttributeSet mutableAttributeSet, int i) {         if((crudeOilCrumb) && tag ==  javax.swing.             text.html.HTML.Tag.TD){             String val = (String) mutableAttributeSet.             getAttribute(HTML.Attribute.CLASS);             if(val != null && val.equalsIgnoreCase("Current")){                 foundCurrPrice=true;             }         }     }     public String toString() {         return getResult();     } }

A class named EnergyHarvester contains a list (in an object called scraperMap) of various callback classes (such as OilpriceCallback) that scrape prices for the different fuel types. Here is the code from EnergyHarvester for returning the requested price.

public String getNugget() throws ClassNotFoundException,     IllegalAccessException, InstantiationException, IOException {     String nm = ((String)scraperMap.get(priceType));     ParserCallback callback = (ParserCallback) Class.forName(nm).     newInstance();     URL eia = new URL(((Scraper) callback).getUrlSource());     BufferedReader webPagestream = new BufferedReader(       new InputStreamReader(eia.       openStream()));     super.parse(webPagestream,callback,true);     return callback.toString(); }

Here's the JSP component our web page calls. The code uses an instance of EnergyHarvester, which in turn uses different implementations of the HTML-parsing code to fetch the various energy prices.

<%@taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %> <% response.addHeader("Content-Type","text/plain");     response.addHeader("Cache-Control","no-cache"); %> <jsp:useBean  /> <jsp:setProperty name="parser" property=      "priceType" value="${param.priceTyp}"/> <c:out value="${parser.nugget}"/>