Section 23.5. Making It Simple Is Complicated

23.5. Making It Simple Is Complicated

Internet Explorer's extensibility, particularly its Browser Helper Object facility, made Bugnosis's unobtrusive user interface possible. But things didn't always snap together as easily as we would have liked. In the following sections, we'll look at some of the details of the architecture that required special attention.

23.5.1. Using Browser Helper Objects and the Document Object Model

The design decision with the largest impact was where to put the analysis engine. Fairly obviously, it had to run on the end user's computer (rather than on a remote server) in order to maintain acceptable speeds. We also considered the idea of running an HTTP proxy on the user's computer, which would be able to intercept the browser's HTTP requests involved in rendering its web pages. But there were many problems with the proxy approach:

Some users already rely on a proxy for web access

To support these users, we would have to install our proxy in front of theirs, and then their old proxy would appear to vanish from their Internet Options configuration dialog. How confusing!

The web browser visually indicates the multiple steps in fetching a page

These include resolving the DNS name, contacting the remote host, fetching the page, etc. But when a proxy is installed, it has to perform those tasks itself, and there's no standard way for a proxy to send updates directly to the browser's status line. Less visual feedback overall makes the combination of browser and proxy seem sluggish.

The proxy would have to parse HTML in order to apply the relevant web bug tests

For example, the once test (which excludes images with URLs that are repeated on a page) requires counting how many times the image appears on the page, not just counting how many times it is fetched from the network. These two measures are not identical because of browser caching. Similarly, an image's visible size is determined both by its native size (from the image file itself) and by any HTML width and height attributes used in the <IMG> element that fetches the image.

Proxies cannot intercept HTTPS (encrypted) connections to perform any kind of page analysis

This is the case because proxies don't have the decryption keys. But web bugs are easily deployed on HTTPS pages, and we wanted Bugnosis to be able to detect them. In fact, this would present a valuable lesson about security on the Web: seeing the little lock icon doesn't mean that you're safe from all forms of surveillance.

Some web bugs are created by client-side scripts, usually JavaScript programs

In fact, we've seen web bugs generated by JavaScript code that sense the user's operating system type, screen resolution, window size, and JavaScript version, and that encode that information in a web bug. We'd like Bugnosis to detect these dynamically created web bugs as well. But no one could actually write a proxy that would reliably detect a JavaScript program's intent to create a web bug.

The Bugnosis web page analysis needs to be visible somewhere

If it is provided by a proxy, it would either have to be displayed in a separate window or mixed into the HTML content of the analyzed page. Keeping a separate window open just because Bugnosis is installed is clunkier than we'd like, and the alternative of adding extra HTML content to an arbitrary external page without disturbing its layout is very tricky.

So, instead of the proxy approach, we built the analysis directly into the Internet Explorer (IE) web browser using its Browser Helper Object (BHO) hooks.^[11] Naturally, this restricted our audience to IE users. But it gave us direct access to the browser's view of the world, including its toolbars, status line, and sidebars. Most importantly, it provided read/write access to the Document Object Model (DOM) representation of the page.^[12] This conferred a number of advantages:

^[11] Scott Roberts, Programming Internet Explorer 5 (Microsoft Programming Series) (Microsoft Press, 1999). See also the Microsoft Developer Network at http://msdn.microsoft.com/.

^[12] World Wide Web Consortium, "Document Object Model (DOM) Level 2 Core Specification" (Nov. 2000); http://www.w3.org/DOM/.

• In order to find the images on the page, we don't have to parse any HTML at all: we just walk through the DOM data structures, looking for elements that are images.

• To find out an image's size, we query its width and height properties: this tells us precisely how the image is rendered, taking into account both the image's native size and its scaling by HTML's width and height attributes.

• Internet Explorer handles the loading of pages, and our code just sees the results. So, we can analyze encrypted web pages just as easily as unencrypted ones.

• By hooking into DOM events, our code can notice when images are created as a result of client-side scripting and analyze them as well.

23.5.2. The Event Model

Bugnosis is an event-driven BHO. When IE starts up, it creates our Component Object Model (COM) object called IEMonitor, which then identifies itself as an event sink for the containing WebBrowser object. IE subsequently calls IEMonitor::Invoke( ) with appropriate parameters when it receives reportable events, such as UI manipulation and element download progress. The events of particular interest to us are:

BeforeNavigate2

This event indicates that the browser is loading a new web page, and a parameter gives the new page's URL. Later, we'll compare the embedded element URLs against this URL so that we can identify those that are third party.

DownloadBegin and DownloadEnd

When IE downloads an embedded element, it brackets the download with this pair of events. Unfortunately, the events don't carry parameters indicating which element is being downloaded. These events are really just an indication that something is happening. However, individual elements can be inspected for completeness by querying their ReadyState property, and you can gain access to all DOM elements on the page by walking through the web browser's Document property.

DocumentComplete

Once all of the elements on a page have been downloaded (or had their downloads aborted) and the page is fully rendered, IE fires this event along with a parameter indicating which window (such as the main window, or a frame in a frameset, or a pop-up) is complete. This is our main indication to analyze the page for web bugs and to trigger an alert, if any are found.

23.5.2.1 Provisional analysis

Of course, web pages don't load instantlythere can be a significant delay between the BeforeNavigate2 and DocumentComplete events. Bugnosis analyzes the document provisionally during this period so that it can update its toolbar status line as the embedded elements arrive. This makes the tool appear alive and responsive, and ensures that the status line always refers to the current page somehow, not to a page that it visited previously. (Bugnosis also puts the main page URL in its status line in order to avoid confusion about exactly what it's analyzing; see Figure 23-4). During the downloading period, it never alerts the user about a web bug, but it may begin to mark elements as suspicious.

These provisional analyses are triggered by the arrival of ProgressChange events, which IE fires when it updates its visual meter indicating how much of the page has been loaded. The accompanying parameter is merely the percentage of the line that has been drawn on the screen so far; it is not a certain indicator of anything, but it arrives regularly and roughly in sync with page-loading activity, so it's a reasonable signal to rescan. DownloadComplete events also cause a provisional scan.

23.5.2.2 Rescanning, refreshing, and Old Paint

Things change during the post-DocumentComplete period: Bugnosis remains alert so that it can see more web bugs arrive. This can happen if a client-side script loads an image after the main page has been fully delivered. If so, and the image is not served from the browser's cache, then the new image will trigger DownloadComplete and ProgressChange events, and so we will rescan the document and react to the addition.

At the same time, we don't want Bugnosis to keep chirping "uh-oh!" whenever a web bug is found somewhere on the page. The goal is to alert the user once when the first web bug on a page is found, and perhaps again later if a new web bug appearsbut more than that would be really annoying!

What if the user clicks the Refresh button to reload the page? This puts us in kind of a spot. Because the URL of the page doesn't change, no BeginNavigate2 event is posted: in fact, no event is delivered indicating that the page is being reloaded at all. All we can count on is some DownloadBegin, DownloadComplete, and ProgressChange activity: this is indistinguishable from client-side scripts loading data. But we'll probably recognize all of the images being loaded because they were loaded on this page already, so nothing will seem new, and so no alarm will soundeven if the user specifically pressed Reload in order to have Bugnosis rescan the page! Our compromise is that Bugnosis rescans the entire document whenever a triggering event arrives without considering whether the images are new or old, but it imposes a limit of one "uh-oh!" every three seconds.

Finally, complications arise when Bugnosis makes web bugs visible:

• The standard galloping bug ( we affectionately call him "Old Paint") is delivered as a GIF that is embedded inside the Bugnosis executable. After a web bug has been identified, it's a simple matter to change its image's src property to point to this URL, and IE changes the image. However, this also triggers the image-downloading events, which causes the document image to be rescanned. What was once a tiny image obtained from a third-party site with an accompanying cookie is now an 18 x 18 image served out of the local filesystem, which Bugnosis wouldn't find suspicious at all.

• To further complicate matters, changing an image's src property sometimes disturbs its ReadyState property. Bugnosis uses this property to determine whether a page is not ready for a full web bug scan. But we had observed that in IE 6, whenever a web bug is made visible by changing its src, its ReadyState never again becomes "fully ready."

To address these issues, before making any web bug visible, Bugnosis adds properties to the image object that record its previous dimensions and URL. When rescanning the document later on, Bugnosis recognizes them and reports these original characteristics rather than its updated ones, and excludes the image from further testing.

Unfortunately, on HTTPS (encrypted) web pages, changing an embedded https:// URL to a file:// URL in order to make the web bug visible displays a warning dialog; IE complains that this encrypted web page contains unencrypted content. We couldn't find an acceptable workaround, so in this case, Bugnosis simply leaves the original invisible image in place. Sorry, Old Paint!

23.5.3. The Analysis Pane and Toolbar

The analysis pane in the lower part of the main IE window (Figure 23-1) is not intrinsically attached to the browser helper object; it's another type of IE extension called a comm bar. Comm bars (as well as explorer bars, which are oriented vertically instead of horizontally) serve only to take control of part of the IE window. Once claimed, IE provides some decoration for the pane including space for the pane's title ("Bugnosis 1.3" rotated 90°), a "close" button, and the horizontal line separating IE's main window from the comm bar. The user can move the separator with the pointer and in this way change the relative size of the two windows.

23.5.3.1 The discovery dance

The toolbar is yet another type of IE extension, similarly claiming screen space and window decoration (namely, the ability to move the toolbar around). IE invokes these three types of extensionsthe browser helper object for sinking events, the toolbar, and the comm barat startup time, by individually calling their IObjectWithSite::SetSite(IWebBrowser2*) functions with the surrounding web browser as a parameter. IE knows to do so because entries in the system registry indicate that these objects are installed. And, naturally, the three objects need to communicate with each other at runtime. However, IE doesn't call these objects' SetSite functions in a predictable order. Bugnosis therefore uses a data structure shared by all three of the objects (which luckily run in the same process) as a rendezvous point so that the objects can discover each other.

23.5.3.2 Making the Analysis Pane feel natural

As mentioned earlier, IE doesn't manage the space it gives to the analysis pane (comm bar). Bugnosis populates this blank space with a newly created web browser controla self-contained web-rendering engine provided by IE itself that provides only an HTML rendering space, and no toolbars of any kind. Bugnosis causes its analysis to appear in this pane by obtaining the web browser control's <body> element and writing onto its innerHTML property. By providing this text output as HTML, Bugnosis matches the look-and-feel of the web browser, and also simplifies the problem of saving or printing Bugnosis output.

Because the main IE window doesn't know that it contains an embedded IE window (only Bugnosis knows that), a little more work is required to make the embedded window feel natural. Specifically, when the user moves the mouse pointer over a link, we want the URL of the link to appear in the main browser window status bar. Although the web browser control does generate an event indicating that some status bar should be updated during a mouse-over, this event is not connected to the user's main web browser, because the main web browser doesn't even know that the embedded control is there. Bugnosis connects the dots by intercepting status bar update events emitted by the control and transforming them into explicit changes to the status bar text in the main web browser.

Right-clicking the pointer within the analysis pane brings up a little menu (see Figure 23-5), allowing a user to export the analysis in various forms. This is essentially the same menu available through the drop-down menu on the Bugnosis toolbar, aside from some extra options available here (such as Copy text). The export options cause the analysis to be regenerated in a form that removes some of the embedded JavaScript and other constructs that would not work properly if viewed outside of a web browser control managed directly by Bugnosis.

Figure 23-5. Context menu in the analysis pane

23.5.3.3 Analyzing pop-up windows

Pop-up windows were another challenge. Where and how can we present analysis results for these windowswindows that normally don't contain any toolbar at all? We considered taking screen space from pop-ups, just as we do for the main browser window, but decided against it. Whereas ordinary web pages don't have control over the dimensions of their window, pop-ups are often made to be just large enough to hold their content. Taking away screen space from pop-ups would usually mean rendering their content poorly. Instead, Bugnosis keeps a record of all of the analyses it has made, and allows the user to bring up an old analysis with the Pages Analyzed submenu in Figure 23-5. Therefore, the user can get the analysis by controlling the Bugnosis analysis that is visible. This also allows users to retrieve a page's analysis if the page has changed quicklyfor example, by an automatic redirect.

Pop-up windows also caused us a fair number of coding headaches because of an apparent bug in IE 6. We observed that pop-up windows sometimes just don't generate a DocumentComplete event, which we need as our trigger to begin real (nonprovisional) scans of the document. To compensate for this, Bugnosis treats newly created windows very suspiciously: if no DocumentComplete arrives in a new window but other (less reliable) indicators of completeness are present, then Bugnosis performs its full scan anyway. We observed similar problems during "file download" operations (fetches of content that are not rendered natively by IE); sometimes, these generate DocumentComplete events carrying a parameter that doesn't appear to refer to any previously encountered window.

In spite of Bugnosis's attempt to compensate for these irregularities, it still occasionally waits for a signal that never arrives and thus stops analyzing documents properly. Sometimes it seems that IE is itself confused: for example, we have seen the IE status line continuing to display "Opening page [URL]..." (and we can observe that Bugnosis is also still awaiting a completion signal), yet the animated icon indicating "document loading" in the upper-righthand corner of IE has stopped swirling and the page appears to be fully present. Bugs like these are hard to isolate and hard to fix; we are merely grateful that Bugnosis's own errors tend to under-alarm rather than over-alarm users.

23.5.4. Installation and Uninstallation

Installation and uninstallation are extremely important because they form the first and last impression of a piece of software. On this account, the first release of Bugnosis failed miserably. Here's what went wrong.

Because Bugnosis is packaged as a COM object inside a dynamic link library (DLL), it was possible to use IE's ActiveX delivery capability for installation. This technique is very straightforward: an <object> tag on a web page specifies a URL for the object. If it's not already present on the user's computer, IE fetches it. This appealed to uswe envisioned a potential user being able to read about Bugnosis at our web site, and then being able to install it with just a click or two. IE would post a security warning dialog showing information from the software publisher's certificate (the Privacy Foundation, in our case) and ask the user's permission before proceeding. To uninstall, users would select the Uninstall option from the Bugnosis menu, which would delete the Bugnosis entries from the registry and effectively decouple Bugnosis from IE for the future.

23.5.4.1 Installation/uninstallation problems

In theory, and in closed test environments, our approach to installation and uninstallation worked reasonably well. But there were many disadvantages:

• Although this method was consistent with IE under IE's default security settings, some users ran IE with ActiveX downloads disabledperhaps they were privacy fundamentalists, or just extra careful. It was a little embarrassing to tell them that they had to temporarily lower their security settings in order to install Bugnosis.

• Immediately after an installation attempt, Bugnosis would display a web page that detected whether it worked, and would give appropriate advice to the user. The sensing mechanism turned out to be unreliable for unknown reasons. Sometimes Bugnosis would activate quickly, and other times it would require an IE restart. When a restart was required, our web page gave the wrong advicethis was, of course, confusing and frustrating to users.

• On a few systems, Bugnosis simply would not install, in spite of apparently correct settings. We never learned why.

• Probably the worst scenario happened when the Bugnosis browser helper object installed successfully, but the comm bar didn't. This left the user with a browser in which Bugnosis was workingand occasionally saying "uh-oh!"but with no visible controls. Even the Uninstall menu option was hidden! To get it to shut up, users had to hunt down and remove the .DLL file, or scrape out a registry entry or two, or run regsvr32.exe manually, or perform some other equivalently complicated maneuver. The simplest-sounding of theseremoving the .DLL filemay have actually been the hardest, because the file was automatically stored by IE in a hidden directory (which was named differently on various Windows platforms).

• We thought it was nice that IE would pick a directory in which to store the .DLL file all by itself. And because Bugnosis is not very large (less than 1 MB), we thought it wasn't critical to remove it during uninstallation. Some users, however, aren't comfortable with merely disconnecting Bugnosis from IEthey really want to delete it when they uninstall. So, these users also had to navigate to the hidden directory to remove it.

We soon realized this was becoming a disaster and put together a traditional installation mechanism: a self-extracting .EXE holding a wizard-driven installation dialog. This had the advantage of familiarity, which we should have valued more highly up front. To uninstall Bugnosis, the user is now directed to the Add/Remove Programs dialog in the control panel.

23.5.4.2 Windows XP Service Pack 2

Microsoft's Service Pack 2 for Windows XP now restricts ActiveX installations even further, so it's good we got away from that installation model when we did. In SP2, IE also provides an "add-in manager" to give the user finer control over the currently installed browser extensions. On the whole, this is great, but it now also gives the user the ability to individually disable the Bugnosis toolbar or the Bugnosis browser helper object. This results in runtime configurations that previously weren't easily accessible. The next version of Bugnosis ought to ensure that all of the newly available configurations result in sensible behavior.