Hack 86. Track MDS HTTP Requests

Figure out where your users are going using the BlackBerry Browser. You may even find some useful site that you want to visit.

Do your users use the Mobile Data Service? How frequently? Do your users go to Internet sites or intranet sites more often? Which sites are visited most?

All these questions are interesting ones that will be different in any environment. The more data you mine from the logs will translate into more knowledge about how your users are using the service and how their experience can be improved. Perhaps an intranet application that you know has not been optimized for the BlackBerry is nevertheless being visited but not fully utilized by your BlackBerry users. You may want to go to the application developer to push for better wireless support in the web application.

You might even see frequent access to various HTTP traffic for online games [Hack #32]. Perhaps you would want to disallow access to these and other non business related sites.

A single HTTP request in your MDS logs takes up at least 16 lines, and the hostname and the rest of the URL are on different lines. This makes using a simple grep command to extract the URLs impossible. Here is some Perl code that keeps track of HTTP GET requests, reconstructs the original URLs and prints them by the number of requests made to each.

8.3.1. The Code

 use strict; # change the $mds_log variable to your log file my $mds_log = "BESNAME_MDAT_01_20050607_0002.txt"; my $http_data = {}; open MDS, $mds_log; while (<MDS>) {         chomp; my $id = ""; my $host = ""; my $get = ""; if (/EVENT = ReceivedFromDevice/ and           /HTTPTRANSMISSION = Host:([^>]+)>/) {               $host = $1;             ($id) = (/CONNECTIONID = (\d+),/); $http_data->{$id}->{host} = $host;         } elsif (/EVENT = SentToServer/ and           /HTTPTRANSMISSION = GET (.+) HTTP\/1\.1>/) {               $get = $1;             ($id) = (/CONNECTIONID = (\d+),/); $http_data->{$id}->{get} = $get;         } } close MDS; my %urls = (); foreach my $id (keys %{ $http_data }) {         my $host = $http_data->{$id}->{host}; my $get = $http_data->{$id}->{get}; next if not $host and $get; my $url = "http://$host" . $get; $urls{$url}++; } foreach my $url (sort { $urls{$b} <=> $urls{$a} } keys %urls) {         print "$urls{$url}\t$url\n";     } 

Your MDS logs appear with your other BlackBerry logs in the following directory where YYYYMMDD is the current date (assuming you've installed your BES on the C: drive):

 C:\Program Files\Research In Motion\BlackBerry Enterprise Server\Logs\YYYYMMDD\ 

The Mobile Data Service logs will start with your BES name followed by the string MDAT.

 BESNAME_MDAT_01_20050607_0002.txt 

Type the code into your text editor and save it as site_extract.pl.

8.3.2. Run the Code

Bring up a command prompt, change directory (cd) to the directory where you saved the file, and type the following command to run the script:

 C:>perl site_extract.pl  

You'll get a list of URLs preceded by the number of instances they appear in your logfile, as shown in the following:

 4 http://www.cnn.com/ 3 http://dave.runningland.com/index.php 1 http://www.google.com/ 

You could easily pipe the output to a file and open the data using a spread-sheet program such as Microsoft Excel.

 C:>perl site_extract.pl > todays.frequent.sites.txt