Hack 71. Improve Maintainability, Performance, and Reliability for Large JavaScript Applications

Discover options for improving the maintainability, performance, and reliability of Ajax applications that have a large amount of JavaScript code.

Most Ajax applications contain far more JavaScript than typical web pages. The total amount of JavaScript in an Ajax app can easily exceed three or four hundred kilobytes, segmented across many separate files. Making sure these files are easy to maintain while also ensuring that they download quickly is very important. In addition, some browsers have bugs that manifest when new releases of JavaScript files are pushed out, hindering the reliability of large-scale JavaScript applications.

This hack presents techniques for compressing JavaScript files without sacrificing readable, maintainable code; improving page load times when dealing with many JavaScript files; and increasing the reliability of pushing out new versions of your JavaScript applications to web browsers.

It explains three techniques:

• Merging all JavaScript files into a single file

• Running your JavaScript through a compression tool

• Solving JavaScript caching issues

bash shell commands are used to implement these techniques. These bash shell commands are meant to be run as part of a large-scale JavaScript application's build process, before pushing the code to a production or development server. In your own application, you could choose to use other build tools to implement these techniques, such as Ant, Make, or Rake.

If you are on Windows and wish to use the bash shell commands in this hack, download the free, open source Cygwin package from http://www.cygwin.com. Having Cygwin installed on Windows is an absolute must for serious Ajax and web development.

Merging All JavaScript Files into a Single File

A typical, large-scale Ajax application incorporates many separate JavaScript files. Some files will be third-party frameworks and libraries that ease development, while others will be portions of the application that have been segmented into different files to ease development and maintenance. Due to network latency, in many Ajax applications a significant amount of time can be required for the browser to fetch all of these JavaScript files.

One way to minimize page load times is to simply concatenate all of the individual JavaScript files into a single JavaScript file before pushing your application to production. This has been found to have a drastic effect on the startup performance of many Ajax applications.

To merge a series of JavaScript files into a single file named all.js, for example, you can run the following bash shell commands:

cat script1.js \\     script2.js \\     script3.js \\     > all.js

cat is a shell command for creating, viewing, and concatenating files.

You should typically not use a wildcard when merging your JavaScript files because the ordering of the merged files will be unknown. Most JavaScript files have dependencies and must usually be loaded in a certain order. For example, if your application is using the Prototype or Dojo Toolkit frameworks, you should merge those together first:

cat dojo.js \\     prototype.js \\     myScript.js all.js

Once you have merged the files, in your main application's HTML page, simply load the all.js file:

<!-- Our merged JavaScript --> <script src="/books/4/254/1/html/2/all.js"></script>

If you keep a commented-out code block that loads each individual file separately, you can simply uncomment this while commenting out the all.js script load to ease debugging if errors arise:

<!-- Our merged JavaScript --> <!-- <script src="/books/4/254/1/html/2/all.js"></script> --> <!-- Individual JavaScript files; useful for debugging. --> <script src="/books/4/254/1/html/2/dojo.js"></script> <script src="/books/4/254/1/html/2/prototype.js"></script> <script src="/books/4/254/1/html/2/myScript.js"></script>

When you are finished debugging, uncomment the all.js script load and recomment the loading of the individual files:

<!-- Our merged JavaScript --> <script src="/books/4/254/1/html/2/all.js"></script> <!-- Individual JavaScript files; useful for debugging. --> <!--  <script src="/books/4/254/1/html/2/dojo.js"></script> <script src="/books/4/254/1/html/2/prototype.js"></script> <script src="/books/4/254/1/html/2/myScript.js"></script> -->

Make sure that the last line of each file has an extra carriage return, or the merged JavaScript files may overlap in ways that can cause errors.

Running Your JavaScript Through a Compression Tool

Traditionally, programmers have been faced with two options when designing JavaScript-heavy DHTML applications: they can either write extremely terse code with no comments, in order to minimize the size of their files; or they can choose to segment their JavaScript across several files, with descriptive method names and ample source comments.

In the past, programmers had to make a decision between small JavaScript file sizes and improved ease of maintenance and readability of the code. Most programmers chose smaller file sizes, leading to nightmare code that was unreadable and difficult to scale to larger applications.

This hack removes the need to compromise: you can write your application with descriptive source comments and fully object-oriented methods, and an open source JavaScript compression tool from the Dojo Toolkit (http://www.dojotoolkit.org) will strip out all comments and fully compress your code.

The Dojo Toolkit has created a full open source JavaScript compression tool based on the Mozilla Foundation's Rhino JavaScript parser. Commercial companies can use this compression tool freely. Full documentation on the Dojo compressor is available at http://www.dojotoolkit.org/docs/compressor_system.html.

To use the Dojo compressor in your own system, first download and install the Java JDK 1.4+ on the machine you will use for compression (if necessary). Next, download the compression tool from http://www.dojotoolkit.org/svn/dojo/trunk/buildscripts/lib/custom_rhino.jar, and save it to your hard disk. Once you have merged your JavaScript files, as described earlier in the section "Merging All JavaScript Files into a Single File," run the following command:

java -jar custom_rhino.jar -c all.js > all_compress.js 2>&1 mv all_compress.js all.js

If you look at the all.js file, you will see that the file size has been considerably reduced.

The Dojo Toolkit and the JavaScript compressor are currently in beta, and the generated compressed code can cause an error in some rare cases, such as when using JavaScript closures combined with certain styles of object-oriented JavaScript programming. If you find that you are getting JavaScript errors after compressing your JavaScript, go into the original premerged, precompressed source code and slightly rewrite the line of code on which you received the error. This usually fixes the problem.

Solving JavaScript Caching Issues

The final major technique for improving large-scale JavaScript applications concerns caching. Internet Explorer currently has a serious bug in which cached versions of JavaScript files are used even if the server has newer versions. If the HTML page references JavaScript functions that are not included in the older files in the cache, users will see Internet Explorer's script error dialog and experience a broken application.

In the past, this bug was often encountered when rapid iterations of an Ajax application were pushed out, forcing users to know either how to manually clear the browser cache or to press Ctrl and click the Refresh icon to force the browser to grab the file from the server. Neither method is user-friendly or reliable enough.

The secret to solving the caching bug in Internet Explorer is to trick the browser into encoding a version into the filename:

<script src="/books/4/254/1/html/2/all.js?version=1"></script>

Internet Explorer will "see" the filename of this JavaScript file as all.js?version=1. If you then created a new version of all.js, you could simply increment the version number:

<script src="/books/4/254/1/html/2/all.js?version=2"></script>

To Internet Explorer, these are two different JavaScript files. The version parameter doesn't affect the JavaScript file's execution, but it tricks the caching system into correctly loading any newer versions of affected JavaScript files.

Hand-editing these values can get tedious quickly. One possible workaround is to introduce a token into your HTML (or JSP, PHP, etc.) files that holds the current version. In the following code, the token version= is used, starting with an initial version of 1:

<script language="JavaScript" src="/books/4/254/1/html/2/./scripts/all.js?version=1"></script> <!-- Uncomment when debugging --> <!-- <script language="JavaScript" src="/books/4/254/1/html/2/./scripts/script1.js?version=1"></script> <script language="JavaScript" src="/books/4/254/1/html/2/./scripts/script2.js?version=1"></script> -->

You then create a bash shell script that can load our HTML, JSP, and other files, grab the current value after version= and increment its value, and then rewrite it into the files:

export HTML_FILE=$SRC/sample.html # Any file that wishes to use this must have the following magic # token, version=######, that we read in, increment, # and then write back out. Internet Explorer incorrectly  # caches JavaScript files even if they have changed, # causing versioning issues when new ones are pushed out; # this solves this problem. # read in the current version oldVersion=\Qgrep -o \\version=[0-9]* $HTML_FILE | tail -n 1 |  sed "s/version=//"\Q # increment the value newVersion=$(( oldVersion + 1)) # write the new version back out into the file sed "s/version=$oldVersion/version=$newVersion/" $HTML_FILE >  $HTML_FILE.new mv $HTML_FILE.new $HTML_FILE

Your files will now always increment when you push out a new version of the JavaScript file, solving the caching issues in Internet Explorer.

This technique can be implemented in other ways, such as using an application-scope JavaServer Pages variable that is dynamically written into your JSPs, rather than using bash shell scripts.

All Together

These three techniques are normally used together during the build phase of a project, so putting them into a single shell script makes sense. The following bash shell script can achieve all three together:

#!/bin/bash # Performance booster for page load time; we bring all of the JavaScripts into # one file, which prevents having to fetch each JavaScript file individually, # which can be the number one impact on page load performance for Ajax apps. # Then, we compress the final JavaScript file to reduce its size. # We also use a technique to solve Internet Explorer's JavaScript cache bugs. # # @author, Brad Neuberg, [e-mail] # This script is under a BSD license and is freely usable export SRC=./demo/compress export SCRIPTS=$SRC/scripts export HTML_FILE=$SRC/sample.html rm -fr dist mkdir dist # Any file that wishes to use this must have the following magic  # token, version=######, that we read in, increment, and then # write back out. Internet Explorer incorrectly caches JavaScript # files even if they have changed, causing versioning issues when  # new ones are pushed out; this solves this problem. # read in the current version oldVersion=\Qgrep -o \\version=[0-9]* $HTML_FILE | tail -n 1 | sed "s/version=//"\Q # increment the value newVersion=$(( oldVersion + 1)) # write the new version back out into the file sed "s/version=$oldVersion/version=$newVersion/" $HTML_FILE > $HTML_FILE.new mv $HTML_FILE.new $HTML_FILE # concatenate all code into one file cat $SCRIPTS/script1.js \\     $SCRIPTS/script2.js \\     > dist/all.js      # now compress it java -jar bin/compress.jar -c dist/all.js > dist/all_compress.js 2>&1 # install it cp dist/all_compress.js $SCRIPTS/all.js # clean up rm -fr dist

You have to tailor this script for your own application. Change the SRC variable to point to the directory where you keep all of your application's source files, change SCRIPTS to point to the directory in SRC in which you hold your JavaScript, and change HTML_FILE to point to the HTML file that loads your JavaScript. You must also edit the section that merges the JavaScript files so that your JavaScript files are loaded in the correct order.

Brad Neuberg