Optimizing with AMD JavaScript loaders

In a previous blog post I talked about writing JavaScript optimizers that run dynamically. That is to say when a Web Application is serving JavaScript resources the server itself is performing optimizations to ensure the JavaScript is returned to the web client in the most efficient way possible,

AMD (Asynchronous Module Definition) is rapidly becoming the preferred type of loader to use when developing Web Client based JavaScript. Its "define' API provides a way for modules to declare what other resources (other JavaScript modules, text, i18n etc) they depend on. This information is key to how an optimizer running in the server can ensure that when a request is received for a JavaScript resource all its dependencies can be included in the response too. This avoids the loader code running in the web client having to make additional HTTP requests for these dependencies.

So how does one obtain this dependency information? Probably the most reliable way is to use a JavaScript Lexer/Parser to parse the code and analyze the calls to the "define" and "require" API's.  There are a number of options here (Mozilla Rhino and Google Closure, both Java based, provide AST parsers) however this typically mean having to pick a environment or language. For the AMD Optimizer that I wrote I decided to use the parser that is part of Uglify-JS. As it is written in JavaScript itself I had more flexibility in the environments where I wanted to run it.

The Uglify-JS AST parser API provides a mechanism to walk the AST's that are returned by the parser. The parse call to obtain the AST object is a one time step. You can then walk the AST as many times as you need. The module "process.js" exports an function called "ast_walker". You can use this function to walk the AST with your provided walker function.

Here is an example of an AST walker that obtains the position within a module where a name/id should be inserted into a "define" call if one is not provided (The AMD optimizer I have written uses this walker to perform any name insertions that are required).

var jsp = require("uglify-js").parser;
var uglify = require("uglify-js").uglify;
var ast = jsp.parse(src, false, true); 
var w = uglify.ast_walker(); 
var nameIndex = -1; 
w.with_walkers({
    "call": function(expr, args) {
        if (expr[0] === "name" && expr[1] === "define") {
            if (args[0][0] !== "string") {
                nameIndex = w.parent()[0].start.pos + (src.substring(w.parent()[0].start.pos).indexOf('(')+1);
            }
        }
    } 
}, function(){
    w.walk(ast); 
});

The walker looks for "call" statements that have a name of "define". The "expr" argument provides this information. The "args" argument provides the details of the type and value of the arguments provided to the "call". In this example the code checks that the type of the first argument is not a string and records the position that the name should be inserted. The walker object (w) provides access to the parent object that contains the position in the source code.

Using it in a Dynamic Environment

For the Optimizer that I have written for AMD based environments (the walker code can be seen here) I use a single AST walker function. The walker code itself recursively walks down the chain of modules recording the relevant information that each module provides. Given a module url for a starting point it obtains the information for each dependency module, placing it in a data structure and adding it to a module map using the module uri as a key.

The walker records the following :

• For each module a list of JavaScript dependencies
• A list of "text" dependencies
• A list of "i18n" dependencies
• If a module does not contain an id then records its index within the source where and id should be placed.

With this information now available it's possible to write a HTTP handler that can write a single streams of content that contains :

• An AMD compliant loader 
• A "i18n" plugin handler
• A "text" plugin handler
• Each "text" resource as a separate "define"'d module
• Each "i18n" resource (see below for more details)
• Each required module in it correct dependency order

Note:  All modules with have an id added if one is not present

i18n dependencies
An i18n dependency is declared in the form "i18n!<..../nls/...>". If the required locale value is available (with HTTP this can be obtained by parsing the "accept-language" for the best fit) the set of messages can be provided in separate AMD modules that will be merged together by the i18n plugin. When processing what has to be written into the response stream the Optimizer will provide up to 3 separate modules based on :

• The root locale
• The intermediate locale
• The full locale

For example given the the i18n dependency "i18n!amdtest/nls/messages" and the request locale is "fr-fr" then the Optimizer will look for the :

• root module in "amdtest/nls/messages.js"
• intermediate module in "amdtest/nls/fr/messages.js"
• full module in "amdtest/nls/fr_fr/messages.js"

Seeing it in action
You can see some samples of the AMD optimizer if you download

• amdoptimizerjsp.war  (load into a JEE webcontainer and access "/amdoptimizerjsp/amdcalendar.jsp" or "/amdoptimizerjsp/amddeclarative.jsp")
• zazlnodejs.tar.gz (run node zazlserver.js ./amdsamples ./dojo16 requirejs.json)