Writing your own CommonJS module loader

As more and more JavaScript code uses  the CommonJS module spec for loading it became apparent that the work I had been doing in the Dojo Zazl project was going to have to support loading CommonJS modules. One of the driving factors for me was wanting to use the fantastic JavaScript compressor and parser Uglify-JS which is written in JavaScript itself.

The Uglify-JS code uses "require" for its dependency loading and if I wanted to use it in my Zazl code I would have to either modify the Uglify-JS code to remove the "require" statements and manually load them or make the Zazl JavaScript loader be CommonJS module compliant. Also I was interested in making my Zazl code run in NodeJS and that meant supporting CommonJS modules in the Zazl JavaScript too.

The existing Zazl JavaScript loader is implemented as a native method called "loadJS". When using Rhino "loadJS" uses the Rhino API's to compile and load. Likewise for V8 it's API's are used also. Loading a CommonJS module is more involved than this though. In addition to compiling and loading it has to also minimally :

1. Provide a sandbox for the module such that access to other globals is restricted
2. Provide a "require" method to enable loading of other modules. 
3. Track paths such that requests to "require" containing relative paths (starts with ./ or ../) are resolved correctly
4. Provide an "exports" variable that the module can attach its exports to.
5. Provide a "module" variable that contains details about the current module.

My current loadJS support now had to be somewhat smarter to support these requirements. The solution I came up with was to write a common "loader" JavaScript module that can be used in both Rhino and V8 environments. In support of this new loader code a new native method had to be provided for both Rhino and V8 called "loadCommonJSModule". In addition to the "path" parameter passed to "loadJS" this new method also expects a "context" object that contains the values for "exports" and "module". The native methods ensures that this context object is used for the modules parent context, basically its global object.

Doing this in Rhino is fairly straightforward. The java code that supports the "loadCommonJSModule" call has to create a org.mozilla.javascript.Scriptable object. The commonjs loader makes the call :

loadCommonJSModule(modulePath, moduleContext);

And the Rhino based Java code does this :

Scriptable nativeModuleContext = Context.toObject(moduleContext, thisObj);

classloader.loadJS(resource, cx, nativeModuleContext);

The thisObj parameter is what Rhino has passed to the java invocation of "loadCommonJSModule". The classloader object is a RhinoClassLoader (see another blog post for more details) used to create an instance of the module script and use the moduleContext object for its scope.

In V8 it's a little more involved. A new V8 Context is created for each module load. Creating Contexts in V8 is cheap so the performance overhead should be minimal. For the new Context object created each attribute in the provided moduleContext is copied in. This new Context object is then used to run the module script in. The following is some code snippets from the v8javabridge.cpp file.

v8::Handle<v8::ObjectTemplate> global = CreateGlobal();

v8::Handle<v8::Context> moduleContext = v8::Context::New(NULL, global);

v8::Handle<v8::Value> requireValue =

context->Global()->Get(v8::String::New("require"));

v8::Context::Scope context_scope(moduleContext);

moduleContext->Global()->Set(v8::String::New("require"), requireValue);

v8::Local<v8::Object> module = args[1]->ToObject();

v8::Local<v8::Array> keys = module->GetPropertyNames();
            

unsigned int i;

for (i = 0; i < keys->Length(); i++) {
       v8::Handle<v8::String> key = keys->Get(v8::Integer::New(i))->ToString();
       v8::String::Utf8Value keystr(key);
       v8::Handle<v8::Value> value = module->Get(key);
       moduleContext->Global()->Set(key, value);

}

You can see the JavaScript loader code here

Running CommonJS code in Zazl now is just a matter of :

loadJS('/jsutil/commonjs/loader.js');

require('mymodule');

For validation I used the CommonJS set of unit tests to check my loader runs correctly to the spec.

I have some gists that provide both a RhinoCommonJSLoader and a V8CommonJSLoader. Using a Multiple Rooted Resource Loader its pretty easy to run common js based code from Java:

File root1 = new File("/commonjs/example");
File root2 = new File("/commonjs/runtime");
File[] roots = new File[] {root1, root2};
ResourceLoader resourceLoader = new MultiRootResourceLoader(roots); 
try {
    RhinoCommonJSLoader loader = new RhinoCommonJSLoader(resourceLoader);
    loader.run("program"); 
} catch (IOException e) {
    e.printStackTrace(); 
} 

The /commonjs/runtime root must contain the contents of the Zazl jsutil JavaScript directory pathed such that the runtime directory contains the jsutil directory. You can place a "program.js" and its dependencies in the /commonjs/example directory and the loader will perform a "require('program'); to load it.