Integration of Require JS into the system

Introduction

For those of you who don't know, Require JS is a JavaScript module loader. It is one of the available implementations of the CommonJS API standard. And I should add that it is quite good at what it does

Currently in the system we are in a state of a mess when it comes to JavaScript. What I mean is that we are using a combination of libraries, modules, single JS files, etc., which are written both in CommonJS module style, some other module style, plain-old let's-polute-the-global-namespace style, or support both. On top of that, all JS scripts are added to pages via <script> tags, or are inserted into the <head> element at run-time by the mitx/common/lib/xmodule/xmodule/js/src/javascript_loader.coffee loader. This is not optimal. Why?

There are many reasons, some of which I will outline:

No standard way to specify dependencies from within JS files, that would then be loaded automatically. We have to include all dependencies as <script> tags beforehand.
No out-of-the-box support for asynchronous loading of JS files. When all JS scripts are specified on the page as <script> tags, the browser will load and execute them one by one.
No easy way to modularize your code. This is especially important, because sometimes a large JavaScript application is written which is then used on separate pages, where each page requires some part of it's functionality. This naturally suggest that code modularization should be used, and that only the necessary modules should be loaded on a page. I will repeat myself, in the current system such implementation would have to be written in a custom way.

RequireJS fixes all of the above problems, and adds a lot more cool functionality on top of that.

There is one problem though. We can't use it directly in our current system.

If we just include it before everything else gets loaded, then things will break. This is because JS libraries that have been written with CommonJS in mind, test to see if the function define() is available in the global namespace. If it is, then the behaviour of the module changes. I will not go into the detail of how it changes, but what I will point out is that most of the time when a library discovers that define() was used, it will try to use it to define itself as an anonymous module. As soon as this happens, RequireJS breaks, because it assumes that all modules that use define() are loaded by it, rather than being included in the page in a <script> tag. Please see the official explanation of this problem. Other conflicts can also arise.

So, in short, we can't use RequireJS in a standard way without rewriting current JavaScript implementation system-wide. But there are ways to use RequireJS on a local level for creating stand-alone JS applications. I will described one of them. This solution was arrived at in a joint-effort/discussion by Jean-Michel Claus, Carlos Andrés Rocha, Christina Roberts, Alexander Kryklia, and me (Valera Rozuvan).

Proposal

Instead of including the original RequireJS, we should include a modified RequireJS. The modified RequireJS would be namespaced, and available for use to any script that is aware of the existence of the namespaced RequireJS. We should make it available globally, so that all developers can use it alongside of the giants jQuery, and Underscore. It will make life much easier.

Here is actual code implementation of how to namespace RequireJS. What it does, is create a global object RequireJS, putting the functions requirejs(), require(), and define() (which are normally defined by RequireJS in the global namespace) as it's members:

var RequireJS = function() {

/*
 RequireJS 2.1.2 Copyright (c) 2010-2012, The Dojo Foundation All Rights Reserved.
 Available via the MIT or new BSD license.
 see: http://github.com/jrburke/requirejs for details
*/
var requirejs,require,define;
(function(Y){ ... })(this);

return {
    'requirejs': requirejs,
    'require': require,
    'define': define
};
}();

This would be placed in a file /mitx/common/static/js/vendor/RequireJS.js, and also added to the configuration array main_vendor_js in the file /mitx/lms/envs/common.py like so:

main_vendor_js = [
     'js/vendor/RequireJS.js',
     'js/vendor/jquery.min.js',
     'js/vendor/jquery-ui.min.js',
     'js/vendor/jquery.cookie.js',
     'js/vendor/jquery.qtip.min.js',
     'js/vendor/swfobject/swfobject.js',
]

Then, whenever a developer wants to use RequireJS for his JavaScript project, he will simply wrap all of his code (either one file, or doing so in each of his files), in an anonymous function call, which will make available the standard requirejs(), require(), and define() functions like so:

// Wrapper for RequireJS. It will make the standard requirejs(), require(), and
// define() functions from Require JS available inside the anonymous function.
(function (requirejs, require, define) {

// For documentation please check:
//     http://requirejs.org/docs/api.html
requirejs.config({
    // Because require.js is included as a simple <script> tag (and is
    // forcefully namespaced) it does not get it's configuration from a
    // predefined 'data-main' attribute. Therefore, from the start, it assumes
    // that the 'baseUrl' is the same directory that require.js itself is
    // contained in - i.e. in '/static/js/vendor'. So, we must specify a
    // correct 'baseUrl'.
    //
    // Require JS initially searches this directory for all of the specified
    // dependencies. If the dependency is
    //
    //     'sylvester'
    //
    // then it will try to get it from
    //
    //     baseUrl + '/' + 'sylvester' + '.js'
    //
    // If the dependency is
    //
    //     'vendor_libs/sylvester'
    //
    // then it will try to get it from
    //
    //     baseUrl + '/' + 'vendor_libs/sylvester' + '.js'
    //
    // This means two things. One - you can use sub-folders to separate your
    // code. Two - don't include the '.js' suffix when specifying a dependency.
    //
    // For documentation please check:
    //     http://requirejs.org/docs/api.html#config-baseUrl
    'baseUrl': '/static/3.091x/js',

    // If you need to load from another path, you can specify it here on a
    // per-module basis. For example you can specify CDN sources here, or
    // absolute paths that lie outside of the 'baseUrl' directory.
    //
    // For documentation please check:
    //     http://requirejs.org/docs/api.html#config-paths
    'paths': {

    },

    // Since all of the modules that we require are not aware of our custom
    // RequireJS solution, that means all of them will be working in the
    // "old mode". I.e. they will populate the global namespace with their
    // module object.
    //
    // For each module that we will use, we will specify what is exports into
    // the global namespace, and, if necessary, other modules that it depends.
    // on. Module dependencies  (downloading them, inserting into the document,
    // etc.) are handled by RequireJS.
    //
    // For documentation please check:
    //     http://requirejs.org/docs/api.html#config-shim
    'shim': {
        'raphael': {
            'exports': 'Raphael',
            'deps': []
        },
        'sylvester': {
            'exports': 'Sylvester',
            'deps': []
        }
    }
}); // End-of: requirejs.config({

// Start the main app logic.
requirejs(['crystallography_module'], function (crystallography_module) {
    crystallography_module.configureProblems();
}); // End-of: requirejs(['crystallography_module'], function (crystallography_module)

// End of wrapper for RequireJS. As you can see, we are passing
// namespaced Require JS variables to an anonymous function. Within
// it, you can use the standard requirejs(), require(), and define()
// functions as if they were in the global namespace.
}(RequireJS.requirejs, RequireJS.require, RequireJS.define)); // End-of: (function (requirejs, require, define)

This code is taken from an actual implementation of this approach. It was tested, and is available on:

in files:

https://github.com/MITx/3.091x/blob/feature/alex/crystallography-update/static/js/crystallography.js (main JS script)
https://github.com/MITx/3.091x/blob/feature/alex/crystallography-update/static/js/crystallography_module.js (module JS script)
https://github.com/MITx/mitx/blob/feature/alex/crystallography-update-mitx/common/lib/capa/capa/templates/crystallography.html (the HTML template which includes main JS script)

The RequireJS.js and it's inclusion are here:

TODO

It is possible to automate the process of wrapping developed JavaScript applications in anonymous functions. RequireJS optimization tool can help with this. See the example configuration file for r.jshttps://github.com/jrburke/r.js/blob/master/build/example.build.js, especially the option wrap.
Add Jasmine tests to check that requirejs(), require(), and define() do not get through into the global namespace. Also can check that the global object RequireJS is available, and that it provides the 3 mentioned functions as its members. This has been implemented. See the section Testing below.

Testing

Jasmine tests for the namespaced RequireJS have been setup in the repository mitx on the branch valera/require-js. Three test specs have been added, and all three pass. Their addition did not change the number of other passing and failing specs. Please see the added spec file at https://github.com/MITx/mitx/blob/valera/require-js/lms/static/coffee/spec/requirejs_spec.coffee.

Currently the test specs do:

test for the fact that the functions requirejs(), require(), and define() have been removed from the global namespace;
test that the above mentioned functions are available via the global variable RequireJS;
test that no Require JS functionality has been broken - i.e. that we can define a module with RequireJS.define() and then require it with RequireJS.require().

Xmodule

I have also made progress towards getting the proposed namespaced RequireJS approach working with the Xmodule system. It is simple if you follow a few rules.

First of all, when you specify a list of JavaScript dependencies for a given Xmodule, it will make them available via <script> tags. Another thing to look out for, is that the scripts will not be named as they were originally, and the folder hierarchy will be transparent to the browser. I.e., if your source is positioned like so (this is just an example of my current setup for the project I am working on):

mitx/common/lib/xmodule/xmodule/js/src/graphical_slider_tool/mod1.js
mitx/common/lib/xmodule/xmodule/js/src/graphical_slider_tool/mod2.js
mitx/common/lib/xmodule/xmodule/js/src/graphical_slider_tool/mod3.js
...
mitx/common/lib/xmodule/xmodule/js/src/graphical_slider_tool/subfolder/mod4.js
...
mitx/common/lib/xmodule/xmodule/js/src/graphical_slider_tool/gst.js

and you specify them in your module.py file like so:

    js = {
      'js': [
        resource_string(__name__, 'js/src/graphical_slider_tool/gst_main.js'),
        resource_string(__name__, 'js/src/graphical_slider_tool/mod1.js'),
        resource_string(__name__, 'js/src/graphical_slider_tool/mod2.js'),
        resource_string(__name__, 'js/src/graphical_slider_tool/mod3.js'),
		...
        resource_string(__name__, 'js/src/graphical_slider_tool/subfolder/mod4.js'),
		...
		resource_string(__name__, 'js/src/graphical_slider_tool/gst.js')
		]
    }

then Xmodule will insert something along the lines of:

<script type="text/javascript" src="/static/coffee/module/0-53c44c016974a4e7b767192c8fa16fe0.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/0-4c266e3c951cb743e23a1ba5ea4e894c.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/0-294fdd5e1c647d8c8fe25151f72318df.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/0-b6433f29bab2b97672aec55279db3b63.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/0-0b45db8d4ae67b439cff8d0c38d807a8.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/0-5a98d4e2455b2de4c877452e90b088f5.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/0-8288ce4c66083efc8d6cb78d1b39d554.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/0-4d162d2de1aac3828ff8f47d82e5ed3c.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/1-d3dbe673fbfadd58a9a82117e804109d.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/1-d1c608cd2f93103ca8bff9971ad348b8.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/1-d69f48cece74fe77a82b1dcc7ba0275c.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/1-1450c43fbd4f4d4a15dc8b1c4041187e.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/1-da3bf0da3066f06a945b9a760a7b93e4.js" charset="utf-8">/script>
<script type="text/javascript" src="/static/coffee/module/2-e644bb83a9c91fdda7efceedb2dff7a9.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/2-53c44c016974a4e7b767192c8fa16fe0.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/2-6c67559dda8ce46b7aa8939a7a130358.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/2-2fc3119609d5119aac69775387420efd.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/3-ae4cdda5267741a8d62efa124a924f6f.js" charset="utf-8"></script>
<script type="text/javascript" src="/static/coffee/module/3-216700109846976775784acddc5c95c2.js" charset="utf-8"></script>

I have traced it, and found that the naming scheme is done in such a way, that each JavaScript file specified in the module.py file gets an index (starting with 0, and increasing by one). It also gets a hash (computed from several things, one being the original file name). Then, in the original order they were specified in, they get added to the HTML file being generated. This means that the order they will be processed by the browser is defined by the order in which they are listed in the module.py file (the browser processes all <script> tags one by one). What does this mean for the use of namespaced RequireJS?

First, RequireJS will get thrown off when trying to load dependencies. There is no simple way to map a module name to the name of the file it is in (that is being generated by Xmodule). This means that we must not require modules via RequireJS that are not already loaded into the browser. I.e. we can't require a module that RequireJS will have to load on it's own - the path it will build to get the module file will always result in a "file not found" error. But this is not a problem. It turns out that when a module is being defined by Require JS define() method, two things happen: the module ID (the name of the module by which it can then be required) gets recorded into Require JS internal modules stack, and the module callback function is stored but not executed. The module callback is called once, and only when that particular module is required by some other module. When some code does require the defined module, Require JS will first check it's internal modules stack to see if the requested module is already defined. If it is defined, it will call the callback (if it has not already done so), and return the module's result back to the requiring code. Only if a module was not defined before (it is missing from the internal modules stack), will it then try to get the JavaScript file associated with the module's name. This means that all modules we want to use have to be defined before we try to require them.

Second, we should have a single entry point with a require() call that lists all module dependencies, and a callback that will use those dependencies. And this should follow after all of the module definitions because we have to make sure all of the required modules get into the Require JS module stack so it doesn't start to get them on it's own via broken URLs (as explained, this will result in errors, and we don't want errors). Returning back to the module.py, you can see that gst.js is specified in the end. It is the main entry point. There is a require call which requires gst_main, mod1, mod2, mod3, mod4, etc. (not necessarily all at once). They are being defined in the appropriate files (graphical_slider_tool/gst_main.js, graphical_slider_tool/mod1.js, etc.) with define() calls.This works perfectly because each script is being included as a <script> tag and is processed one by one by the browser. No asynchronous misbehavior will be observed. As long as we include the JavaScript file with the main entry point last in the list, it will get executed last, after all of our modules have been added to Require JS module stack.

A sample project can be seen in the repository mitx on the branch valera/require-js-xmodule. Take a look at the files:

common/lib/xmodule/xmodule/gst_module.py

common/lib/xmodule/xmodule/js/src/graphical_slider_tool/gst.js
common/lib/xmodule/xmodule/js/src/graphical_slider_tool/gst_main.js
common/lib/xmodule/xmodule/js/src/graphical_slider_tool/mod1.js
common/lib/xmodule/xmodule/js/src/graphical_slider_tool/mod2.js
...

I am still testing this approach, but so far it gives no errors. It is a great way to bring about modularization to code - Require JS handles all the complex inter-module dependency loading. You just have to remember to only use require() once in one place after all of your necessary module define() statements.

NOTE: this page was migrated from https://edx-wiki.atlassian.net/wiki/display/ENG/Integration+of+Require+JS+into+the+system