Note that we have migrated from Vagrant to Docker for our devstack. Read the new instructions instead.
The Open edX devstack is a Vagrant instance designed for local development. The instance:
The Open edX devstack is also designed to run code and tests, but you can do most development in the host environment:
The Open edX devstack configuration has the following components:
For comparison, look at the Full Stack page.
Follow the instructions at http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/installation/devstack/index.html
In section 3.3.1.2 Install Devstack, be sure to set the OPENEDX_RELEASE environment variable to "master". This downloads the latest version of devstack, instead of a named release.
Once the VM is created, log in to the VM via the command: vagrant ssh
If you need more control over the vagrant configuration, you can download the Vagrantfile and run vagrant yourself:
curl -OL https://raw.github.com/edx/configuration/$OPENEDX_RELEASE/vagrant/release/devstack/Vagrantfile vagrant plugin install vagrant-vbguest vagrant up |
--provider virtualbox" to "vagrant up" when running it for the first time, for example in case of some libvirt-related error occurring if you have more Vagrant providers installed on your system.OPTIONAL [pertains to Windows and others having issues with NFS]: To use VBOXFS, set in your shell the environment variable VAGRANT_USE_VBOXFS = true
OPTIONAL [pertains to unix/linux and mac hosts]: To use "preview" from within Studio, you will need to add a line, 192.168.33.10 preview.localhost to your development machine's/etc/hosts file. For example:
$ cat /etc/hosts # Host Database 127.0.0.1 localhost 255.255.255.255 broadcasthost ... $ # if preview.localhost doesn't already exist: $ sudo bash -c "echo '192.168.33.10 preview.localhost' >> /etc/hosts" $ cat /etc/hosts # Host Database 127.0.0.1 localhost 255.255.255.255 broadcasthost ... 192.168.33.10 preview.localhost |
OPTIONAL - if you wish to customize the location of the source code that gets cloned during devstack provisioning, or if you prefer to work on source trees that already exist on your system, you can set the VAGRANT_MOUNT_BASE env var, which sets the base dir for the edx-platform and cs_comments_service. By default it will be wherever you run vagrant up.Example:
$ cd $HOME/my-workspace/my-edx-workspace $ # This is where you either want vagrant provisioning to clone, $ # or you have existing repositories (although note that vagrant provision will overwrite them): $ ls -F $ # You might optionally need to rename edx-ora to ora cs_comments_service/ edx-platform/ ora/ $ # Now, set the environment for vagrant $ # (you may want to put this in a file, and 'source' it instead) $ export VAGRANT_MOUNT_BASE=$PWD |
vagrant up, vagrant sshsudo su edxapp/edx/app/edxapp/edxapp_env) so that the venv python and rbenv ruby are in your search path./edx/app/edxapp/edx-platform). If you get an error: Unknown task: devstack, the working directory has not been updated properly, simply do cd /edx/app/edxapp/edx-platform and re-run the command.If you are running the developer stack with the standard environment file, lms/envs/devstack.py, then you can use this command:
paver devstack lms |
There is a faster version of this command that starts the server without updating requirements or compiling static assets. To ONLY start the server:
paver devstack --fast lms |
Note: you can type paver --help to list commands, or paver [command] --help for command options.
TEMPORARY FIX: If, upon first bringup of lms, you get an error message similar to this:
OperationFailure: command SON([('authenticate', 1), ('user', u'edxapp'), ('nonce', u'nonce-value'), ('key', u'key-value')]) failed: auth fails |
Then sudo run this script: fix-mongo.sh. To download the script to your dev env, visit the link above and copy its url from the url bar. Then in your devstack use this command, pasting the url inside the quotes:
curl -O "fix-mongo-url" |
Note: you'll need to 'sudo' as the vagrant user, not the edxapp user. If you've already run:
sudo su edxapp
you will need to run exit to switch context back to the Vagrant user. For more info, see this thread.
Open a browser on your host machine and navigate to http://localhost:8000/ to load the LMS. (Vagrant will forward port 8000 to the LMS server running in the VM.)
The latest version of devstack has the demo course pre-loaded and dummy accounts, you can log in to the website as:
If you are bringing up LMS with a custom configuration (For example devstack as for developer stack and devstack.py, but you name yours as you require):
Update requirements, and compile Sass and CoffeeScript:
paver update_assets --settings devstack lms |
If there is a change to layout, or if you ran the server with the --fast flag from the start and are wondering why all of the css/images are missing, you will need to recompile the css and other assets as well:
paver update_assets |
Note: paver -h will show other available commands
Start your custom configured server:
./manage.py lms runserver --settings=devstack 0.0.0.0:8000 |
Note: ./manage.py lms -h --settings=devstack will show other available commands on the server.
Within the VM instance, switch to the edxapp user:
sudo su edxapp
This will source the edxapp environment (/edx/app/edxapp/edxapp_env) so that the venv python and rbenv ruby are in your search path. It will also set the current working directory to the edx-platform repository (/edx/app/edxapp/edx-platform).
Note: the following steps are for running with the standard configuration file. To run Studio with a custom configuration, follow the instructions as for LMS, above, replacing lms with cms in the commands.
If you are running the developer stack with the standard environment file, cms/envs/devstack.py, then you can use this command:
paver devstack studio
There is a faster version of this command that starts the server without updating requirements or compiling assets. To ONLY start the server:
paver devstack --fast studio
Open a browser on your host machine and navigate to http://localhost:8001/ to load Studio. (Vagrant will forward port 8001 to the Studio server running in the VM.)
OPTIONAL: you can see what other commands are available on the server (such as creating users, collecting static files, etc) with:
./manage.py cms -h --settings=devstack |
Within the VM instance, switch to the forum account
sudo su forum |
Update Ruby requirements (Note that if you get a message for entering a password to install the bundled RubyGems to the system, you can safely ctrl-c out of that. The gems will still be installed correctly for the forum user.)
bundle install |
Start the server
ruby app.rb -p 18080 |
If you are running both LMS and Studio without the --fast switch, you will notice that while starting the either server, the LMS server will continuously redeploy itself and show a variant of this message repeatedly:
June 09, 2016 - 14:50:05 Django version 1.8.7, using settings 'lms.envs.devstack' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. June 09, 2016 - 14:50:14 Django version 1.8.7, using settings 'lms.envs.devstack' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. . . . |
This is due to the fact that when not runing in --fast mode, there is a watcher in place that will redeploy the server if a file change is detected, and during the startup of the Studio server, several files in the LMS instance are touched. As a workaround for this issue, either run LMS in --fast mode, or just wait for the file touching to complete.
If you plan to make changes that will result in a pull request for any of the devstack applications, you will likely want to update your devstack to the latest from the upstream master. To do so, first make sure that your devstack instance was from the latest devstack download and NOT from one of the named releases. If you defined OPENEDX_RELEASE before downloading devstack, than this is NOT the case. Once this is true, you'll want to switch to the edxapp user and update the code and databases.
First, Bring up the VM and log into it:
$ vagrant up . . . $ vagrant ssh |
Within the VM instance, switch to the edxapp user:
sudo su edxapp |
This will bring you to the edx-platform directory. Next, update the code to the latest on master:
$ git checkout master $ git pull |
Once you have the latest code, the database will be out of sync with the code changes. You will need to update the database by running the scripted migrations:
$ paver update_db |
See Developing on the edX Developer Stack once you are ready to start developing on devstack.
edxapp user:sudo su edxapp
paver test_python
NOTE: if these tests fail with an error about needing a sudo password, use visudo to update your sudoers file for the edxapp user:
visudo -f /etc/sudoers.d/01-sandbox
edxapp ALL=(sandbox) SETENV:NOPASSWD:/edx/app/edxapp/venvs/edxapp-sandbox/bin/python edxapp ALL=(sandbox) SETENV:NOPASSWD:/usr/bin/find edxapp ALL=(ALL) NOPASSWD:/usr/bin/pkill |
See also the CodeJail readme.
Run the JavaScript unit tests
paver test_js |
Run the LMS and Studio acceptance tests
paver test_acceptance |
See the docs on testing edx-platform for detailed information about writing and running tests.
Use vagrant provision to update the configuration and all repositories to the master branch. NOTE: this will perform a git clean on your repositories, so make sure you have committed any changes you want to save.
If you want to only update the configuration repo, vagrant ssh, sudo /edx/bin/update configuration master.
Logs for the running applications can be found in various application specific folders in/edx/var/log.
If you can't find the solution to your problem here, there is more documentation on the Open Source Project Page and ReadTheDocs
Sometimes there's a failure to checkout the edx-platform repo (as shown below). This can occur if the edx-platform repository is already checked out on your machine and the git URL for the origin remote is set to an SSH URL (e.g. git@github.com:edx/edx-platform.git) instead of HTTPS e.g.https://github.com/edx/edx-platform.git).
SSH keys are not installed on the Vagrant box so SSH remotes will not work. Use an HTTPS remote instead to resolve this issue. This can be set by using git remote set-url origin <https url> as the edxapp user.
TASK: [edxapp | checkout edx-platform repo into {{edxapp_code_dir}}] **********
failed: [localhost] => {"cmd": ["/usr/bin/git", "ls-remote", "origin", "-h", "refs/heads/master"], "failed": true, "item": "", "rc": 128}
stderr: Warning: Permanently added 'github.com,192.30.252.130' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
msg: Warning: Permanently added 'github.com,192.30.252.130' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
FATAL: all hosts have already failed -- aborting
PLAY RECAP ********************************************************************
to retry, use: --limit @/root/vagrant-devstack.retry
localhost : ok=125 changed=20 unreachable=0 failed=1
Stderr from the command:
stdin: is not a tty |
If you get the following error:
mount.nfs: mount to NFS server '192.168.33.1:/path/to/edx-platform' failed: timed out, giving up
This can be caused by a few issues:
Some Linux-based NFS servers do not support shares inside a FUSE mounted directory. The easiest fix is to move your development directory somewhere else.
You can fix the VPN conflict with these steps:
vagrant haltvagrant upNFS will now work whether your VPN is running or not.
VirtualBox can have issues when Guest Additions don't match the version installed when it comes to NFS mounts, which is fixed in version 4.3.12. In particular, VirtualBox 4.3.10 Guest Additions has an NFS mounting bug and can be fixed with
vagrant up
vagrant ssh -c 'sudo ln -s /opt/VBoxGuestAdditions-4.3.10/lib/VBoxGuestAdditions /usr/lib/VBoxGuestAdditions'
vagrant reload
If you try to reinstall devstack you might come upon this error:
NFS is reporting that your exports file is invalid. Vagrant does
this check before making any changes to the file. Please correct
the issues below and execute "vagrant reload":
that is because there are entries already in your /etc/exports file. To fix this issue, delete the block of the devstack that you're reinstalling within your /etc/exports file. An export block for devstack looks like this:
# VAGRANT-BEGIN: 502 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
"/Users/user/path/to/devstack/cs_comments_service" 192.168.33.10 -alldirs -mapall=502:20
"/Users/user/path/to/devstack/edx-platform" 192.168.33.10 -alldirs -mapall=502:20
"/Users/user/path/to/devstack/ora" 192.168.33.10 -alldirs -mapall=502:20
"/Users/user/path/to/devstack/themes" 192.168.33.10 -alldirs -mapall=502:20
# VAGRANT-END: 502 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
You can have a few this is you installed more devstacks, make sure you delete the right one. You can find the vagrant id for a particular devstack install by running this inside the devstack folder:
cat .vagrant/machines/default/virtualbox/id
If you see a ValueError like the following when working in the Django admin:
ValueError at /admin/certificates/certificatehtmlviewconfiguration/ Database returned an invalid value in QuerySet.datetimes(). Are time zone definitions for your database and pytz installed? |
Run the following:
$ mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql |
If that doesn't work, follow the instructions at https://bugs.mysql.com/bug.php?id=68861.
git clone https://github.com/edx/configuration.git
cd configuration/vagrant/base/devstack
vagrant up
/edx/app/edx/edx-platform directory in the Vagrant image is empty, it means the provisioning scripts did not complete successfully. Try re-running the provisioning scripts using vagrant provision.We need to verify your email address
Almost there! In order to complete your sign up we need you to verify your email address.
An activation message and next steps should be waiting for you there.
However, when running the stack locally, there is no email message sent. Rather the message comes out on the console, giving the activation key to use:
Thank you for signing up for edX Studio! To activate your account, please copy and paste this address into
your web browser's address bar:
http://localhost/activate/0699e17df6024fb382fadfabed100691
If you didn't request this, you don't need to do anything; you won't receive any more email from us. Please do
not reply to this e-mail; if you require assistance, check the help section of the edX web site.
Using the domain address ‘http://localhost/activate/… may not work when you copy this address into the web browser’s address field as instructed. if a ’Server not found’ message comes up in the browser, substitute
http://127.0.0.1:8001
for localhost. For example:
http://localhost/activate/a19de8c6330f4213ac3313c147297990
becomes
http://127.0.0.1:8001/activate/a19de8c6330f4213ac3313c147297990
Exception Type: DatabaseError
Exception Value:
(1146, "Table 'edxapp.new_foo' doesn't exist")
Exception Location: /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/MySQLdb/connections.py in defaulterrorhandler, line 36
there have been database migrations that you need to perform. Note this will not destroy your data. You can use the following command:
paver update_db -s devstack
pymongo.errors.ConnectionFailure: could not connect to localhost:27017: [Errno 111] Connection refused
or this one, when attempting to run the forums:
MOPED: Could not connect to any node in replica set <Moped::Cluster nodes=[<Moped::Node resolved_address="127.0.0.1:27017">]>, refreshing list.
The Vagrant instance not shut down properly. Use vagrant halt to safely stop the Vagrant instance. To fix this, try:
vagrant up
vagrant ssh
sudo rm /edx/var/mongo/mongodb/mongod.lock
sudo mongod -repair --config /etc/mongod.conf
sudo chown -R mongodb:mongodb /edx/var/mongo/.
sudo service mongod start
If using a pre-lavash version of devstack, do this instead:
vagrant up
vagrant ssh
sudo rm /edx/var/mongo/mongodb/mongod.lock
sudo mongod -repair --config /etc/mongodb.conf
sudo chown -R mongodb:mongodb /edx/var/mongo/mongo/.
sudo service mongod start
DatabaseError: (1146, "Table 'edxapp.dark_lang_darklangconfig' doesn't exist")
Then this means that your database is outdated and needs to be migrated.
sudo su edxapp
./manage.py lms migrate --settings=devstack
and then do the same thing for Studio with
sudo su edxapp
./manage.py cms migrate --settings=devstack
If you are seeing the following error when running your python tests, you will need to force an upgrade on one of the dependencies:
FactoryError: Cannot generate instances of abstract factory ClientFactory; Ensure ClientFactory.Meta.model is set and ClientFactory.Meta.abstract is either not set or False.
If you want to verify that you have this edx-oauth2-provider issue, you can use the following:
sudo su edxapp
~/edx-platform$ cat ~/edx-platform/requirements/edx/github.txt | grep "edx-oauth2-provider"
# Should display 0.5.7 or greater if this is your issue. This is the dependency you want.
pip freeze | grep “edx-oauth2-provider"
# Should display 0.5.1. This is the dependency you have.
To fix (still as edxapp user):
# force uninstall of bad dependency
pip uninstall edx-oauth2-provider
# run previously failing test that will install updated dependency and now pass
paver test_system -t lms/djangoapps/oauth2_handler/tests.py:UserInfoTest.test_request_staff_courses_with_claims
Your gems need updating (might happen when switching versions of cs_comment_service). Runbundle install to resolve this, and then retry.
If you are getting a password prompt after running bundle install, you will need to fix some directory permissions:
# NOTE do this as user vagrant, not user forum
sudo chmod -R 0770 /edx/app/forum/.rbenv
sudo chmod -R 0770 /edx/app/forum/.gem
Then sudo back to user forum, do bundle install again, and you should no longer be prompted.
If running tests seems broken, try removing the .testids from your project root. "git clean -fxd .testids" should also work. Aborting a test mid-run can leave those in a bad state. One error message we've seen a few times from this is "insecure string pickle". No doubt there are many other error messages which might result from this problem, but none that are as poetic!
It appears your machine doesn't support NFS, or there is not an
adapter to enable NFS on this machine for Vagrant. Please verify
that `nfsd` is installed on your machine, and try again. If you're
on Windows, NFS isn't supported. If the problem persists, please
contact Vagrant support.
you need to install nfs using this command:
sudo apt-get install nfs-common nfs-kernel-server
If you get an error such as NS_ERROR_FAILURE, try upgrading VirtualBox
If you are working on an Ubuntu workstation and your home directory is encrypted you will likely run into NFS problems. If you see this message:
exportfs: Warning: ... does not support NFS export.
To work around the issue set VAGRANT_MOUNT_BASE to a dir outside the encrypted volume.
IOError: [Errno 37] No locks available, that indicates that the host machine isn't running rpc.statd or statd. This has been seen to affect Ubuntu >= 15.04 (running systemd), and the resolution is to run systemctl rpc-statd start and systemctl rpc-statd enable.sudo systemctl enable rpcbind.service, sudo systemctl start rpcbind.service, sudo systemctl restart rpcbind.serviceTASK: [edxapp | Install edx-platform npm dependencies] ************************
failed: [localhost] => {"changed": true, "cmd": "npm install ", "delta": "0:00:0
3.358306", "end": "2014-03-24 16:51:37.509494", "item": "", "rc": 1, "start": "2
014-03-24 16:51:34.151188"}
stderr: npm http GET http://registry.npmjs.org/coffee-script
npm http 200 http://registry.npmjs.org/coffee-script
npm ERR! error installing coffee-script@1.6.1
npm ERR! Error: UNKNOWN, unknown error '../coffee-script/bin/coffee'
npm ERR! You may report this log at:
npm ERR! <http://bugs.debian.org/npm>
npm ERR! or use
npm ERR! reportbug --attach /edx/app/edxapp/edx-platform/npm-debug.log npm
npm ERR!
npm ERR! System Linux 3.2.0-23-generic
npm ERR! command "node" "/usr/bin/npm" "install"
npm ERR! cwd /edx/app/edxapp/edx-platform
npm ERR! node -v v0.6.12
npm ERR! npm -v 1.1.4
npm ERR! path ../coffee-script/bin/coffee
npm ERR! code UNKNOWN
npm ERR! message UNKNOWN, unknown error '../coffee-script/bin/coffee'
npm ERR! errno {}
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /edx/app/edxapp/edx-platform/npm-debug.log
npm not ok
FATAL: all hosts have already failed -- aborting
Go to Troubleshooting Vagrant and follow the instructions towards the bottom of the page about this and other symlink errors in Windows.
Sometimes, NFS fails the first time you start Vagrant on Mac OS X. If you see this message:
[default] Mounting NFS shared folders...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
mount -o 'vers=3,udp' 192.168.33.1:'/path/to/devstack/edx-platform' /edx/app/edxapp/edx-platform
Stdout from the command:
Stderr from the command:
stdin: is not a tty
mount.nfs: requested NFS version or transport protocol is not supported
then reload and re-provision the instance using these commands:
vagrant reload
vagrant provisionIf the virtual machine hangs on the "Mounting NFS shared folders..." stage indefinitely without any errors when attempting to run it, it could mean the default NFS version (3) used by Vagrant may be incompatible with your host system. It was noticed for instance on a Fedora 23 installation. The workaround is to edit the Vagrantfile and add "nfs_version: 4" there in the MOUNT_DIRS part, as follows:
MOUNT_DIRS.each { |k, v|
config.vm.synced_folder v[:repo], v[:local], create: true, nfs: true, nfs_version: 4
} |