Use AWS for Data Storage

Owned by Alison Hodges
Last updated: Sept 12, 2016 by Tim KronesVersion comment

By default, the Open edX platform uses Amazon S3 (Simple Storage Service) for data storage. If you need to reconfigure an instance of Open edX to use S3, you update the /edx/app/edx_ansible/server-vars.yml file.

Before proceeding, review the Guidelines for Updating the Open edX Platform.

Preparing to Configure Amazon S3 for Data Storage

Before you change your data storage configuration, be sure to determine the values that apply to your S3 data storage service, including the name of your bucket.

If you want to ensure that the data you store on S3 is secure, and you plan to use signed URLs that are valid for only a short time, be sure to set querystring_auth: true and default_acl: 'private', as shown in the topic that follows. In addition, be sure to verify the following third-party configuration settings.

  • Files in S3 should be private.
  • The signed URLs should have temporary authentication that allows private files to be accessed.

Use Amazon S3 for Data Storage

To configure S3 data storage, follow these steps.

1. Stop the LMS server.

2. Create or edit the file to define the following settings.

Be sure to replace the example values for the settings that follow with values that apply to your instance.

COMMON_OBJECT_STORE_LOG_SYNC: true

EDXAPP_SETTINGS: "aws"

VHOST_NAME: "aws"

EDXAPP_FILE_UPLOAD_STORAGE_BUCKET_NAME: "bucket_name"

...

EDXAPP_GRADE_STORAGE_CLASS: 'storages.backends.s3boto.S3BotoStorage'

EDXAPP_GRADE_STORAGE_KWARGS:

  bucket: 'example-student-grades'

  custom_domain: null

  locations:  '{{ COMMON_ENVIRONMENT }}-{{ COMMON_DEPLOYMENT }}'

  querystring_expire: 300

  gzip: true

  querystring_auth: true

  default_acl: 'private'

...

EDXAPP_VERIFY_STUDENT:

DAYS_GOOD_FOR: 365

SOFTWARE_SECURE:

STORAGE_CLASS: "storages.example.s3.S3Storage"

STORAGE_KWARGS:

bucket: "example-certs-test"

custom_domain: null

querystring_expire: 432000

querystring_auth: true

default_acl: 'private'

...

ENABLE_GRADE_DOWNLOADS: true

XQUEUE_SETTINGS: "aws_settings"

XQUEUE_UPLOAD_BUCKET: "xqueue_bucket_name"

XQUEUE_UPLOAD_PATH_PREFIX: "xqueue_path_prefix"

If you were previously using Swift, your /edx/app/edx_ansible/server-vars.yml file might contain a number of Swift-related variables. (All of these variables have SWIFT in their name. See Use Swift for Data Storage for details.) They should not interfere with the AWS settings above, so it's not strictly necessary to remove them from /edx/app/edx_ansible/server-vars.yml before running the commands listed below. But to be on the safe side, it is still recommended to remove them.

3. Run the /edx/bin/update command.

sudo /edx/bin/update edx-platform <your-branch-name>

sudo /edx/bin/update xqueue <your-branch-name>

4. Restart the LMS server.

5. Repeat this procedure for each instance that has data that you want to store on Amazon S3.