Native installs

Make sure everything is running. Services might be listed as something other than RUNNING:

vagrant@vagrant:~$ sudo /edx/bin/supervisorctl status
analytics_api                    RUNNING   pid 2050, uptime 0:15:35
certs                            RUNNING   pid 8656, uptime 0:00:02
ecommerce                        RUNNING   pid 2084, uptime 0:15:35
ecomworker                       RUNNING   pid 2045, uptime 0:15:35
edxapp:cms                       RUNNING   pid 2083, uptime 0:15:35
edxapp:lms                       RUNNING   pid 2079, uptime 0:15:35
edxapp_worker:cms_default_1      RUNNING   pid 2099, uptime 0:15:34
edxapp_worker:cms_high_1         RUNNING   pid 2095, uptime 0:15:34
edxapp_worker:cms_low_1          RUNNING   pid 2086, uptime 0:15:34
edxapp_worker:lms_default_1      RUNNING   pid 2096, uptime 0:15:34
edxapp_worker:lms_high_1         RUNNING   pid 2101, uptime 0:15:34
edxapp_worker:lms_high_mem_1     RUNNING   pid 2090, uptime 0:15:34
edxapp_worker:lms_low_1          RUNNING   pid 2100, uptime 0:15:34
forum                            RUNNING   pid 2046, uptime 0:15:35
insights                         RUNNING   pid 2104, uptime 0:15:34
notifier-celery-workers          RUNNING   pid 2049, uptime 0:15:35
notifier-scheduler               RUNNING   pid 2059, uptime 0:15:35
programs                         RUNNING   pid 2048, uptime 0:15:35
xqueue                           RUNNING   pid 2065, uptime 0:15:35
xqueue_consumer                  RUNNING   pid 2044, uptime 0:15:35
vagrant@vagrant:~$

E-Commerce Test

Make sure FEATURES.ENABLE_OAUTH2_PROVIDER is set to true in /edx/app/edxapp/lms.env.json
Follow the Configure the LMS section of Test Your E-Commerce Application steps 1–9, see setting changes below. Skip the step for starting the app server, since it's already started.
Follow the Configure E-Commerce section of Test Your E-Commerce Application steps 1–8, see setting changes below. Instead of starting the app server, reboot the server before trying to access E-Commerce so the settings can take effect.

Configuration Differences

For configuring E-Commerce to talk to the LMS, you'll need to use a fully qualified domain name or IP address assigned to your machine. In this case, assuming we're using a machine with the IP address of 192.168.33.10.

/edx/app/edxapp/lms.env.json

"ECOMMERCE_API_URL": "http://192.168.33.10:18130/api/v2"
"ECOMMERCE_PUBLIC_URL_ROOT": "http://192.168.33.10:18130"
"JWT_ISSUER": "http://192.168.33.10/oauth2"
"OAUTH_OIDC_ISSUER": "http://192.168.33.10/oauth2"

LMS Django Admin Oauth2 Client

URL: http://192.168.33.10:18130
Redirect URI: http://192.168.33.10:18130/complete/edx-oidc/
Logout URL: http://192.168.33.10:18130/logout/

/edx/etc/ecommerce.yml

JWT_ISSUERS:
  - http://192.168.33.10/oauth2

When Creating A New Site Linking to the LMS

Be sure to use --site-domain=192.168.33.10:18130 and --lms-url-root=http://192.168.33.10.

Open edX Developer Stack ("Devstack") Release Test Plan

Smoke tests to ensure a new devstack release is functioning properly.

LMS

Make sure you can start LMS.

CMS

Make sure you can start CMS.

Create a small course, make sure you can see it in the LMS.

Make sure you can preview an unpublished course unit.

Forum

Make sure you can start Forum.

In the LMS, start a discussion thread, and add a reply.

edx-platform Unit Tests

For devstack:

$ sudo su edxapp
$ paver test

edx-platform bok-choy Tests

Set up the bok-choy test server. This will take about 15 minutes.
paver test_bokchoy -r
-r stands for --serversonly.
In another terminal window, run tests. The test modules are relative to common/test/acceptance/tests/.
paver test_bokchoy -a -o -t video/test_video_events.py
paver test_bokchoy -a -o -t video/test_studio_video_editor.py
paver test_bokchoy -a -o -t lms/test_lms_cohorted_courseware_search.py
-a stands for --fasttest and -o stands for --testsonly. Both options must be set while in client mode.
In another terminal window, run tests. The test modules are relative to common/test/acceptance/tests/.
paver test_bokchoy -a -o -t lms/test_problem_types.py
There are known failures for this test suite currently.
Make sure X11 forwarding work. You must be on the host machine (physically or via Screen Sharing). X11 forwarding doesn't work if you're SSHed into the host machine.
1. Install XQuartz
brew cask install xquartz
2. Let Vagrant know you want to forward X11.
export VAGRANT_X11=1
3. Start the guest box and SSH into it.
vagrant up && vagrant ssh
4. Verify X11 forwarding is working in the guest VM.
$ echo $DISPLAY
localhost:10.0
You can also start Firefox, for example, and it should show up on your desktop via XQuartz.
firefox

E-Commerce Test

Follow the Configure the LMS section of Test Your E-Commerce Application steps 1–9.
Follow the Configure E-Commerce section of Test Your E-Commerce Application steps 1–8.

We still need to have smoke tests for the following:

~~ecommerce~~
ecomworker
programs
notifier

Once a release is created, it needs to be tested

Installation: each installation method must complete successfully
Automated tests
- For devstack: run "paver test"
Manual tests
- Create a sandbox installation of Native, and have the manual testers test it.

For the future

Consider the end-to-end tests (github.com/edx/edx-e2e-tests). They are probably not going to make it in for Ficus, but they are getting more repeatable.

OpenCraft release testing procedures

OpenCraft uses the OpenCraft Instance Manager to test deployments and upgrades on OpenStack.

Overall procedure:

Review the release notes, configuration changelog, and automatic upgrade scripts and determine if any configuration variables need to be updated etc.
Deploy a new single-VM instance from scratch using the new release candidate and validate it (see below).
Deploy a new multi-VM instance from scratch using the new release candidate and validate it (see below).
Deploy/use a single-VM instance from the previous release, then change its branch to use the new release candidate, deploy the upgrade, and then validate it
Deploy/use a multi-VM instance from the previous release, then change its branch to use the new release candidate, deploy the upgrade, and then validate it

Validation procedure for each deployment:

This manual testing procedure is designed to cover some of the more common failure modes / deployment errors encountered:

Make sure user registration works.
1. Make sure you received the activation email. If using an external SMTP server (e.g. SES), verify that the email was actually sent via that external service and not directly from the EC2 instance.
2. Make sure that the activation link in the email works.
Make sure course creation works.
1. Log into the studio and create a new course.
2. Add a new unit to the course.
3. Make sure the "Preview" link works correctly.
Test course in the LMS.
1. Log into the LMS as a staff user.
2. Click through each of the tabs ("Home", "Course", "Discussion", ...) to make sure the section loads correctly.
3. Go to the "Course" tab and make sure the "View unit in Studio" link works correctly.
4. Go to the "Discussion" tab and submit a new post, and add an image. Make sure the image is uploaded to S3/ObjectStorage successfully, and is displayed in the post correctly.
5. Go to the "Instructor" tab and select "Data Downloads" section. Click the "Download profile information as a CSV" button and wait until the report is generated. Make sure the link to the report works correctly.
Make sure tracking logs are being synced to S3 (We always want to sync tracking logs even if the instance is not currently using analytics. Note that tracking logs don't rotate until they're larger than 1Mb.)
If using Insights: make sure link from "Instructor" tab points to Insights correctly and make sure that OAuth between LMS and Insights works.
If using a custom theme:
1. Make sure the theme looks ok in general.
2. Pay special attention to discussion forums and the wiki (they are using different layouts).