Support running pipeline acceptance tests locally

Description

1. modify make targets and automation to make it easy to run them
2. add documentation explaining how to run these
3. tag (vertica/exporter/other) tests as optional
4. modify tests so that they can work with either S3 or HDFS

Steps to Reproduce

None

Current Behavior

None

Expected Behavior

None

Reason for Variance

None

Release Notes

None

User Impact Summary

None

Activity

Show:
Gabe Mulley
December 23, 2015, 2:20 PM
Edited

Here is the current procedure:

1) rebase the branch on master of edx/edx-analytics-pipeline (this allows you to run acceptance tests on devstack)
2) create a file called "/var/tmp/acceptance.json" in your analytics devstack with the following content:

Note that I've put the two most important keys at the top of this JSON blob, they must match the repo where the branch lives and the branch that contains the tests you want to run.

3) As the vagrant user, navigate to "/var/lib/analytics-tasks/devstack/repo" and checkout the branch you want to test
4) Run "source ../venv/bin/activate"
5) Run "make uninstall; python setup.py develop; python setup.py install_data; make test-requirements"
6) Run "sudo mysql" at the prompt run this query: GRANT ALL PRIVILEGES ON `acceptance%` . * TO 'pipeline001'@'localhost'
7) Switch to the hadoop user and source "../venv/bin/activate"
8) Run the following command:

Note that it a path to a particular test can optionally be appended to this command, such as "edx.analytics.tasks.tests.acceptance.test_student_engagementerStudentEngagementAcceptanceTest"

8) Make changes to your branch, push to github, repeat step 6. Note that if you change the acceptance test setup code you will also need to update the branch checked out into "/var/lib/analytics-tasks/devstack/repo". Continue doing this until the test passes.

Done
Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Unassigned

Reporter

Gabe Mulley

Labels