Support running pipeline acceptance tests locally
1. modify make targets and automation to make it easy to run them
2. add documentation explaining how to run these
3. tag (vertica/exporter/other) tests as optional
4. modify tests so that they can work with either S3 or HDFS
Steps to Reproduce
Reason for Variance
User Impact Summary
Here is the current procedure:
1) rebase the branch on master of edx/edx-analytics-pipeline (this allows you to run acceptance tests on devstack)
2) create a file called "/var/tmp/acceptance.json" in your analytics devstack with the following content:
Note that I've put the two most important keys at the top of this JSON blob, they must match the repo where the branch lives and the branch that contains the tests you want to run.
3) As the vagrant user, navigate to "/var/lib/analytics-tasks/devstack/repo" and checkout the branch you want to test
4) Run "source ../venv/bin/activate"
5) Run "make uninstall; python setup.py develop; python setup.py install_data; make test-requirements"
6) Run "sudo mysql" at the prompt run this query: GRANT ALL PRIVILEGES ON `acceptance%` . * TO 'pipeline001'@'localhost'
7) Switch to the hadoop user and source "../venv/bin/activate"
8) Run the following command:
Note that it a path to a particular test can optionally be appended to this command, such as "edx.analytics.tasks.tests.acceptance.test_student_engagementerStudentEngagementAcceptanceTest"
8) Make changes to your branch, push to github, repeat step 6. Note that if you change the acceptance test setup code you will also need to update the branch checked out into "/var/lib/analytics-tasks/devstack/repo". Continue doing this until the test passes.