Page Comparison

This page exists to communicate the design behind the minimal prototype of OEP-37: Test Dev Data.

Visualization of whole system

Expand
title Click to expand visualization

Prototype implementation

You can find implementation details in this expand. PR links also included in expand.

dev.load_data:
	python load_data.py ${data_spec_top_path}

#!/usr/bin/env python3

import yaml
import sys
import subprocess


def main(input_yaml_path):
    with open(input_yaml_path, 'r') as f:
        top_data_spec_yaml = yaml.safe_load(f)
        for data_spec_path in top_data_spec_yaml:
            ida_name = data_spec_path['ida_name']
            ida_data_spec_yaml = data_spec_path['data_spec_path']
            print(f"Creating test data in {ida_name} based on {ida_data_spec_yaml}")
            if ida_name == "lms" or ida_name == "cms":
                subprocess.run(f"docker-compose exec -T {ida_name} bash -c 'source /edx/app/edxapp/edxapp_env && cd /edx/app/edxapp/edx-platform/ && python manage.py {ida_name} load_data --data-file-path {ida_data_spec_yaml}'", shell=True)
            else:
                subprocess.call(f"docker-compose exec -T {ida_name} bash -c 'source /edx/app/{ida_name}/{ida_name}_env && cd /edx/app/{ida_name}/{ida_name}/ && python manage.py load_data --data-file-path {ida_data_spec_yaml}'", shell=True)


if __name__ == "__main__":
    if len(sys.argv) == 2:
        main(sys.argv[1])
    else:
        print("Path to data spec yaml not specified")

- ida_name: lms
  data_spec_path: openedx/core/djangoapps/util/management/commands/test_command.yaml
- ida_name: ecommerce
  data_spec_path: ecommerce/core/management/commands/test_data.yaml
- ida_name: lms
  data_spec_path: openedx/core/djangoapps/util/management/commands/test_command.yaml

users:
  - username: verified
    email: verified@example.com
  - username: robot1
    email: robot1@example.com
enrollments:
  - username: verified
    course_id: 'course-v1:edX+DemoX+Demo_Course'
    mode: verified
  - username: robot1
    course_id: 'course-v1:edX+DemoX+Demo_Course'
    mode: verified

#!/usr/bin/env python3
from django.core.management.base import BaseCommand
from django.db.utils import IntegrityError
from common.djangoapps.student.tests.factories import CourseEnrollmentFactory, UserFactory
from common.djangoapps.course_modes.models import CourseMode
from django.contrib.auth import get_user_model
import yaml
import codecs

from lms.djangoapps.verify_student.tests.factories import SoftwareSecurePhotoVerificationFactory

class Command(BaseCommand):
    """
    Use to populate your database with some essential basic data
    """

    def add_arguments(self, parser):
        parser.add_argument('--data-file-path', type=str, required=True, help="Path to file where your data is specified.")


    def handle(self, *args, **options):
        with open(options["data_file_path"], 'r') as f:
            data_spec = yaml.safe_load(f)
        for user_spec in data_spec.get('users', []):
            self.create_user(user_spec)
        for enrollment_spec in data_spec.get('enrollments', []):
            self.create_enrollment(enrollment_spec)

    def create_user(self, user):
        """
        Use to create users in your database.
        """
        return UserFactory.create(**user)

    def create_enrollment(self, enrollment_spec):
        """
        Use to create enrollments in your database.
        """
        User = get_user_model()
        try:
            user = User.objects.get(username=enrollment_spec['username'])
        except User.DoesNotExist:
            raise exception(f"User:{enrollment_spec['username']} not created before trying to create enrollment")
        if enrollment_spec['mode'] in CourseMode.VERIFIED_MODES:
            verfication = SoftwareSecurePhotoVerificationFactory(user=user)
        enrollment = CourseEnrollmentFactory(user=user, course_id=enrollment_spec['course_id'])
        if enrollment.mode != enrollment_spec['mode']:
            enrollment.mode = enrollment_spec['mode']
            enrollment.save()
        return enrollment

class UserFactory(DjangoModelFactory):  # lint-amnesty, pylint: disable=missing-class-docstring
    class Meta:
        model = User
        django_get_or_create = ('email', 'username')

    _DEFAULT_PASSWORD = 'test'

    username = factory.Sequence('robot{}'.format)
    email = factory.Sequence('robot+test+{}@edx.org'.format)

#.........
#stuff skipped
#.........

    @factory.post_generation
    def profile(obj, create, extracted, **kwargs):  # pylint: disable=unused-argument, missing-function-docstring
        if create:
            obj.save()
            return UserProfileFactory.create(user=obj, **kwargs)
        elif kwargs:
            raise Exception("Cannot build a user profile without saving the user")
        else:
            return None
#.........
#stuff skipped
#.........

How to use implementation:

• Checkout all the branches

  • edx-platform: msingh/oep37/mvp/userenrollments

  • ecommerce: diana/test-data-prototype

  • devstack: msingh/oep37/mvp/interface

• start a virtual env and run make requirements in devstack repo

• run make dev.load_data data_spec_top_path=test_data/data_spec_top.yaml from devstack repo

  • you should now have new data in your database

  • if something went wrong, here are individual commands to run in each of the container’s shells

    • lms shell: python manage.py lms load_data --data-file-path openedx/core/djangoapps/util/management/commands/test_command.yaml

    • ecommerce shell: python manage.py load_data --data-file-path ecommerce/core/management/commands/test_data.yaml

Design Decisions

• Data will be specified in multiple Yaml files

  • A top level yaml file will list other yaml files

  • The order by which the data is built in specific IDA will be specified in the top level yaml file

  • The data contained within the yaml files will be as minimal as possible

  • foreign keys will be linked via some unique identifier for lookup (i.e. course_key, domain, username)

• For each use of the load_test_data management command, there will be a separate yaml file with data specified.

• The management command will read the specified yaml file and pass on the data specification to the appropriate data generation function.

• We will be reusing existing factories used for unit tests

  • Benefits of reuse

    • Decrease in code duplication

    • There are ton of factories, so we’d be able to support ton of data creation very quickly

  • Downsides

    • There are ton of factories which were designed for different use cases. Due to the complex nature of some of these factories, it might be hard determine side effects on the database.

Open Design Decisions

• Where do all the yamls live?

• Should a creation function create everything necessary for a given datum? I think, yes. If a particular datum needs other things to set up, it should create it. This would align with how factory boy works

  • The concern here is the sort of ‘circular dependencies’ in data (i.e. lms creates courses, then ecommerce creates seats/modes in lms, then users can enroll as verified.)

• How do we shorten the dev cycle for this and to make it repeatable?

  • Option 1: Take snapshot of provisioned database and make it easier for people to go back to that state

    • We should do this for first pilot with aim of eventually enabling option 2.

  • Option 2: Assume this implementation method has replaced most of provisioning. Create ability to return to an empty database(that still has the data scheme, but none of the data).

• How do we want to pilot this?

  • Likely Option: request one team works with us extensively on this. Capture their use case

• Recommended decision: If we need to create data from outside of a factory, the code to create data should live in its own creation function.

• When the make command is used to create data, it calls a python script which calls the management command in each of the specified service’s container. This requires the user to have pyyaml installed. Should we make it a given that devstack commands should be run from a virtualenv. According to a very quick poll by Tim in devstack-questions, very few people use venvs in devstack.

• These scripts will not be idempotent. Modifications will be made to the database each time.

  • Do we want to limit the changes that these scripts make or do we want to allow them to make a full set of new entries each time?

  • How exactly do we version the creation functions?

• Names?

  •  OEP name: maybe from Test Data to Local Data

  •   management command: from load_data to load_dev_data

  •   yaml file keys: from ida_name to ida, from data_spec_path to path

  •   --data-file-path to --path

  • Name of the whole framework

  • What do we want to call this method of data loading to differentiate it from others in documentation?

• How opinionated do we want to be about what information you can specify about a particular datum? Example: for User model, should we limit it to unique fields: username, email? Or should developers be able to specify whatever they want(max flexibility)? The current implementation goes for max flexibility. But I imagine this might make it harder to do versioning later.

Possible Future Roadmap

Different stages

• Prototype for ARCH-BOM [in-progress]

• Develop MVP

  • tasks to do

    • Finalize and fully implement make interface

    • load_data management command in each of necessary IDA's

    • Add tests for the management command and data generation functions

    • minimum creation functions in each of the management commands

    • Documentation

      • Walkthrough of how to extend system yourself

      • Walkthrough of how to use system to create custom local data

      • Overarching system design

    • Recruit another Squad to work with us on this

  • Goal: have MVP ready to be used by another squad

• User Testing with another Squad

  • Tasks to do

    • Showcase this method to external Squad

      • This could be a synchronous meeting or an email/document

    • [maybe] Ideate with them as to how they would use this method

    • Have person be on stand-by to answer any questions by Squad and to handle any roadblocks

    • Continue improving both implementation and documentation based on feedback

  • Goal: Have method ready to be spread to whole org

• Advertise this tool to org

  • Tasks to be do

    • Further mature implementation and Documentation

    • Email, Slack post, eng-all-hands presentation

    • Have person on-call be ready to continue advertising and answering questions about this method

• Replace most of provisioning with this method of loading data

  • Requirements

    • TBD

...