What should the code structure be of the edx-platform? 

Do we agree that the current state of tangled inter-dependencies is hard to maintain, comprehend, extend, and test?

Goals

Code Structure

• Clearly named, single-purpose, apps and folders, with clearly defined interfaces.
  • So they are easier to maintain, test, and comprehend.
  • So they can be substituted, if needed.
  • Easier to transition to a separate repo, if we choose to do so.
  • Easier to transition to a separately deployed process, if we choose to do so.
• Dependencies
  • Layered:
    
    • Low-level apps/modules do not depend on higher layers.
    • Higher layers can call exposed APIs on lower layers.
    • It's Ok for higher layers to call lower layers that are not immediately below them (i.e., skipping layers is Ok for now).
  • Loosely coupled:
    
    • Where possible, use django signals to notify peer layers.
    • Use clear API abstractions defined in api.py or via REST APIs
      • which include the exposed public interfaces
      • and the expected integration points for those modules
  • Separate CMS/LMS: No code dependencies between LMS and CMS. Their lower layer common folders should not call back up into LMS/CMS.
• Assets
  
  • Assets that belong to an app should be co-located within the app, this includes:
    
    • Mako Templates
    • Underscore Templates
    • App-specific Images
  • For any assets shared across apps, they can go into
    
    • UI toolkit - Shared Sass
    • Pattern Library - Shared JS
    • common folder - Shared Images
• Models
  
  • Need to consider them on a case-by-case basis.
  • By using foreign keys, it tightly couples the relationship - and requires the apps deploy within the same container.
• URLs
  
  • URLs exposed by an app should be co-located within the app.
  • The top-level urls.py should then include the urls from each app's urls (with specified namespaces).
• Interfaces
  • api.py should include all public in-process python interfaces
  • urls.py should include all public REST APIs
• Tests (exact details TBD - here's a proposal)
  • Unit tests 
    • Definition
      • Do not have dependencies on peer or higher layers.  
      • May either depend on or mock lower layers.
      • White-box tests of all edge conditions and paths.
    • Guidelines
      • Can use/test internal methods of the module.
      • Use override_settings instead of requiring settings in a central envs/tests.py.
  • Integration tests 
    • Definition  (Note: this differs from testing.rst.)
      • End-to-end python integration tests that test dependencies between apps.
      • Inter-dependencies are not mocked.
      • May test multiple paths, including non-happy paths if impacted by inter-dependencies.
    • Guidelines
      
      • Stored separately (maybe centrally).
      • Only use exposed public interfaces (see Interfaces above)
  • Acceptance tests
    • Definition
      • End-to-end UI-level black-box tests.
      • Mostly only happy paths are tested.  Regularly occurring exceptional paths that are integral to the user-facing feature may also be tested.
      • All scenarios described in user-documentation are typically covered.

Enforcement

• Have edx-linter automatically find exceptions to prevent regressions to the agreed rules.

Non-Goals

• Don't over-package.

Long-term Goals

• edx.org project is separate from the open-edx platform.
• Use decorators on views to conditionally enable them instead of using django settings in urls.py.
• Use course-scoped xblock settings (once implemented) instead of adding app-specific fields in course_module.py.
• Have unit tests in openedx/core run on their own, outside of the context of LMS/CMS.

Short-term Goals

Open Questions

• How do we support composability using Require.js?
  • For now, just keep as is - where we configure all js files in one location.
• Karma cannot discover decentralized tests.
• Can each app locally declare its own django configuration settings so they are part of its public interface?
  • They can then be defined/set centrally for the entire project, as needed.

Initial Cleanup Tasks

Here's a list of directories to address in common:

Common Directory	Who's working on this?	Status
djangoapps/auth_exchange	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13658
djangoapps/cache_toolbox	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13673
djangoapps/config_model	Julia Eskew (Deactivated)	I've split config_model into a separate repo: https://github.com/edx/django-config-models Both: https://github.com/edx/django-splash and https://github.com/edx/edx-platform currently depend on the app. edx-platform: https://github.com/edx/edx-platform/pull/13708
djangoapps/contentserver	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13699
djangoapps/cors_csrf	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13698
djangoapps/course_action
djangoapps/course_mode
djangoapps/dark_lang	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13677
djangoapps/datadog	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13688
djangoapps/django_comment_common
djangoapps/edxmako	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13683 https://github.com/edx/configuration/pull/3407
djangoapps/embargo	Julia Eskew (Deactivated)	https://github.com/edx/edx-platform/pull/13706
djangoapps/enrollment
djangoapps/external_auth	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13659
djangoapps/geoinfo	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13693
djangoapps/header_control	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13694
djangoapps/heartbeat	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13690
djangoapps/lang_pref	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13677
djangoapps/microsite_configuration
djangoapps/monitoring	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13692
djangoapps/monkey_patch	Julia Eskew (Deactivated)	https://github.com/edx/edx-platform/pull/13667
djangoapps/performance	Nimisha Asthagiri (Deactivated)	https://github.com/edx/edx-platform/pull/13691
djangoapps/pipeline_js	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13664
djangoapps/pipeline_mako	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13665
djangoapps/request_cache
djangoapps/reverification	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13668
djangoapps/service_status	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13675
djangoapps/session_inactivity_timeout	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13700
djangoapps/startup_configurations
djangoapps/static_replace	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13665
djangoapps/status
djangoapps/student
djangoapps/terrain
djangoapps/third_party_auth
djangoapps/track		Blocked until this merges: https://github.com/edx/edx-platform/pull/13682
djangoapps/util
djangoapps/xblock_django		Too many cross dependencies to move right now
djangoapps/xmodule_django	Julia Eskew (Deactivated)	https://github.com/edx/edx-platform/pull/13671
lib/calc	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13707
lib/capa		Don't move as ChristinaR (Deactivated) is migrating the CoffeeScript
lib/chem	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13707
lib/dogstats	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13707
lib/i18n_tests	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13674
lib/safe_lxml	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13707
lib/sandbox-packages	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13707
lib/symmath	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13707
lib/xmodule		Too many cross dependencies to move right now
test/acceptance	Andy Armstrong (Deactivated)	https://github.com/edx/edx-platform/pull/13689

To see all the open PRs in GitHub for the Hackemon Hackathon project, look in the "clean_platform" GitHub milestone:

https://github.com/edx/edx-platform/milestone/1

Important Hackathon Points

• edx-platform/common should no longer be used, in favor of openedx/core.
• Don't move any code to openedx/core that depends on code in lms or cms!
  • Some dependencies have snuck in to openedx/core and common.
  • We want openedx/core to be a place to put lower-level code.
    • Modules in openedx/core can depend on each other (and common for now) - but never lms/cms.
• In some cases, it's appropriate to move Django applications completely outside of edx-platform.
  • The config_models Django application was moved out of edx-platform for the hackathon.
  • edx-platform and django-splash (and maybe others) were dependent on the app.
• Avoid gigantic urls.py files!
  • Keep the url patterns for particular Django applications with the actual Django application.
  • Include those urls into the lms/cms urls.py files.
• Django signals can be used to de-couple code and remove dependencies.
  • Instead of direct calls into higher-level code, lower-level code can emit a signal which is caught by higher-level code.
• GitHub milestones are great!
  • Consider using milestones for informal PR grouping.

Current issues in common folder

CMS

cms.lib.xblock.tagging:StructuredTagsAside

LMS

lms.djangoapps.oauth_dispatch

lms.djangoapps.verify_student.models

VerificationDeadline

lms.djangoapps.commerce.utils

EcommerceService

lms.djangoapps.commerce.utils

audit_log

lms.djangoapps.verify_student.models

VerificationDeadline

SoftwareSecurePhotoVerification

lms.djangoapps.badges.utils

badges_enabled

lms.lib.comment_client

User.from_django_user

lms.djangoapps.badges.events.course_meta

award_enrollment_badge

lms.djangoapps.commerce.utils

EcommerceService

lms.djangoapps.verify_student.models

SoftwareSecurePhotoVerification

lms.djangoapps.courseware

lms.djangoapps.certificates.models

lms.djangoapps.grades

courses.get_course_by_id

GeneratedCertificate

course_grades.summary

lms.lib.comment_client

lms.djangoapps.ccx.modulestore

CCXModulestoreWrapper

Current issues in openedx/core folder

CMS

cms.djangoapps.contentstore.courseware_index

LMS

unquote_slashes

lms.djangoapps.courseware.courses

get_course_by_id

lms.djangoapps.instructor.access

list_with_level

lms.djangoapps.django_comment_client.utils

is_discussion_enabled

lms.djangoapps.certificates.api

get_active_web_certificate

lms.djangoapps.ccx.utils

get_ccx_from_ccx_locator

lms.djangoapps.django_comment_client.utils

get_discussion_category_map,

get_discussion_categories_ids

lms.djangoapps.verify_student.models

SkippedReverification,

VerificationStatus

lms.djangoapps.django_comment_client.utils

JsonResponse

lms.djangoapps.certificates

lms.djangoapps.commerce.utils

lms.djangoapps.badges.utils

badges_enabled

lms.djangoapps.courseware.courses

get_course_with_access

CoursewareAccessException

lms.djangoapps.courseware.access

_has_access_to_course

lib