What does the exported file format look like?TODO

Answer: Didn’t get a chance to dive into this.

What happens to V2-library-referenced content when a course is exported and imported?

...

Decreasing library centralization ➡

Decreasing size of course exports ⬇

(a) Library only exists on original instance, so course-library relationship is severed when exporting to other instances.

(b) Library versions are centrally managed by original instance, but replicated on other instances.

(c) Library versions managed in a distributed fashion across instances, identifying versions by git-like hashes.

(1) Entire library is bundled with the course export.

🥡 = ( 🎓 + 📚 )

(1a) nonsensical

(1b) The original library is brought to the new instance along with the course, although the library does not become part of the instance’s list of libraries available for authoring. Instead, the library maintains a connection with the original instance, and can receive updates from the library on that instance. In this sense, the library copy is a replicate of the “centrally” managed library.

Pro: Libraries can be used across instances, while still maintaining simplicity of central management.

Con: Library bundling increases export size.

Challenge: Would need to figure out how to stop remote library keys (from original instance) from clashing with local library keys (on new instance).

Challenge: We’d need to figure out the replication strategy. Push-by-original-instance? Or pull-by-new-instance? And, how we keep track of the URL of the original instance?

(1c)

Specific version(s) of each library can be chosen for export. Multiple versions of the same library can be imported into an Open edX instance without disrupting one another, and importing a library at a version does not necessarily mark it as the “newest”. Library versions are uniquely identified and referenced by a hash of their contents, avoiding conflicts that may arise between human-friendly version names/numbers across instances.

If the required version of the required library exists on an Open edX instance, then the course automatically uses it. Otherwise, library blocks in the course gracefully degrade to rendering hidden blocks.

Example story: https://lucid.app/lucidchart/4950383b-a703-4bd8-b08e-f94a6f144c1a/edit?invitationId=inv_73bdba67-80b6-45d3-9c7b-bf2e5d8e0ac3&page=Mm5j-6d_dnBg# (in the “Import/Export Story” tab, start at the very top and read downwards.

(2) Necessary pieces of library are bundled with course export.

🥡 = ( 🎓 + 📘 )

🥡 = 📚

(2a) Essentially, library blocks turn into ordinary blocks when a course is exported.

Pro: Courses will work out-of-the-box when exported and imported. We avoid wrestling with cross-instance library versioning altogether.

Con: Use of a library is confined to a single instance.

Challenge: Would need to develop a non-library version of randomized content banks.

(2b) Same pros and cons of 2b, except:

remove Con: Library bundling increases export size.
new Challenge: Including just the necessary library blocks in the course export.

(2c) Same pros and cons of 1c, except:

remove Con: Library bundling increases export size.
new Challenge: Including just the necessary library blocks in the course export.

(3) Library is excluded from course export.

🥡 = 🎓

🥡 = 📚

(3a) nonsensical

(3b) Same pros and cons of 1b, except:

remove Con: Library bundling increases export size.
New Con: Course doesn’t work out-of-the box until first replication-from-original-instance occurs.

(3c) Same pros and cons of 1c, except:

remove Con: Library bundling increases export size.
New Con: Course doesn’t work out-of-the box. User must import correct library version into instance.

Ideas: I have a couple competing ideas for how import/export could work…

Approach 1: Distributed version control

This is the first approach that came to my mind, after thinking about how to preserve course-library relationships across instances while avoiding library version conflicts from arising.

Library blocks don’t export with a course’s OLX. However, library content customizations do export with the OLX (since they are saved on the LibraryReferenceBlock).
Libraries can be exported. Notably, specific version(s) of each library can be chosen for export. Multiple versions of the same library can be imported into an Open edX instance without disrupting one another, and importing a library at a version does not necessarily mark it as the “newest”. Library versions are uniquely identified and referenced by a hash of their contents, avoiding conflicts that may arise between human-friendly version names/numbers across instances.
If the required version of the required library exists on an Open edX instance, then the course automatically uses it. Otherwise, library blocks in the course gracefully degrade to rendering hidden blocks.

Example story: https://lucid.app/lucidchart/4950383b-a703-4bd8-b08e-f94a6f144c1a/edit?invitationId=inv_73bdba67-80b6-45d3-9c7b-bf2e5d8e0ac3&page=Mm5j-6d_dnBg# (in the “Import/Export Story” tab, start at the very top and read downwards.

Approach 2: Centralized version control

Dave Ormsbee , in reaction to Approach 1:

It might be useful to export [library blocks], even if it doesn’t import back on the other side. There are a number of tools that do content analysis of exports, and if you’re using content from multiple libraries, it’d be convenient to have that in the same export file.
….
I’m leery of making that step [(exporting at any version)] into full blown distributed version control. A question: Can we get away with only having one primary source of a library, and allow instances to essentially make read-replicas?
So for instance, a particular library could exist on edx.org and have some set of users who can modify it, and a simple linear history. But we can configure something so that Edge keeps an updated read-replica, so that it should theoretically always have that library’s data available.
This could be low level functionality that gets built at the blockstore layer as well. Some kind of namespaced replication so that Edge’s blockstore still has edx.org / {library_uuid} . Right now, the OLX export for a library assumes that the library belongs on whatever instance the course is in, but we could explicitly reference a fully qualified domain.

Essentially, instead of relying on library content hashes to uniquely identify libraries across instances, we’d somehow annotate and/or namespace libraries with their origin Open edX instance. Furthermore, libraries would be bundled with course exports.

TODO: write more

Approach 3: Insanity

TODO write more

Decision

Urgency: High

Answer: TBD

What does an OLX export with V2 library content look like?

Potential augmentation of the current course OLX export:

...

breakoutMode	wide

...

Decision

Urgency: High

Answer: Approach 1b.

Some additional decisions & implications:

To support this approach, library keys ought to be namespaced by “domain” (a string identifying the Open edX instance, if not the literal domain of the instance). For example, instead of library keys like lib:MITx:Problems1, you’d have lib:edx.org:MITx:Problems1 and lib:labxchange.org:LabXChange:lib-abcdef123.
Course authors in an Open edX instance ought to be able to “subscribe” to libraries on other instances, an action that would require an “access key” for non-public libraries. For example, if a hypothetical user of studio.labxchange.org wanted to use the library lib:edx.org:HarvardX:ScienceLib, they would need to request an access token from a ScienceLib staff user, and enter it into LabXChange’s Studio instance in order to make the external library available to LX authors.

What does an OLX export with V2 library content look like?

Urgency: High

Answer, Iteration 1:

Code Block

breakoutMode	wide

./                         # Root directory of an exported .tar.gz for a course.
  course/                  #   As before, the ./course/ folder contains:
    assets/                #     (1) an assets folder,
    chapter/               #     (2) one folder for
    .../                   #         each
    vertical/              #         block
    video/                 #         type, and
    course.xml             #     (3) the root course.xml file.
  libraries/               #   New folder! Contains all referenced librari1es, one XML file per library.
    Greendale_VideoLib.xml #     Given a library with the key `lib:ORG:LIBSLUG`, filename is `ORG_LIBSLUG.xml`.
    Greendale_ChemLib.xml  #     <- For example, this defines 'lib:Greendale:ChemLib'.
    CityClg_mathlib.xml    #     XML files contain just metadata, including bundle UUIDs, but no block content.
  bundles/                 #   New folder! Contains definitions for bundles that were referenced in ./libraries/.
    a578e5...5b181d/       #     In the blockstore-course future, ./course/ would also refernce ./bundles/.
    259fa0...71f87f/       #     The idea is that all content lives in ./bundles/ and other folders are metadata.
    d8b24f...2640c8/       #     Bundle folder names are the full 32-char, lowercase, un-hyphenated bundle UUID.
      bundle.xml           #       bundle.xml contains title, slug, version #, and snapshot hash.
      files/               #       Finally, files/ contains the OLX serialization of the bundle contents.
        problem/           #
        .../               #
        video/             #

Dave pointed out that the structure o the bundles/ directory would not be very human-friendly, even if it reflects the blockstore data model.

Answer, Iteration 2:

Code Block

breakoutMode	wide

./                         # Root directory of an exported .tar.gz for a course.
  course/                  #   As before, the ./course/ folder contains:
    assets/                #     (1) an assets folder,
    chapter/               #     (2) one folder for
    ...                    #         each
    vertical/              #         block
    video/                 #         type, and
    course.xml             #     (3) the root course.xml file.
  libraries/               #   New folder! Contains all referenced libraries and their content.
    edx.org/               #     Folder is structured by domain,
       Greendale/          #       then organization,
         VideoLib/         #         then library slug,
           v3/             #           and finally library version, mirroring the library key structure.
             ...           #
  Root  directory of an exported .tar.gz for a course. ChemLib/     course/     #         Each library contain an #OLX serialization of Asits beforecontents,
the ./course/ folder contains:     assets/   v2/             #
    (1) an assets folder,     chapter/ meta.xml      #        #   along with (2) one folder for
    .../  a "meta.xml" file to hold versioning #, snapshot hash, et al.
             problem/   #   #
     each     vertical/   ...           #
        block     video/        #
       CityCollege #        #
type, and     course.xml   mathlib/          #
    (3) the root course.xml file.   librariesv12/            #
     #   New folder! Contains all referenced librari1es,... one XML file per library.     Greendale_VideoLib.xml #
     Given   a library with thev13/ key `lib:ORG:LIBSLUG`, filename is `ORG_LIBSLUG.xml`.     Greendale_ChemLib.xml  #
     <-  For example, this defines 'lib:Greendale:ChemLib'.  ...   CityClg_mathlib.xml    #    #
XML files contain just metadata, including bundle UUIDs, but no block content.
  bundles/ labxchange.org/        #
      ...            #   New folder! Contains definitions for bundles that were referenced in ./libraries/.
    a578e5...5b181d/       #     In the blockstore-course future, ./course/ would also refernce ./bundles/.
    259fa0...71f87f/       #     The idea is that all content lives in ./bundles/ and other folders are metadata.
    d8b24f...2640c8/       #     Bundle folder names are the full 32-char, lowercase, un-hyphenated bundle UUID.
      bundle.xml           #       bundle.xml contains title, slug, version #, and snapshot hash.
      files/               #       Finally, files/ contains the OLX serialization of the bundle contents.
        problem/           #
        .../               #
        video/             #

Urgency: High

Answer: TBD

Referencing V2 library content from V2 courses

V2 (blockstore-backed) courses don’t exist yet, so we don’t need to worry about implementing this as part of BD-14. Rather, these questions are here so that we keep in mind the eventual migration to V2 courses, avoiding decisions during BD-14 that would complicate that process.

How might V2 library content referencing differ in V2 courses (as opposed to V1 courses)?

TODO

Archive

Misc notes and stuff

Expand

title	a couple competing ideas for how import/export could work…

Approach 1: Distributed version control

This is the first approach that came to my mind, after thinking about how to preserve course-library relationships across instances while avoiding library version conflicts from arising.

Library blocks don’t export with a course’s OLX. However, library content customizations do export with the OLX (since they are saved on the LibraryReferenceBlock).
Libraries can be exported. Notably, specific version(s) of each library can be chosen for export. Multiple versions of the same library can be imported into an Open edX instance without disrupting one another, and importing a library at a version does not necessarily mark it as the “newest”. Library versions are uniquely identified and referenced by a hash of their contents, avoiding conflicts that may arise between human-friendly version names/numbers across instances.
If the required version of the required library exists on an Open edX instance, then the course automatically uses it. Otherwise, library blocks in the course gracefully degrade to rendering hidden blocks.

Example story: https://lucid.app/lucidchart/4950383b-a703-4bd8-b08e-f94a6f144c1a/edit?invitationId=inv_73bdba67-80b6-45d3-9c7b-bf2e5d8e0ac3&page=Mm5j-6d_dnBg# (in the “Import/Export Story” tab, start at the very top and read downwards.

Approach 2: Centralized version control

Dave Ormsbee , in reaction to Approach 1:

It might be useful to export [library blocks], even if it doesn’t import back on the other side. There are a number of tools that do content analysis of exports, and if you’re using content from multiple libraries, it’d be convenient to have that in the same export file.
….
I’m leery of making that step [(exporting at any version)] into full blown distributed version control. A question: Can we get away with only having one primary source of a library, and allow instances to essentially make read-replicas?
So for instance, a particular library could exist on edx.org and have some set of users who can modify it, and a simple linear history. But we can configure something so that Edge keeps an updated read-replica, so that it should theoretically always have that library’s data available.
This could be low level functionality that gets built at the blockstore layer as well. Some kind of namespaced replication so that Edge’s blockstore still has edx.org / {library_uuid} . Right now, the OLX export for a library assumes that the library belongs on whatever instance the course is in, but we could explicitly reference a fully qualified domain.

Essentially, instead of relying on library content hashes to uniquely identify libraries across instances, we’d somehow annotate and/or namespace libraries with their origin Open edX instance. Furthermore, libraries would be bundled with course exports.

TODO: write more

Expand

title	Misc notes from convo w/ jennifer et al on how v1 libraries are used today, specifically in reference to library content customization

jennifer’s opinion
- content should be edited in lib
- settings should be edited in course
- distinction is sometimes fuzzy–e.g. content is sometimes corrected within the context of a course today (minor changes to text)
fields that are ambiguous content v setting?
- display_name?
  - renaming after import isn’t usually done in v1 libraries
  - confusion around problem titles coming from libraries sometimes
    - Names in libraries often have different meaning: e.g. 1a, 1b, 1c. vs. what it would be in the course (“Problem 1: {helpful description}”).
- video start time
  - not used much currently?
  - would need to verify with CourseGraph
in units with library content, is there other stuff?
- often a preamble HTML followed by random problems
- mixture of HTML+video+problems in library very uncommon, since specific-block selection has been broken for a while
- why: a single LTI launch for the entire unit would be cool
  - save ourselves an iframe layer
stress test
- 30 blocks pulled in from a library of 50 in a single Unit
  - this is because it’s the only way right now to make sure that content doesn’t repeat.
    - interesting data model implications. Have a saved state for this inclusion of this library, but allow it to be launched across multiple pages?
LMS interactions
- Can one LTI-launched thing map to multiple scores? This would be nice for grading purposes, but may be hard to retrofit.
  - Reactive use case: Need to correct and re-score one problem from the possible selection of the library.
  - Grades information.
  - Deep linking here?
- Does it make sense for there to be two inclusion modes: one with single and one for a whole unit?
- Settings tend to be re-used across a whole subsection, or even type “e.g. when using it as a HW, always give them five attempts”
  - Rare case: We find that one problem is too hard and adjust it to allow for more attempts.

...

Versions Compared

Old Version 61

New Version Current

Key

What happens to V2-library-referenced content when a course is exported and imported?

Approach 1: Distributed version control

Approach 2: Centralized version control

Approach 3: Insanity

Decision

What does an OLX export with V2 library content look like?

Decision

What does an OLX export with V2 library content look like?

Urgency: High

Referencing V2 library content from V2 courses

How might V2 library content referencing differ in V2 courses (as opposed to V1 courses)?

Archive

Referencing V2 library content from V2 courses

How might V2 library content referencing differ in V2 courses (as opposed to V1 courses)?

Archive

Approach 1: Distributed version control

Approach 2: Centralized version control

Page Comparison

Versions Compared

Old Version 61

New Version Current

Key

What happens to V2-library-referenced content when a course is exported and imported?

Approach 1: Distributed version control

Approach 2: Centralized version control

Approach 3: Insanity

Decision

What does an OLX export with V2 library content look like?

Decision

What does an OLX export with V2 library content look like?

Urgency: High

Referencing V2 library content from V2 courses

How might V2 library content referencing differ in V2 courses (as opposed to V1 courses)?

Archive

Referencing V2 library content from V2 courses

How might V2 library content referencing differ in V2 courses (as opposed to V1 courses)?

Archive

Approach 1: Distributed version control

Approach 2: Centralized version control