Kyle/T&L Content Libraries V2 syncs (2023)

Owned by Kyle McCormick
Last updated: Jan 18, 2024 by Connor Haugh

2023-1-18 Handoff meeting

2023-1-8 meeting

  • Connor:

    • Now:

      • Import Bug (can you see this?: )

        • Local attempt : succeeded

        • Stage attempt : succeeded

        • Prod Attempt: failed

          • I assume this is because only the matching library is on prod. This is good because it allows me to replicate locally.

          • going to follow this through the changed parts we added

          • timebox to today, then work on the implementaiton of turning off sync we suggested.

          • connor, search for get_block_original_usage(...)

            • nope, nvm, this wouldn’t work with export/import when the source lib doesn’t exist on the target instance

          • for loop it is

          •  

        • Ray’s got one last frontend bug

    • Next:

      • Get current branch to fully functional (lint, unit tests, round trip working with candidates)

        • Path forward:

          • in order to completely de-LCB children, Connor is going to create a LCB child → library child static conversion method.

            • Connor is going to timebox that effort to half a day.

            • If that fails, connor is going to remove editing canidacy from the “view“ page.

            • get_block_original_usage ()

      • Peter Pinch: V1 Libraries Page loading slowly

        • @Connor Haugh to ask partner support

      •  

2023-12-18 meeting

  •  

  • Work In Flight:

    • Connor:

      • On-Call wrecked Bernie Jesper & Ray’s week last week. They are picking up where they left off pretty much. Bernie is adding code to blockstore to make bundles inspectable in django admin by more than superusers as an app-plugin (app-permissions type role).

      • I have one dumb mock thing to debug on: Library Redirects.

      • I have to wait on the react-based editor for:

        • Kyle -- another pass

      • I need your final thumb on https://github.com/openedx/edx-platform/pull/33628 (super quick).

        • Connor To look at.

      • just needs to wait on me testing Braden’s fix on stage.

    • Kyle:

        • Quick + Dirty:

          • Fail loudly (to users or to logs) (Silence the exception)

          • Default settings changed will be missing

          • OR Refeesh blocks from the library on duplicate (harmless) + import export (NOT harmless, could have big bugs).

        • My rec if you want to go down this path

          • Refresh children for duplication

            • copy-paste does this now (see Braden’s PR)

          • Let the defaults be missing for import/export

    •  

  • Moving forward:

    • I do not think I have the “decision makers” with me today because of vacation and some personal stuff for them. I do grok the intentions behind what 2U’s perspective is: “We need to be only working on things that help our bottom line in the short term.” Libraries' ability to do that is less clear than other TNL projects for 2024. Consequently, I sit in a moment of tumult, where I should be pivoting, but don’t want my good work to go to waste. Last week we talked about the need to figure that out. I would love to spend some time coming up with a plan here that we can take to both axim and 2u leaders and feel good about. In theory, If I’m still developing libraries stuff ~january 8th, there is a problem.

    • Increments:

      • New Library Block Editor, Specific Reference (doesn’t require V2).

        • Ray dependant at this point.

        • TNL will focus on this.

      • New Library Authoring Experience live on edx.org

        • v1s migrated & mapped to v2

        • The main piece of work left here is “get blockstore-backed CL by version.“ If you would rather work on dave’s Learning core backed storage, maybe I can hop on that specific bug this week?

      • New backing storage for V2 libraries.

  •  

2023-12-11 meeting

  • Work schedule:

    • 2U OO 12/22-1/2 exclusive

    • Axim: Kyle TBD, but probably similar

  • What are folks up to

    • Ray → Editor work is wrapping up.

    • Jesper → local editing

    • Bernie → misc review, defense. fixed email bug

    • Put PR requests in such in #openedx-content-libraries-v2 channel

  • Keys & Learner State

    • two bugs

      • Upgrading an LCB's source lib from V1 to V2 might disassociate learner state, because it seems child use keys are derived differently between V1 and V2.

        • Open question: why is this done

        • What is the format for a block id for a block which is NOT imported from a v1 library?

        • Possible alternative solution:

          • Use a version of the v2 library import client which has the field “use_course_key_as_suffix“= True or something like that

          • In that case, don’t add the course key as an additional hash to the block id.

          • This would allow for a much easier fix to the generate_block_key function where we could, regardless of the v2 block id, blindly pass it into the derive_key function?

          •  

          • LIB KEY MAPPING: library-v1:ORG+LIB <-> lib:ORG:LIB LIB BLOCK USAGE KEY MAPPING: block-v1:ORG+LIB+type@TYPE+block+BLOCK <-> lb:ORG:LIB:TYPE:BLOCK dest_usage_key = dest_course_key.make_usage_key( derive_key( lib_key_v2_to_v1(source_lib_key), source_lib_block_usage_key.block_id, dest_lcb_usage_key.block_id, ) )

      • Copy-paste followed by update-library-version might also disassociate learner state, because (I believe) copy-paste randomly generates child keys instead of deriving them.

        • I’m hoping Braden will take this on.

    •  

2023-12-04 meeting

  • The Cavalry is coming:

    • Going to bring on Jesper & Bernie for this sprint so we can wrap stuff up hopefully in the next two weeks.

  • Wanna pair on review?

2023-11-27 meeting

  • capa_type filtering – just to confirm, we are leaving this out of the new library_content block editor for launch, right?

    • Yes

  • How’s the duplication bug (and import bug too, I think?) going? Need any help? -Kyle

    • Importing libraries

      • Import is working, but submitting an error message. This is I think because no courselike key is being applied to the function.

      • Prooobbably not the most recent PR, related to Course Authoring work.

    • Importing courses with library blocks

      • Haven’t tested in prod recently

      • No reports of issues

      • But this code overlaps with duplication code, so keep an eye out

    • Figuring out why duplication tasks fail has been trickier, however last night I determined that it was only duplications with overrides that failed. Going to add some logging there by eod.

      • Makes sense -Kyle

      • user_tasks is helpful but doesn’t have detailed errors

        • but we were seeing “Failed”

        • Connor is adding logging

    • old V1 editor for library_content takes a while to load

      • transient?

      • Are import tasks in progress? Spinner for editor loading (upgrade_and_sync) until load is complete.

        • even spins when no library sometimes??

        • theoretically we should only need to block children display, not settings editor

  • Kyle’s update: old lib versions in V2 - still working on this

  • Static reference

    • Connor is straightening out merge, then can incorporate feedback

    • Kyle needs to respond to some comments

  •  

2023-11-20 meeting

  • LETS GO WE FINALLY MERGED and it didn’t totally exploded (yet)

  • Priorities:

    • What is there to learn from stage migration? Do we put that over bug fixes like v2 duplication?

    • .

  • Things to keep in mind

    • Bug: duplication task failing. Affects:

      • duplication

      • course import?

    • Getting specific v2 library by version. Affects:

      • duplication

      • course import

    • Emails

      • All lib content syncing happens in a celery task now

        • @Connor Haugh will fix extraneous emails ^

          • user_tasks

      • That celery task sends an email when any of the following happens:

        • picking a library

        • “Update Now” button

        • duplicating library blocks

  • What’s blocking testing the migration on stage?

    • Key mapping

      • (soft blocker: static reference Pr is the basis for this)

        • @Kyle McCormick review ^

  • Shoot for next week for stage migration

 

2023-11-13 meeting

  • status: The Library Block Editor is nearing completion.

  • this pr:

    • apply fix from the bug connor found

  • up next

    • Review v1->v2 key mapping

    • Review static reference PR

    • Ray will review add-editor-to-edx-platform PR

      • will be done once editor is working

    • content_libraries test suite - turn it back on

    • If V2 fully enabled, then redirect V1 library links to V2

    • Library import/export

    • Import-library-from-course-content

      • functionally works

      • missing UI stuff like header, filtering

      • we should either build it out, or hide it

 

2023-11-06 meeting

    • wrap this up in an hour, then review & test.

      • merge tomorrow morning??

    • versioning of v2 libs

      • Why do we need versions for refresh children:

        • for post_import (pull in the requisite version from the xml doc)

        • duplicate functionality (pull in the version from the copy version.)

      • ^ Kyle will do this as follow up to #33262

  • Status (TNL):

    • Library Authoring

  • Up next

    • Review v1->v2 key mapping

    • Review static reference PR

    • Ray will review add-editor-to-edx-platform PR

    • content_libraries test suite - turn it back on

    • If V2 fully enabled, then redirect V1 library links to V2

    • Library import/export

    • Import-library-from-course-content

      • functionally works

      • missing UI stuff like header, filtering

      • we should either build it out, or hide it

2023-10-30 meeting

  • Fallout from V2 in library xblocks.

    • Address Connor’s dupe code comment

    • Then merge (“soft rollout of V2 content”)

    • Improvements to Permissions + Roles

    • Brian S is fixing a library-authoring path issue in Tutor

  • Ray is working on new library editor

  • Kristin is working on library authoring UI/UX bugs

  • @Connor Haugh Look into the “libraries list“ in xblock is amenable for merging.

  • Just FYI

  • Post-thanksviging tuesday (28th of Nov) for cutover event -- soft deadline

    • downtime-

 

2023-10-23 meeting

  • import/export tests?

    • need django settings to be correct for cms for devstack?

    • I might ask for some pairing on this.

  • [Kyle] After Tutor being broken thurs->friday, I finally have a working async test environment! I’m testing now. Let’s aim to merge first thing tomorrow (Tues) unless I find anything broken.

  • Migration strategies ()

    • Modify keys in the database w/ mgmt command

    • Runtime mapping w/ waffle flag

    • should “Runtime mapping“ be persistent or not?

      1. I think not, after some pros-cons:

        1. pros:

          1. we never have to repeat this operation

          2. we may be able to remove the code as usage decreases.

        2. cons:

          1. manipulating code is bad.

      2. This can always be done later, and we should only be writing keys back, when we are confident the migration succeeded.

        1. Migration succeeded == V2 libraries are in use on edx.org and stable.

      3. DECISION: We will not write back v2 keys until we are fully confident that the v2 experience will not be reverted.

2023-10-16 meeting

From Connor:

  • PTO was much needed.

  • We’ve polished up the migration PRs after hitting some minor snarls with running on more-production environments (sandbox) and having more complex states to validate, but that has been helpful.

  • I need a little more time to get the V2 library block editor off the ground, but it is going well. The backend work should be ready to review though?

  • Who should be in charge of merging the V2 libs in xblocks pr? I’m happy to once tests are green.

  • TODO: we should make a matching runbook for Tutor deployments because they will have slightly different steps, right?

    • Configuration

    • Running management commands.

  • To do in the next two weeks

    • Apply feedback from sandbox

      • TNL will handle this (probably Ray, etc)

    • merge V2 libs in xblocks PR

      • Connor and Kyle both will check locally that this doesn’t break V1

      • linting too obvi (Kyle)

      • Then we can merge

    • Static reference support

      • Backend block changes

        • This are mostly good to go

      • New frontend-lib-content-components editor for static reference

        • edx-platform work required

        • A non-trivial amount of FE work has been done, more required

  • Thinking about migration

    • Dave’s comment:

    • Connor will look into the current implementation + block transformers.

2023-10-2 meeting

From Connor:

TL;DR: working on one library block, I’m happy with where the “V2 libraries“ PR is at, and am looking for a final review.

  • Status of "user tasks" plugin in LMS:

    • I tried moving the required code to the following places:

      • cms/djangoapps/contenstore/libraries/tasks.py

        • This didn’t work because there are several circular dependencies. Also, this is a “V1“ location, so it doesn’t make sense to keep the code here.

      • openedx/core/djangoapps/content_libraries/tasks.py

        • This did not remove the requirement for “user tasks“ to be installed in the LMS. I’m not sure why that would be happening, as we can see that content libraries is installed in both the cms and lms. I’m not an expert on celery, but I feel like the amount of required work to fix this outweighs the potential damage of having user tasks in the LMS (which from a conceptual standpoint seems fine to me).

      •  

    • Consequently, as this code should live with openedx/content_libraries, I have left it there.

    • Kyle is worried about the bug that this implies, will look into this

  • Status of “One Xblock work“:

    • Status:

      • I have gotten the CMS side working, and now have run into a few odd errors with the library_content block in the student view. Also will need to probably make sure v1 library editing still works. Mostly confident in my ability to get this done, but might have some opportunities to pair this week.

      • Still will need to write a bunch of unit tests.

    • Minor schema tweaks:

      • I represented “Shuffle“ and “Manual“ as a 2x2 matrix, and assigned each element of the matrix as an entry in the “Mode“ dropdown. This allows us to represent the states in english to the author in a clean way that makes sense. It also makes the backend logic better separated, as we can have well-delineated selection modes for each one.

      • I added a “candidates“ field which holds that block ids selected by the user for “manual selection.“ This is because unselected children still need to be represented in the children object, for display on the selection screen.

    • Connor will make a V2 editor for this



 

2023-09-27 meeting

For discussion, From the ADR:

2023-09-11 meeting

 

Since last time

  • Connor is on call, kyle is filling this out:

Plan

  • Connor works on library_content v2 PR

  • Kyle fills out and reviews PR

  • Connor will be able to use ^ to test PR

2023-09-05 meeting

  • Prioritization decision

    • v2 randomized reference in library_content: first priority. Release blocker.

    • v2 specific reference in library_content: immediate next priority. Want in release (one xblock type) but may release without it.

    • library->library import/export: next priority, might miss release.

  •  

  • [kyle] Recommendations for library reference.

    • Don’t release library_sourced. Instead, add non-randomized reference support to library_content. (We’ve already talked about this, but I’m doubling down :)

      • Tech reasoning: the library_content block is already hard enough to understand. Adding a second block type, with overlapping features, will only make that worse, especially when the two implementations begin to drift, double-especially during the migration period where each block would have two backends (v1 and v2).

      • Product reasoning: It gives us less flexibility with the end-user experience. With two blocks on the backend, we would have to show it as two different component types in the UI. With one unified block on the backend, we can decide whether to present it to the course author as one component type or two, and we can change that decision at any point.

      • This change will have to be made before initial release. Releasing the library_sourced block means supporting it potentially forever.

    • Persist default library block settings in course’s OLX (“defaults.json”) so that library_content blocks work on any instance, regardless of whether the source library is there.

      • This addresses the fact that, today, importing a course with a library_content block into an instance without the source library will result in a semi-broken library_content block. Specifically, any library-defined settings, such as title, will be missing from library content.

      • This change is backwards-compatible. It could be made before, as part of, or after the initial V2 release.

      • Note: Not a 2U priority. Perhaps Axim/funded/CC followup.

    • [Connor] as the person who posed this question in the first place you have a backer in me.

    •  

  • [kyle] Outstanding technical Qs

    • UPDATE: I moved the questions here:

    • Haven’t figured out yet how v1 libraries figure out that an update is available, and whether/how that works in v2 libraries.

    • When v1 lib updates are pulled into a library_content block, any overrides to settings will survive, but any updates to content will be clobbered. Still working on confirming that this is the case with V2 libraries (@Connor Haugh , have you confirmed this?). I’m concerned that it could differ because in V1, courses shared block content definitions with libraries; but in v2, content definitions are copied imported from blockstore to modulestore.

    • Does the update-from-latest thing work with the v2 library_content block?

    • Does the v2 library export need to change at all, or could it be identical to the v1 export?

  • [kyle] Product Qs

    • Mass course->library export: what state is that in? is it something we could eventually polish up into a nice UI for authors?

    • Assuming that we release a single block for randomized and non-randomized content, and assuming that subsets of libraries can be selected (like they can in library_sourced today), does that mean that we get “randomized library subset” as a new feature?

      • Nice to have, not for initial release though.

      • Note: CAPA type theoretically is a version of this, but doesn’t seem to be working on edx.org.

        • JK Connor just fixed that

        •  

2023-08-28

Connor update:

  • working branch at

    • Needs: test, lint

    • additional feature is WIP: switch (waffle) between v1 + v2 xblocks being shown as editor options in courses (adding waffle flag) Should be a follow-on PR

    •  

    • Learning: library_source and library_content block should be the same block.

      • Why weren’t they? Are xblock field migrations taxing?

      • How do we think about the pros/cons of merging them?

    • It’s a big PR with a lot of commits, some more atomic than others. My review plan is this:

      • TNL internal: create a sandbox for UAT

      • Braden, Dave, Kyle, and Bernie: take a glance at the code.

        • Dave + kyle will look into this ASAP.

        • Do we want to split it up?

Kyle Update:

  • Building understanding of 2020 library incident

  • V1->V2 library import - Still working on it

  • Talking with Dave on eventual Learning Core migration, and what to do (or not to do) now in order to make that less painful.

 

V1->V2 rollout

  • Copy step: copy all v1 into v2 libraries

  • Don’t delete V1 libraries until migration clearly successful

    • A few days/weeks

  •  

 

2023-08-21

  • Connor update

    • Re-merging changes that RG had made back in Dec 2022

    • Improvements/source to library sourced block.

  • The three things Kyle will work on

    • v1->v2 library import : Now

    • v2->v2 library import : Soon (Dave’s returning from PTO)

    • importing v2 library content into v1 courses : Now (Kyle will reach out to learn more about the bug that happened)

  • Shared #content-libraries-v2 channel: can we get everyone working on v2 libraries to use it?

    • Done.

  • Importing courses with library content: Let’s not recreate this situation.

  • State of other authoring work in flight

    • “import/export rest api”: That’s a thing you guys are working on, right? How does it differ from the current ajax import handler?

      • It actually isn’t based on import/export. Instead it simply offers a set of existing content apis as “public” apis which can then be interacted with by trusted parties.

    • ENABLE_NEW_STUDIO_* toggles in contentstore - what’s enabled now? where can I follow along with RG’s Studio MFE efforts?

      • I’ll ask kristin for the status spreadsheet

        • current status sheet not public, but currently all waffle flags are off in prod on edx.org, with some in internal/external beta.

  • Less important for me right now (not import-related)

    • Blockstore library-ification

      • Why reuse existing Blockstore RDS rather than just running migrations to create new Blockstore tables in the edxapp db?

        • running migrations, SRE choice.

    • Merging course authoring and library authoring mfes - do you think it’ll happen?

      • I certainly want it to, and TNL is attempting to earn some leeway to work on ownership.

  • What connor is working on DRAFT PR FOR LIBRAY SOURCE xblock