/
Kyle/T&L Content Libraries V2 syncs (2023)

Kyle/T&L Content Libraries V2 syncs (2023)

2023-1-18 Handoff meeting

  • Ray:

  • Specs/Designs

  • Feature Tickets

    • Library to Library Import+Export

      • AC:

        • v2 Library Authoring experience must support import of a package that contains: Either a v1 library or a v2 library

        • v2 Library Authoring experience must support export of a package that contains: a v2 library

        • NOT IN SCOPE: exporting v2 libs as v1 libs is not scope,

        • NOT IN SCOPE: importing existing v2 libs that the user can read is also not in scope.

        Discovery sub-task questions:

        • What is the file format of v2 library packages?

        • What info should be in and out of the package?

          • How do library-specific permissions work with exports and imports?

        • Do we need to call out that import is from v1 or v2 to the user?

        I see several tasks here:

        • Determine a format/spec for v2 lib exports.

        • Create the library export page

        • Write the export celery task

        • Write the import from v2 celery task

        • add the logic to handle imports of courses, v1 and v2 libs.

    • [Post Rollout] Bulk Delete of Content Libraries

      • Mockups needed here, as this is not a feature that currently exists.

    • [Post-Rollout] Update the “Advanced” Component access to be for developers only.

      • This is blocked by the addition of the new editors inside the LA MFE.

        ACs:

        • We want developers to be able to use the old editing experience developed by opencraft

          • We’re not sure why though.

        • When the dev has this mode enabled, clicking edit on a block brings them to the old experience.

        Technical notes: 

        Use an MFE level feature flag to enable/disable this behavior.

    • [POST ROLLOUT] Remove routing code & remove waffle switch

      • We previously (in this epic) removed the need to have a waffle flag which controls whether blockstore is an app or service.

        ACs:

        Notes:

        Test that blockstore CRUD for libraries still works!

    • copy paste does not exist in V2 (non blocker for release)

    • text templates are not available in V2 (not a blocker for release)

      • text templates for the text xblock like “announcement“ are not available in V2. Will require some feature development.

    • https://github.com/openedx/frontend-app-library-authoring/issues/81

    • Small Fixes to Course Import into Libraries Page

      • On the course import into libraries page:

        • Make the header display the header info for the current library

        • Make the filtering features conditional on a feature flag, set that feature flag to default to false.

  • Bug tickets

    • https://2u-internal.atlassian.net/browse/TNL-11365 (Will be resolved by EOD for edx.org, but will also need a fix)

      • MIT encountering gateway timeout after course import failure when trying to edit library content in course-v1:MITx+CTL.SC1x_1+1T2024

        • MIT’s randomized content blocks are not loading when they try to edit them in the course at course-v1:MITx+CTL.SC1x_1+1T2024.

        • Users are timing out when loading the studio handler. We can see this is because it is calling bundleversion select a bunch

        •  

        • turns out, no bundles exist on prod atm (and no v2 libs)

        • So we need to figure out how gracefully return no libraries when there are no V2 libraries, because this breaks LCB editing.

    • Library Authoring UAT Feedback: Hovering over many links (e.g. in the "Libraries" list) does not show the URL that they link to in the browser's status bar

      • Hovering over many links (e.g. in the "Libraries" list) do not show the URL that they link to in the browser's status bar.  This is an a11y issue - they aren't "real" links; but they should be as they go to a different page.

    • The Library Details “Submitting” state does not disable actions.

      • The Library Details “Submitting” state is not ideal.  Buttons are still clickable, and there is no spinner or graying out of the page, and no feedback is provided to the user.

        1. Saw this when saving library details.

    •  

2023-1-8 meeting

  • Connor:

    • Now:

      • Import Bug (can you see this?: https://2u-internal.atlassian.net/browse/TNL-11339 )

        • Local attempt : succeeded

        • Stage attempt : succeeded

        • Prod Attempt: failed

          • I assume this is because only the matching library is on prod. This is good because it allows me to replicate locally.

          • going to follow this through the changed parts we added

          • timebox to today, then work on the implementaiton of turning off sync we suggested.

          • connor, search for get_block_original_usage(...)

            • nope, nvm, this wouldn’t work with export/import when the source lib doesn’t exist on the target instance

          • for loop it is

          •  

        • Ray’s got one last frontend bug

    • Next:

      • Get current branch to fully functional (lint, unit tests, round trip working with candidates)

        • Path forward:

          • in order to completely de-LCB children, Connor is going to create a LCB child → library child static conversion method.

            • Connor is going to timebox that effort to half a day.

            • If that fails, connor is going to remove editing canidacy from the “view“ page.

            • get_block_original_usage ()

      • Peter Pinch: V1 Libraries Page loading slowly

        • @Connor Haugh (Deactivated) to ask partner support

      •  

2023-12-18 meeting

  •  

  • Work In Flight:

  • Moving forward:

    • I do not think I have the “decision makers” with me today because of vacation and some personal stuff for them. I do grok the intentions behind what 2U’s perspective is: “We need to be only working on things that help our bottom line in the short term.” Libraries' ability to do that is less clear than other TNL projects for 2024. Consequently, I sit in a moment of tumult, where I should be pivoting, but don’t want my good work to go to waste. Last week we talked about the need to figure that out. I would love to spend some time coming up with a plan here that we can take to both axim and 2u leaders and feel good about. In theory, If I’m still developing libraries stuff ~january 8th, there is a problem.

    • Increments:

      • New Library Block Editor, Specific Reference (doesn’t require V2).

        • Ray dependant at this point.

        • TNL will focus on this.

      • New Library Authoring Experience live on edx.org

        • v1s migrated & mapped to v2

        • The main piece of work left here is “get blockstore-backed CL by version.“ If you would rather work on dave’s Learning core backed storage, maybe I can hop on that specific bug this week?

      • New backing storage for V2 libraries.

  •  

2023-12-11 meeting

  • Work schedule:

    • 2U OO 12/22-1/2 exclusive

    • Axim: Kyle TBD, but probably similar

  • What are folks up to

    • Ray → Editor work is wrapping up.

    • Jesper → local editing

    • Bernie → misc review, defense. fixed email bug

    • Put PR requests in such in #openedx-content-libraries-v2 channel

  • Keys & Learner State

    • two bugs

      • Upgrading an LCB's source lib from V1 to V2 might disassociate learner state, because it seems child use keys are derived differently between V1 and V2.

        • Open question: why is this donehttps://github.com/openedx/edx-platform/commit/eec72431419664385237cf1dfddfa44d19d610b9#diff-69a28742894c4b286b463a4baede1d8ad03ef4418779f73d8cd4ef42e8879687

        • What is the format for a block id for a block which is NOT imported from a v1 library?

        • Possible alternative solution:

          • Use a version of the v2 library import client which has the field “use_course_key_as_suffix“= True or something like that

          • In that case, don’t add the course key as an additional hash to the block id.

          • This would allow for a much easier fix to the generate_block_key function where we could, regardless of the v2 block id, blindly pass it into the derive_key function?

          •  

          • LIB KEY MAPPING: library-v1:ORG+LIB <-> lib:ORG:LIB LIB BLOCK USAGE KEY MAPPING: block-v1:ORG+LIB+type@TYPE+block+BLOCK <-> lb:ORG:LIB:TYPE:BLOCK dest_usage_key = dest_course_key.make_usage_key( derive_key( lib_key_v2_to_v1(source_lib_key), source_lib_block_usage_key.block_id, dest_lcb_usage_key.block_id, ) )

      • Copy-paste followed by update-library-version might also disassociate learner state, because (I believe) copy-paste randomly generates child keys instead of deriving them.

        • I’m hoping Braden will take this on.

    •  

2023-12-04 meeting

2023-11-27 meeting

  • capa_type filtering – just to confirm, we are leaving this out of the new library_content block editor for launch, right?

    • Yes

  • How’s the duplication bug (and import bug too, I think?) going? Need any help? -Kyle

    • Importing libraries

      • Import is working, but submitting an error message. This is I think because no courselike key is being applied to the function.

      • Prooobbably not the most recent PR, related to Course Authoring work.

    • Importing courses with library blocks

      • Haven’t tested in prod recently

      • No reports of issues

      • But this code overlaps with duplication code, so keep an eye out

    • Figuring out why duplication tasks fail has been trickier, however last night I determined that it was only duplications with overrides that failed. Going to add some logging there by eod.

      • Makes sense -Kyle

      • user_tasks is helpful but doesn’t have detailed errors

        • but we were seeing “Failed”

        • Connor is adding logging

    • old V1 editor for library_content takes a while to load

      • transient?

      • Are import tasks in progress? Spinner for editor loading (upgrade_and_sync) until load is complete.

        • even spins when no library sometimes??

        • theoretically we should only need to block children display, not settings editor

  • Kyle’s update: old lib versions in V2 - still working on this

  • Static reference

    • Connor is straightening out merge, then can incorporate feedback

    • Kyle needs to respond to some comments

  •  

2023-11-20 meeting

  • LETS GO WE FINALLY MERGED https://github.com/openedx/edx-platform/pull/33263 and it didn’t totally exploded (yet)

  • Priorities:

    • What is there to learn from stage migration? Do we put that over bug fixes like v2 duplication?

    • .

  • Things to keep in mind

    • Bug: duplication task failing. Affects:

      • duplication

      • course import?

    • Getting specific v2 library by version. Affects:

      • duplication

      • course import

    • Emails

      • All lib content syncing happens in a celery task now

        • @Connor Haugh (Deactivated) will fix extraneous emails ^

          • user_tasks

      • That celery task sends an email when any of the following happens:

        • picking a library

        • “Update Now” button

        • duplicating library blocks

  • What’s blocking testing the migration on stage?

    • Key mapping

      • (soft blocker: static reference Pr is the basis for this)

        • @Kyle McCormick review ^

  • Shoot for next week for stage migration

 

2023-11-13 meeting

  • status: The Library Block Editor is nearing completion.

  • this pr: https://github.com/openedx/edx-platform/pull/33263

    • apply fix from the bug connor found

  • up next

    • Review v1->v2 key mapping

    • Review static reference PR

    • Ray will review add-editor-to-edx-platform PR

      • will be done once editor is working

    • content_libraries test suite - turn it back on

    • If V2 fully enabled, then redirect V1 library links to V2

    • Library import/export

    • Import-library-from-course-content

      • functionally works

      • missing UI stuff like header, filtering

      • we should either build it out, or hide it

 

2023-11-06 meeting

  • https://github.com/openedx/edx-platform/pull/33263

    • wrap this up in an hour, then review & test.

      • merge tomorrow morning??

    • versioning of v2 libs

      • Why do we need versions for refresh children:

        • for post_import (pull in the requisite version from the xml doc)

        • duplicate functionality (pull in the version from the copy version.)

      • ^ Kyle will do this as follow up to #33262

  • Status (TNL):

    • Library Authoring

  • Up next

    • Review v1->v2 key mapping

    • Review static reference PR

    • Ray will review add-editor-to-edx-platform PR

    • content_libraries test suite - turn it back on

    • If V2 fully enabled, then redirect V1 library links to V2

    • Library import/export

    • Import-library-from-course-content

      • functionally works

      • missing UI stuff like header, filtering

      • we should either build it out, or hide it

2023-10-30 meeting

  • Fallout from V2 in library xblocks.

    • Address Connor’s dupe code comment

    • Then merge (“soft rollout of V2 content”)

    • Improvements to Permissions + Roles

    • Brian S is fixing a library-authoring path issue in Tutor

  • Ray is working on new library editor

  • Kristin is working on library authoring UI/UX bugs

  • @Connor Haugh (Deactivated) Look into the “libraries list“ in xblock is amenable for merging.

  • Just FYI

  • Post-thanksviging tuesday (28th of Nov) for cutover event -- soft deadline

    • downtime-

 

2023-10-23 meeting

  • import/export tests?

    • need django settings to be correct for cms for devstack?

    • I might ask for some pairing on this.

  • [Kyle] After Tutor being broken thurs->friday, I finally have a working async test environment! I’m testing https://github.com/openedx/edx-platform/pull/33263 now. Let’s aim to merge first thing tomorrow (Tues) unless I find anything broken.

  • Migration strategies (https://github.com/openedx/edx-platform/pull/33231)

    • https://2u-internal.atlassian.net/browse/TNL-11138

    • Modify keys in the database w/ mgmt command

    • Runtime mapping w/ waffle flag

    • should “Runtime mapping“ be persistent or not?

      1. I think not, after some pros-cons:

        1. pros:

          1. we never have to repeat this operation

          2. we may be able to remove the code as usage decreases.

        2. cons:

          1. manipulating code is bad.

      2. This can always be done later, and we should only be writing keys back, when we are confident the migration succeeded.

        1. Migration succeeded == V2 libraries are in use on edx.org and stable.

      3. DECISION: We will not write back v2 keys until we are fully confident that the v2 experience will not be reverted.

2023-10-16 meeting

From Connor:

  • PTO was much needed.

  • We’ve polished up the migration PRs after hitting some minor snarls with running on more-production environments (sandbox) and having more complex states to validate, but that has been helpful.

  • I need a little more time to get the V2 library block editor off the ground, but it is going well. The backend work should be ready to review though?

  • Who should be in charge of merging the V2 libs in xblocks pr? I’m happy to once tests are green.

  • TODO: we should make a matching runbook for Tutor deployments because they will have slightly different steps, right?

    • Configuration

    • Running management commands.

  • To do in the next two weeks

    • Apply feedback from sandbox