2023-1-18 Handoff meeting

Ray:
- https://github.com/openedx/frontend-lib-content-components/actions/runs/7546505259/job/20544397615
- has lint + unit test level work left.
Specs/Designs
- [Spec Memo] Content Library Authoring Experience V2
- V1->V2 libraries runbook (I know this is defunct) https://2u-internal.atlassian.net/wiki/spaces/TNL/pages/494633048
Feature Tickets
- Library to Library Import+Export
  - AC:
    - v2 Library Authoring experience must support import of a package that contains: Either a v1 library or a v2 library
    - v2 Library Authoring experience must support export of a package that contains: a v2 library
    - NOT IN SCOPE: exporting v2 libs as v1 libs is not scope,
    - NOT IN SCOPE: importing existing v2 libs that the user can read is also not in scope.
    Discovery sub-task questions:
    - What is the file format of v2 library packages?
    - What info should be in and out of the package?
      - How do library-specific permissions work with exports and imports?
    - Do we need to call out that import is from v1 or v2 to the user?
    I see several tasks here:
    - Determine a format/spec for v2 lib exports.
    - Create the library export page
    - Write the export celery task
    - Write the import from v2 celery task
    - add the logic to handle imports of courses, v1 and v2 libs.
- [Post Rollout] Bulk Delete of Content Libraries
  - Mockups needed here, as this is not a feature that currently exists.
- [Post-Rollout] Update the “Advanced” Component access to be for developers only.
  - This is blocked by the addition of the new editors inside the LA MFE.
    ACs:
    - We want developers to be able to use the old editing experience developed by opencraft
      - We’re not sure why though.
    - When the dev has this mode enabled, clicking edit on a block brings them to the old experience.
    Technical notes:
    Use an MFE level feature flag to enable/disable this behavior.
- [POST ROLLOUT] Remove routing code & remove waffle switch
  - We previously (in this epic) removed the need to have a waffle flag which controls whether blockstore is an app or service.
    ACs:
    - The Waffle Flag blockstore.use_blockstore_app_api should be removed, so that the app api is always used.
    - Conditionals which use that waffle flag should no longer check the waffle flag (most will be completely removed).
      - including, but not limited to:
        https://github.com/openedx/edx-platform/search?q=BLOCKSTORE_USE_BLOCKSTORE_APP_API
    - the instances of the waffle flag’s setting value should be removed from Django admin.
    Notes:
    Test that blockstore CRUD for libraries still works!
- copy paste does not exist in V2 (non blocker for release)
- text templates are not available in V2 (not a blocker for release)
  - text templates for the text xblock like “announcement“ are not available in V2. Will require some feature development.
- https://github.com/openedx/frontend-app-library-authoring/issues/81
- Small Fixes to Course Import into Libraries Page
  - On the course import into libraries page:
    - Make the header display the header info for the current library
    - Make the filtering features conditional on a feature flag, set that feature flag to default to false.
Bug tickets
- https://2u-internal.atlassian.net/browse/TNL-11365 (Will be resolved by EOD for edx.org, but will also need a fix)
  - MIT encountering gateway timeout after course import failure when trying to edit library content in course-v1:MITx+CTL.SC1x_1+1T2024
    - MIT’s randomized content blocks are not loading when they try to edit them in the course at course-v1:MITx+CTL.SC1x_1+1T2024.
    - Users are timing out when loading the studio handler. We can see this is because it is calling bundleversion select a bunch
    - turns out, no bundles exist on prod atm (and no v2 libs)
    - So we need to figure out how gracefully return no libraries when there are no V2 libraries, because this breaks LCB editing.
- Library Authoring UAT Feedback: Hovering over many links (e.g. in the "Libraries" list) does not show the URL that they link to in the browser's status bar
  - Hovering over many links (e.g. in the "Libraries" list) do not show the URL that they link to in the browser's status bar. This is an a11y issue - they aren't "real" links; but they should be as they go to a different page.
- The Library Details “Submitting” state does not disable actions.
  - The Library Details “Submitting” state is not ideal. Buttons are still clickable, and there is no spinner or graying out of the page, and no feedback is provided to the user.
    1. Saw this when saving library details.

2023-1-8 meeting

Connor:
- Now:
  - Import Bug (can you see this?: https://2u-internal.atlassian.net/browse/TNL-11339 )
    - Local attempt : succeeded
    - Stage attempt : succeeded
    - Prod Attempt: failed
      - I assume this is because only the matching library is on prod. This is good because it allows me to replicate locally.
      - going to follow this through the changed parts we added
      - timebox to today, then work on the implementaiton of turning off sync we suggested.
      - connor, search for get_block_original_usage(...)
        nope, nvm, this wouldn’t work with export/import when the source lib doesn’t exist on the target instance
      - for loop it is
    - Ray’s got one last frontend bug
- Next:
  - Get current branch to fully functional (lint, unit tests, round trip working with candidates)
    - Path forward:
      - in order to completely de-LCB children, Connor is going to create a LCB child → library child static conversion method.
        Connor is going to timebox that effort to half a day.
        If that fails, connor is going to remove editing canidacy from the “view“ page.
        get_block_original_usage ()
  - Peter Pinch: V1 Libraries Page loading slowly
    - Connor Haugh to ask partner support

2023-12-18 meeting

Work In Flight:
- Connor:
  - On-Call wrecked Bernie Jesper & Ray’s week last week. They are picking up where they left off pretty much. Bernie is adding code to blockstore to make bundles inspectable in django admin by more than superusers as an app-plugin (app-permissions type role).
  - I have one dumb mock thing to debug on: Library Redirects.
    - https://nedbatchelder.com/blog/201908/why_your_mock_doesnt_work.html
  - I have to wait on the react-based editor for: https://github.com/openedx/edx-platform/pull/33511
    - Kyle -- another pass
  - I need your final thumb on https://github.com/openedx/edx-platform/pull/33628 (super quick).
    - Connor To look at.
  - https://github.com/openedx/edx-platform/pull/33920 just needs to wait on me testing Braden’s fix on stage.
- Kyle:
  - Fun with LibraryContentBlock export/import and duplication
    - Quick + Dirty:
      - Fail loudly (to users or to logs) (Silence the exception)
      - Default settings changed will be missing
      - OR Refeesh blocks from the library on duplicate (harmless) + import export (NOT harmless, could have big bugs).
    - My rec if you want to go down this path
      - Refresh children for duplication
        copy-paste does this now (see Braden’s PR)
      - Let the defaults be missing for import/export
Moving forward:
- I do not think I have the “decision makers” with me today because of vacation and some personal stuff for them. I do grok the intentions behind what 2U’s perspective is: “We need to be only working on things that help our bottom line in the short term.” Libraries' ability to do that is less clear than other TNL projects for 2024. Consequently, I sit in a moment of tumult, where I should be pivoting, but don’t want my good work to go to waste. Last week we talked about the need to figure that out. I would love to spend some time coming up with a plan here that we can take to both axim and 2u leaders and feel good about. In theory, If I’m still developing libraries stuff ~january 8th, there is a problem.
- Increments:
  - New Library Block Editor, Specific Reference (doesn’t require V2).
    - Ray dependant at this point.
    - TNL will focus on this.
  - New Library Authoring Experience live on edx.org
    - v1s migrated & mapped to v2
    - The main piece of work left here is “get blockstore-backed CL by version.“ If you would rather work on dave’s Learning core backed storage, maybe I can hop on that specific bug this week?
  - New backing storage for V2 libraries.

2023-12-11 meeting

Work schedule:
- 2U OO 12/22-1/2 exclusive
- Axim: Kyle TBD, but probably similar
What are folks up to
- Ray → Editor work is wrapping up.
- Jesper → local editing
- Bernie → misc review, defense. fixed email bug
- Put PR requests in such in #openedx-content-libraries-v2 channel

Keys & Learner State

two bugs

Upgrading an LCB's source lib from V1 to V2 might disassociate learner state, because it seems child use keys are derived differently between V1 and V2.

Open question: why is this donehttps://github.com/openedx/edx-platform/commit/eec72431419664385237cf1dfddfa44d19d610b9#diff-69a28742894c4b286b463a4baede1d8ad03ef4418779f73d8cd4ef42e8879687
What is the format for a block id for a block which is NOT imported from a v1 library?

Possible alternative solution:

Use a version of the v2 library import client which has the field “use_course_key_as_suffix“= True or something like that
In that case, don’t add the course key as an additional hash to the block id.
This would allow for a much easier fix to the generate_block_key function where we could, regardless of the v2 block id, blindly pass it into the derive_key function?

LIB KEY MAPPING:
library-v1:ORG+LIB <-> lib:ORG:LIB

LIB BLOCK USAGE KEY MAPPING:
block-v1:ORG+LIB+type@TYPE+block+BLOCK <-> lb:ORG:LIB:TYPE:BLOCK

dest_usage_key = dest_course_key.make_usage_key(
    derive_key(
        lib_key_v2_to_v1(source_lib_key),
        source_lib_block_usage_key.block_id,
        dest_lcb_usage_key.block_id,
    )
)

Copy-paste followed by update-library-version might also disassociate learner state, because (I believe) copy-paste randomly generates child keys instead of deriving them.
- I’m hoping Braden will take this on.

2023-12-04 meeting

The Cavalry is coming:
- Going to bring on Jesper & Bernie for this sprint so we can wrap stuff up hopefully in the next two weeks.
Wanna pair on https://github.com/openedx/edx-platform/pull/33511/files review?

2023-11-27 meeting

capa_type filtering – just to confirm, we are leaving this out of the new library_content block editor for launch, right?
- Yes
How’s the duplication bug (and import bug too, I think?) going? Need any help? -Kyle
- Importing libraries
  - Import is working, but submitting an error message. This is I think because no courselike key is being applied to the function.
  - Prooobbably not the most recent PR, related to Course Authoring work.
- Importing courses with library blocks
  - Haven’t tested in prod recently
  - No reports of issues
  - But this code overlaps with duplication code, so keep an eye out
- Figuring out why duplication tasks fail has been trickier, however last night I determined that it was only duplications with overrides that failed. Going to add some logging there by eod.
  - Makes sense -Kyle
  - user_tasks is helpful but doesn’t have detailed errors
    - but we were seeing “Failed”
    - Connor is adding logging
- old V1 editor for library_content takes a while to load
  - transient?
  - Are import tasks in progress? Spinner for editor loading (upgrade_and_sync) until load is complete.
    - even spins when no library sometimes??
    - theoretically we should only need to block children display, not settings editor
Kyle’s update: old lib versions in V2 - still working on this
Static reference
- Connor is straightening out merge, then can incorporate feedback
- Kyle needs to respond to some comments

2023-11-20 meeting

LETS GO WE FINALLY MERGED https://github.com/openedx/edx-platform/pull/33263 and it didn’t totally exploded (yet)
Priorities:
- What is there to learn from stage migration? Do we put that over bug fixes like v2 duplication?
- .
Things to keep in mind
- Bug: duplication task failing. Affects:
  - duplication
  - course import?
- Getting specific v2 library by version. Affects:
  - duplication
  - course import
- Emails
  - All lib content syncing happens in a celery task now
    - Connor Haugh will fix extraneous emails ^
      - user_tasks
  - That celery task sends an email when any of the following happens:
    - picking a library
    - “Update Now” button
    - duplicating library blocks
What’s blocking testing the migration on stage?
- Key mapping
  - (soft blocker: static reference Pr is the basis for this)
    - Kyle McCormick review ^
Shoot for next week for stage migration

2023-11-13 meeting

status: The Library Block Editor is nearing completion.
this pr: https://github.com/openedx/edx-platform/pull/33263
- apply fix from the bug connor found
up next
- Review v1->v2 key mapping
- Review static reference PR
- Ray will review add-editor-to-edx-platform PR
  - will be done once editor is working
- content_libraries test suite - turn it back on
- If V2 fully enabled, then redirect V1 library links to V2
- Library import/export
- Import-library-from-course-content
  - functionally works
  - missing UI stuff like header, filtering
  - we should either build it out, or hide it

2023-11-06 meeting

https://github.com/openedx/edx-platform/pull/33263
- wrap this up in an hour, then review & test.
  - merge tomorrow morning??
- versioning of v2 libs
  - Why do we need versions for refresh children:
    - for post_import (pull in the requisite version from the xml doc)
    - duplicate functionality (pull in the version from the copy version.)
  - ^ Kyle will do this as follow up to #33262
Status (TNL):
- Library Authoring
Up next
- Review v1->v2 key mapping
- Review static reference PR
- Ray will review add-editor-to-edx-platform PR
- content_libraries test suite - turn it back on
- If V2 fully enabled, then redirect V1 library links to V2
- Library import/export
- Import-library-from-course-content
  - functionally works
  - missing UI stuff like header, filtering
  - we should either build it out, or hide it

2023-10-30 meeting

Fallout from V2 in library xblocks.
- Address Connor’s dupe code comment
- Then merge (“soft rollout of V2 content”)
- Improvements to Permissions + Roles
- Brian S is fixing a library-authoring path issue in Tutor
Ray is working on new library editor
Kristin is working on library authoring UI/UX bugs
Connor Haugh Look into the “libraries list“ in xblock is amenable for merging.
Just FYI
- https://github.com/openedx/blockstore/issues/296
Post-thanksviging tuesday (28th of Nov) for cutover event -- soft deadline
- downtime-

2023-10-23 meeting

import/export tests?
- need django settings to be correct for cms for devstack?
- I might ask for some pairing on this.
[Kyle] After Tutor being broken thurs->friday, I finally have a working async test environment! I’m testing https://github.com/openedx/edx-platform/pull/33263 now. Let’s aim to merge first thing tomorrow (Tues) unless I find anything broken.
Migration strategies (https://github.com/openedx/edx-platform/pull/33231)
- https://2u-internal.atlassian.net/browse/TNL-11138
- Modify keys in the database w/ mgmt command
- Runtime mapping w/ waffle flag
- should “Runtime mapping“ be persistent or not?
  1. I think not, after some pros-cons:
    1. pros:
      1. we never have to repeat this operation
      2. we may be able to remove the code as usage decreases.
    2. cons:
      1. manipulating code is bad.
  2. This can always be done later, and we should only be writing keys back, when we are confident the migration succeeded.
    1. Migration succeeded == V2 libraries are in use on edx.org and stable.
  3. DECISION: We will not write back v2 keys until we are fully confident that the v2 experience will not be reverted.

2023-10-16 meeting

From Connor:

PTO was much needed.
We’ve polished up the migration PRs after hitting some minor snarls with running on more-production environments (sandbox) and having more complex states to validate, but that has been helpful.
I need a little more time to get the V2 library block editor off the ground, but it is going well. The backend work should be ready to review though?
Who should be in charge of merging the V2 libs in xblocks pr? I’m happy to once tests are green.
TODO: we should make a matching runbook for Tutor deployments because they will have slightly different steps, right?
- Configuration
- Running management commands.
To do in the next two weeks
- Apply feedback from sandbox
  - TNL will handle this (probably Ray, etc)
- merge V2 libs in xblocks PR
  - Connor and Kyle both will check locally that this doesn’t break V1
  - linting too obvi (Kyle)
  - Then we can merge
- Static reference support
  - Backend block changes
    - This are mostly good to go
    - https://github.com/openedx/edx-platform/pull/33511
  - New frontend-lib-content-components editor for static reference
    - edx-platform work required
    - A non-trivial amount of FE work has been done, more required
Thinking about migration
- Dave’s comment: https://github.com/openedx/edx-platform/pull/33231#discussion_r1326351956
- Connor will look into the current implementation + block transformers.

2023-10-2 meeting

From Connor:

TL;DR: working on one library block, I’m happy with where the “V2 libraries“ PR is at, and am looking for a final review.

Status of "user tasks" plugin in LMS:
- I tried moving the required code to the following places:
  - cms/djangoapps/contenstore/libraries/tasks.py
    - This didn’t work because there are several circular dependencies. Also, this is a “V1“ location, so it doesn’t make sense to keep the code here.
  - openedx/core/djangoapps/content_libraries/tasks.py
    - This did not remove the requirement for “user tasks“ to be installed in the LMS. I’m not sure why that would be happening, as we can see that content libraries is installed in both the cms and lms. I’m not an expert on celery, but I feel like the amount of required work to fix this outweighs the potential damage of having user tasks in the LMS (which from a conceptual standpoint seems fine to me).
- Consequently, as this code should live with openedx/content_libraries, I have left it there.
- Kyle is worried about the bug that this implies, will look into this
Status of “One Xblock work“:
- Status:
  - I have gotten the CMS side working, and now have run into a few odd errors with the library_content block in the student view. Also will need to probably make sure v1 library editing still works. Mostly confident in my ability to get this done, but might have some opportunities to pair this week.
  - Still will need to write a bunch of unit tests.
- Minor schema tweaks:
  - I represented “Shuffle“ and “Manual“ as a 2x2 matrix, and assigned each element of the matrix as an entry in the “Mode“ dropdown. This allows us to represent the states in english to the author in a clean way that makes sense. It also makes the backend logic better separated, as we can have well-delineated selection modes for each one.
  - I added a “candidates“ field which holds that block ids selected by the user for “manual selection.“ This is because unselected children still need to be represented in the children object, for display on the selection screen.
- Connor will make a V2 editor for this

2023-09-27 meeting

For discussion, From the ADR:

2023-09-11 meeting

Since last time

Connor is on call, kyle is filling this out:https://openedx.atlassian.net/wiki/spaces/COMM/pages/edit-v2/3861413889

Plan

Connor works on library_content v2 PR
Kyle fills out https://openedx.atlassian.net/wiki/spaces/COMM/pages/edit-v2/3861413889 and reviews PR
Connor will be able to use ^ to test PR

2023-09-05 meeting

Prioritization decision
- v2 randomized reference in library_content: first priority. Release blocker.
- v2 specific reference in library_content: immediate next priority. Want in release (one xblock type) but may release without it.
- library->library import/export: next priority, might miss release.
[kyle] Recommendations for library reference.
- Don’t release library_sourced. Instead, add non-randomized reference support to library_content. (We’ve already talked about this, but I’m doubling down :)
  - Tech reasoning: the library_content block is already hard enough to understand. Adding a second block type, with overlapping features, will only make that worse, especially when the two implementations begin to drift, double-especially during the migration period where each block would have two backends (v1 and v2).
  - Product reasoning: It gives us less flexibility with the end-user experience. With two blocks on the backend, we would have to show it as two different component types in the UI. With one unified block on the backend, we can decide whether to present it to the course author as one component type or two, and we can change that decision at any point.
  - This change will have to be made before initial release. Releasing the library_sourced block means supporting it potentially forever.
- Persist default library block settings in course’s OLX (“defaults.json”) so that library_content blocks work on any instance, regardless of whether the source library is there.
  - This addresses the fact that, today, importing a course with a library_content block into an instance without the source library will result in a semi-broken library_content block. Specifically, any library-defined settings, such as title, will be missing from library content.
  - This change is backwards-compatible. It could be made before, as part of, or after the initial V2 release.
  - Note: Not a 2U priority. Perhaps Axim/funded/CC followup.
- [Connor] as the person who posed this question in the first place you have a backer in me.
[kyle] Outstanding technical Qs
- UPDATE: I moved the questions here: Library Content Component User Stories
- Haven’t figured out yet how v1 libraries figure out that an update is available, and whether/how that works in v2 libraries.
- When v1 lib updates are pulled into a library_content block, any overrides to settings will survive, but any updates to content will be clobbered. Still working on confirming that this is the case with V2 libraries (Connor Haugh , have you confirmed this?). I’m concerned that it could differ because in V1, courses shared block content definitions with libraries; but in v2, content definitions are copied imported from blockstore to modulestore.
- Does the update-from-latest thing work with the v2 library_content block?
- Does the v2 library export need to change at all, or could it be identical to the v1 export?
[kyle] Product Qs
- Mass course->library export: what state is that in? is it something we could eventually polish up into a nice UI for authors?
- Assuming that we release a single block for randomized and non-randomized content, and assuming that subsets of libraries can be selected (like they can in library_sourced today), does that mean that we get “randomized library subset” as a new feature?
  - Nice to have, not for initial release though.
  - Note: CAPA type theoretically is a version of this, but doesn’t seem to be working on edx.org.
    - JK Connor just fixed that

2023-08-28

Connor update:

working branch at https://github.com/openedx/edx-platform/pull/33057
- Needs: test, lint
- additional feature is WIP: switch (waffle) between v1 + v2 xblocks being shown as editor options in courses (adding waffle flag) Should be a follow-on PR
- Learning: library_source and library_content block should be the same block.
  - Why weren’t they? Are xblock field migrations taxing?
  - How do we think about the pros/cons of merging them?
- It’s a big PR with a lot of commits, some more atomic than others. My review plan is this:
  - TNL internal: create a sandbox for UAT
  - Braden, Dave, Kyle, and Bernie: take a glance at the code.
    - Dave + kyle will look into this ASAP.
    - Do we want to split it up?

Kyle Update:

Building understanding of 2020 library incident
V1->V2 library import - Still working on it
Talking with Dave on eventual Learning Core migration, and what to do (or not to do) now in order to make that less painful.

V1->V2 rollout https://2u-internal.atlassian.net/wiki/spaces/~60ba5f9548549e0069510f53/pages/494633048/Draft+Libraries+v1+-+v2+runbook

Copy step: copy all v1 into v2 libraries
Don’t delete V1 libraries until migration clearly successful
- A few days/weeks

2023-08-21

Connor update
- Re-merging changes that RG had made back in Dec 2022
- Improvements/source to library sourced block.
The three things Kyle will work on
- v1->v2 library import : Now
- v2->v2 library import : Soon (Dave’s returning from PTO)
- importing v2 library content into v1 courses : Now (Kyle will reach out to learn more about the bug that happened)
Shared #content-libraries-v2 channel: can we get everyone working on v2 libraries to use it?
- Done.
Importing courses with library content: Let’s not recreate this situation.
- Doc I can’t see with more details
- Take aways:
  - We should know went wrong here, in order to prevent future problems.
  - We should test this use case.
State of other authoring work in flight
- “import/export rest api”: That’s a thing you guys are working on, right? How does it differ from the current ajax import handler?
  - It actually isn’t based on import/export. Instead it simply offers a set of existing content apis as “public” apis which can then be interacted with by trusted parties.
- ENABLE_NEW_STUDIO_* toggles in contentstore - what’s enabled now? where can I follow along with RG’s Studio MFE efforts?
  - I’ll ask kristin for the status spreadsheet
    - current status sheet not public, but currently all waffle flags are off in prod on edx.org, with some in internal/external beta.
Less important for me right now (not import-related)
- Blockstore library-ification
  - Why reuse existing Blockstore RDS rather than just running migrations to create new Blockstore tables in the edxapp db?
    - running migrations, SRE choice.
- Merging course authoring and library authoring mfes - do you think it’ll happen?
  - I certainly want it to, and TNL is attempting to earn some leeway to work on ownership.
What connor is working on DRAFT PR FOR LIBRAY SOURCE xblockhttps://github.com/openedx/edx-platform/pull/33057