Kyle/T&L Content Libraries V2 syncs (2023)
2023-1-18 Handoff meeting
Ray:
https://github.com/openedx/frontend-lib-content-components/actions/runs/7546505259/job/20544397615
has lint + unit test level work left.
Specs/Designs
V1->V2 libraries runbook (I know this is defunct) https://2u-internal.atlassian.net/wiki/spaces/TNL/pages/494633048
Feature Tickets
Library to Library Import+Export
AC:
v2 Library Authoring experience must support import of a package that contains: Either a v1 library or a v2 library
v2 Library Authoring experience must support export of a package that contains: a v2 library
NOT IN SCOPE: exporting v2 libs as v1 libs is not scope,
NOT IN SCOPE: importing existing v2 libs that the user can read is also not in scope.
Discovery sub-task questions:
What is the file format of v2 library packages?
What info should be in and out of the package?
How do library-specific permissions work with exports and imports?
Do we need to call out that import is from v1 or v2 to the user?
I see several tasks here:
Determine a format/spec for v2 lib exports.
Create the library export page
Write the export celery task
Write the import from v2 celery task
add the logic to handle imports of courses, v1 and v2 libs.
[Post Rollout] Bulk Delete of Content Libraries
Mockups needed here, as this is not a feature that currently exists.
[Post-Rollout] Update the “Advanced” Component access to be for developers only.
This is blocked by the addition of the new editors inside the LA MFE.
ACs:
We want developers to be able to use the old editing experience developed by opencraft
We’re not sure why though.
When the dev has this mode enabled, clicking edit on a block brings them to the old experience.
Technical notes:
Use an MFE level feature flag to enable/disable this behavior.
[POST ROLLOUT] Remove routing code & remove waffle switch
We previously (in this epic) removed the need to have a waffle flag which controls whether blockstore is an app or service.
ACs:
The Waffle Flag blockstore.use_blockstore_app_api should be removed, so that the app api is always used.
Conditionals which use that waffle flag should no longer check the waffle flag (most will be completely removed).
including, but not limited to:
the instances of the waffle flag’s setting value should be removed from Django admin.
Notes:
Test that blockstore CRUD for libraries still works!
copy paste does not exist in V2 (non blocker for release)
text templates are not available in V2 (not a blocker for release)
text templates for the text xblock like “announcement“ are not available in V2. Will require some feature development.
https://github.com/openedx/frontend-app-library-authoring/issues/81
Small Fixes to Course Import into Libraries Page
On the course import into libraries page:
Make the header display the header info for the current library
Make the filtering features conditional on a feature flag, set that feature flag to default to false.
Bug tickets
https://2u-internal.atlassian.net/browse/TNL-11365 (Will be resolved by EOD for edx.org, but will also need a fix)
MIT encountering gateway timeout after course import failure when trying to edit library content in course-v1:MITx+CTL.SC1x_1+1T2024
MIT’s randomized content blocks are not loading when they try to edit them in the course at course-v1:MITx+CTL.SC1x_1+1T2024.
Users are timing out when loading the studio handler. We can see this is because it is calling bundleversion select a bunch
turns out, no bundles exist on prod atm (and no v2 libs)
So we need to figure out how gracefully return no libraries when there are no V2 libraries, because this breaks LCB editing.
Hovering over many links (e.g. in the "Libraries" list) do not show the URL that they link to in the browser's status bar. This is an a11y issue - they aren't "real" links; but they should be as they go to a different page.
The Library Details “Submitting” state does not disable actions.
The Library Details “Submitting” state is not ideal. Buttons are still clickable, and there is no spinner or graying out of the page, and no feedback is provided to the user.
Saw this when saving library details.
2023-1-8 meeting
Connor:
Now:
Import Bug (can you see this?: https://2u-internal.atlassian.net/browse/TNL-11339 )
Local attempt : succeeded
Stage attempt : succeeded
Prod Attempt: failed
I assume this is because only the matching library is on prod. This is good because it allows me to replicate locally.
going to follow this through the changed parts we added
timebox to today, then work on the implementaiton of turning off sync we suggested.
connor, search for
get_block_original_usage(...)
nope, nvm, this wouldn’t work with export/import when the source lib doesn’t exist on the target instance
for
loop it is
Ray’s got one last frontend bug
Next:
Get current branch to fully functional (lint, unit tests, round trip working with candidates)
Path forward:
in order to completely de-LCB children, Connor is going to create a LCB child → library child static conversion method.
Connor is going to timebox that effort to half a day.
If that fails, connor is going to remove editing canidacy from the “view“ page.
get_block_original_usage
()
Peter Pinch: V1 Libraries Page loading slowly
@Connor Haugh (Deactivated) to ask partner support
2023-12-18 meeting
Work In Flight:
Connor:
On-Call wrecked Bernie Jesper & Ray’s week last week. They are picking up where they left off pretty much. Bernie is adding code to blockstore to make bundles inspectable in django admin by more than superusers as an app-plugin (app-permissions type role).
I have one dumb mock thing to debug on: Library Redirects.
I have to wait on the react-based editor for: https://github.com/openedx/edx-platform/pull/33511
Kyle -- another pass
I need your final thumb on https://github.com/openedx/edx-platform/pull/33628 (super quick).
Connor To look at.
https://github.com/openedx/edx-platform/pull/33920 just needs to wait on me testing Braden’s fix on stage.
Kyle:
Fun with LibraryContentBlock export/import and duplication
Quick + Dirty:
Fail loudly (to users or to logs) (Silence the exception)
Default settings changed will be missing
OR Refeesh blocks from the library on duplicate (harmless) + import export (NOT harmless, could have big bugs).
My rec if you want to go down this path
Refresh children for duplication
copy-paste does this now (see Braden’s PR)
Let the defaults be missing for import/export
Moving forward:
I do not think I have the “decision makers” with me today because of vacation and some personal stuff for them. I do grok the intentions behind what 2U’s perspective is: “We need to be only working on things that help our bottom line in the short term.” Libraries' ability to do that is less clear than other TNL projects for 2024. Consequently, I sit in a moment of tumult, where I should be pivoting, but don’t want my good work to go to waste. Last week we talked about the need to figure that out. I would love to spend some time coming up with a plan here that we can take to both axim and 2u leaders and feel good about. In theory, If I’m still developing libraries stuff ~january 8th, there is a problem.
Increments:
New Library Block Editor, Specific Reference (doesn’t require V2).
Ray dependant at this point.
TNL will focus on this.
New Library Authoring Experience live on edx.org
v1s migrated & mapped to v2
The main piece of work left here is “get blockstore-backed CL by version.“ If you would rather work on dave’s Learning core backed storage, maybe I can hop on that specific bug this week?
New backing storage for V2 libraries.
2023-12-11 meeting
Work schedule:
2U OO 12/22-1/2 exclusive
Axim: Kyle TBD, but probably similar
What are folks up to
Ray → Editor work is wrapping up.
Jesper → local editing
Bernie → misc review, defense. fixed email bug
Put PR requests in such in #openedx-content-libraries-v2 channel
Keys & Learner State
two bugs
Upgrading an LCB's source lib from V1 to V2 might disassociate learner state, because it seems child use keys are derived differently between V1 and V2.
Open question: why is this donehttps://github.com/openedx/edx-platform/commit/eec72431419664385237cf1dfddfa44d19d610b9#diff-69a28742894c4b286b463a4baede1d8ad03ef4418779f73d8cd4ef42e8879687
What is the format for a block id for a block which is NOT imported from a v1 library?
Possible alternative solution:
Use a version of the v2 library import client which has the field “use_course_key_as_suffix“= True or something like that
In that case, don’t add the course key as an additional hash to the block id.
This would allow for a much easier fix to the
generate_block_key
function where we could, regardless of the v2 block id, blindly pass it into the derive_key function?LIB KEY MAPPING: library-v1:ORG+LIB <-> lib:ORG:LIB LIB BLOCK USAGE KEY MAPPING: block-v1:ORG+LIB+type@TYPE+block+BLOCK <-> lb:ORG:LIB:TYPE:BLOCK dest_usage_key = dest_course_key.make_usage_key( derive_key( lib_key_v2_to_v1(source_lib_key), source_lib_block_usage_key.block_id, dest_lcb_usage_key.block_id, ) )
Copy-paste followed by update-library-version might also disassociate learner state, because (I believe) copy-paste randomly generates child keys instead of deriving them.
I’m hoping Braden will take this on.
2023-12-04 meeting
The Cavalry is coming:
Going to bring on Jesper & Bernie for this sprint so we can wrap stuff up hopefully in the next two weeks.
Wanna pair on https://github.com/openedx/edx-platform/pull/33511/files review?
2023-11-27 meeting
capa_type filtering – just to confirm, we are leaving this out of the new library_content block editor for launch, right?
Yes
How’s the duplication bug (and import bug too, I think?) going? Need any help? -Kyle
Importing libraries
Import is working, but submitting an error message. This is I think because no courselike key is being applied to the function.
Prooobbably not the most recent PR, related to Course Authoring work.
Importing courses with library blocks
Haven’t tested in prod recently
No reports of issues
But this code overlaps with duplication code, so keep an eye out
Figuring out why duplication tasks fail has been trickier, however last night I determined that it was only duplications with overrides that failed. Going to add some logging there by eod.
Makes sense -Kyle
user_tasks is helpful but doesn’t have detailed errors
but we were seeing “Failed”
Connor is adding logging
old V1 editor for library_content takes a while to load
transient?
Are import tasks in progress? Spinner for editor loading (upgrade_and_sync) until load is complete.
even spins when no library sometimes??
theoretically we should only need to block children display, not settings editor
Kyle’s update: old lib versions in V2 - still working on this
Static reference
Connor is straightening out merge, then can incorporate feedback
Kyle needs to respond to some comments
2023-11-20 meeting
LETS GO WE FINALLY MERGED https://github.com/openedx/edx-platform/pull/33263 and it didn’t totally exploded (yet)
Priorities:
What is there to learn from stage migration? Do we put that over bug fixes like v2 duplication?
.
Things to keep in mind
Bug: duplication task failing. Affects:
duplication
course import?
Getting specific v2 library by version. Affects:
duplication
course import
Emails
All lib content syncing happens in a celery task now
@Connor Haugh (Deactivated) will fix extraneous emails ^
user_tasks
That celery task sends an email when any of the following happens:
picking a library
“Update Now” button
duplicating library blocks
What’s blocking testing the migration on stage?
Key mapping
(soft blocker: static reference Pr is the basis for this)
@Kyle McCormick review ^
Shoot for next week for stage migration
2023-11-13 meeting
status: The Library Block Editor is nearing completion.
this pr: https://github.com/openedx/edx-platform/pull/33263
apply fix from the bug connor found
up next
Review v1->v2 key mapping
Review static reference PR
Ray will review add-editor-to-edx-platform PR
will be done once editor is working
content_libraries test suite - turn it back on
If V2 fully enabled, then redirect V1 library links to V2
Library import/export
Import-library-from-course-content
functionally works
missing UI stuff like header, filtering
we should either build it out, or hide it
2023-11-06 meeting
https://github.com/openedx/edx-platform/pull/33263
wrap this up in an hour, then review & test.
merge tomorrow morning??
versioning of v2 libs
Why do we need versions for refresh children:
for post_import (pull in the requisite version from the xml doc)
duplicate functionality (pull in the version from the copy version.)
^ Kyle will do this as follow up to #33262
Status (TNL):
Library Authoring
Up next
Review v1->v2 key mapping
Review static reference PR
Ray will review add-editor-to-edx-platform PR
content_libraries test suite - turn it back on
If V2 fully enabled, then redirect V1 library links to V2
Library import/export
Import-library-from-course-content
functionally works
missing UI stuff like header, filtering
we should either build it out, or hide it
2023-10-30 meeting
Fallout from V2 in library xblocks.
Address Connor’s dupe code comment
Then merge (“soft rollout of V2 content”)
Improvements to Permissions + Roles
Brian S is fixing a library-authoring path issue in Tutor
Ray is working on new library editor
Kristin is working on library authoring UI/UX bugs
@Connor Haugh (Deactivated) Look into the “libraries list“ in xblock is amenable for merging.
Just FYI
Post-thanksviging tuesday (28th of Nov) for cutover event -- soft deadline
downtime-
2023-10-23 meeting
import/export tests?
need django settings to be correct for cms for devstack?
I might ask for some pairing on this.
[Kyle] After Tutor being broken thurs->friday, I finally have a working async test environment! I’m testing https://github.com/openedx/edx-platform/pull/33263 now. Let’s aim to merge first thing tomorrow (Tues) unless I find anything broken.
Migration strategies (https://github.com/openedx/edx-platform/pull/33231)
Modify keys in the database w/ mgmt command
Runtime mapping w/ waffle flag
should “Runtime mapping“ be persistent or not?
I think not, after some pros-cons:
pros:
we never have to repeat this operation
we may be able to remove the code as usage decreases.
cons:
manipulating code is bad.
This can always be done later, and we should only be writing keys back, when we are confident the migration succeeded.
Migration succeeded == V2 libraries are in use on edx.org and stable.
DECISION: We will not write back v2 keys until we are fully confident that the v2 experience will not be reverted.
2023-10-16 meeting
From Connor:
PTO was much needed.
We’ve polished up the migration PRs after hitting some minor snarls with running on more-production environments (sandbox) and having more complex states to validate, but that has been helpful.
I need a little more time to get the V2 library block editor off the ground, but it is going well. The backend work should be ready to review though?
Who should be in charge of merging the V2 libs in xblocks pr? I’m happy to once tests are green.
TODO: we should make a matching runbook for Tutor deployments because they will have slightly different steps, right?
Configuration
Running management commands.
To do in the next two weeks
Apply feedback from sandbox
TNL will handle this (probably Ray, etc)
merge V2 libs in xblocks PR
Connor and Kyle both will check locally that this doesn’t break V1
linting too obvi (Kyle)
Then we can merge
Static reference support
Backend block changes
This are mostly good to go
New frontend-lib-content-components editor for static reference
edx-platform work required
A non-trivial amount of FE work has been done, more required
Thinking about migration
Dave’s comment: https://github.com/openedx/edx-platform/pull/33231#discussion_r1326351956
Connor will look into the current implementation + block transformers.
2023-10-2 meeting
From Connor:
TL;DR: working on one library block, I’m happy with where the “V2 libraries“ PR is at, and am looking for a final review.
Status of "user tasks" plugin in LMS:
I tried moving the required code to the following places:
cms/djangoapps/contenstore/libraries/tasks.py
This didn’t work because there are several circular dependencies. Also, this is a “V1“ location, so it doesn’t make sense to keep the code here.
openedx/core/djangoapps/content_libraries/tasks.py
This did not remove the requirement for “user tasks“ to be installed in the LMS. I’m not sure why that would be happening, as we can see that content libraries is installed in both the cms and lms. I’m not an expert on celery, but I feel like the amount of required work to fix this outweighs the potential damage of having user tasks in the LMS (which from a conceptual standpoint seems fine to me).
Consequently, as this code should live with openedx/content_libraries, I have left it there.
Kyle is worried about the bug that this implies, will look into this
Status of “One Xblock work“:
Status:
I have gotten the CMS side working, and now have run into a few odd errors with the library_content block in the student view. Also will need to probably make sure v1 library editing still works. Mostly confident in my ability to get this done, but might have some opportunities to pair this week.
Still will need to write a bunch of unit tests.
Minor schema tweaks:
I represented “Shuffle“ and “Manual“ as a 2x2 matrix, and assigned each element of the matrix as an entry in the “Mode“ dropdown. This allows us to represent the states in english to the author in a clean way that makes sense. It also makes the backend logic better separated, as we can have well-delineated selection modes for each one.
I added a “candidates“ field which holds that block ids selected by the user for “manual selection.“ This is because unselected children still need to be represented in the children object, for display on the selection screen.
Connor will make a V2 editor for this
2023-09-27 meeting
For discussion, From the ADR:
https://github.com/openedx/edx-platform/pull/33231#discussion_r1327689952
https://github.com/openedx/edx-platform/pull/33231#discussion_r1327692539
2023-09-11 meeting
Since last time
Connor is on call, kyle is filling this out:https://openedx.atlassian.net/wiki/spaces/COMM/pages/edit-v2/3861413889
Plan
Connor works on library_content v2 PR
Kyle fills out https://openedx.atlassian.net/wiki/spaces/COMM/pages/edit-v2/3861413889 and reviews PR
Connor will be able to use ^ to test PR
2023-09-05 meeting
Prioritization decision
v2 randomized reference in library_content: first priority. Release blocker.
v2 specific reference in library_content: immediate next priority. Want in release (one xblock type) but may release without it.
library->library import/export: next priority, might miss release.
[kyle] Recommendations for library reference.
Don’t release library_sourced. Instead, add non-randomized reference support to library_content. (We’ve already talked about this, but I’m doubling down :)
Tech reasoning: the library_content block is already hard enough to understand. Adding a second block type, with overlapping features, will only make that worse, especially when the two implementations begin to drift, double-especially during the migration period where each block would have two backends (v1 and v2).
Product reasoning: It gives us less flexibility with the end-user experience. With two blocks on the backend, we would have to show it as two different component types in the UI. With one unified block on the backend, we can decide whether to present it to the course author as one component type or two, and we can change that decision at any point.
This change will have to be made before initial release. Releasing the library_sourced block means supporting it potentially forever.
Persist default library block settings in course’s OLX (“defaults.json”) so that library_content blocks work on any instance, regardless of whether the source library is there.
This addresses the fact that, today, importing a course with a library_content block into an instance without the source library will result in a semi-broken library_content block. Specifically, any library-defined settings, such as title, will be missing from library content.
This change is backwards-compatible. It could be made before, as part of, or after the initial V2 release.
Note: Not a 2U priority. Perhaps Axim/funded/CC followup.
[Connor] as the person who posed this question in the first place you have a backer in me.
[kyle] Outstanding technical Qs
UPDATE: I moved the questions here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3861413889
Haven’t figured out yet how v1 libraries figure out that an update is available, and whether/how that works in v2 libraries.
When v1 lib updates are pulled into a library_content block, any overrides to settings will survive, but any updates to content will be clobbered. Still working on confirming that this is the case with V2 libraries (@Connor Haugh (Deactivated) , have you confirmed this?). I’m concerned tha