DEPR Meeting Notes (2021)
- Nimisha Asthagiri (Deactivated)
- Diana Huang
- Jeremy Bowman (Deactivated)
Action Items
Agenda Backlog
More documentation around deprecation
How do we create a DEPR pipeline?
Review the Toggle Removal report
toggles are a good first step
anything we should add to the report that we’re not capturing today?
How do we engage other squads to help out?
How do we make it visible to squads what their DEPR status?
What are the organization code smells that might indicate DEPR needs?
How do we prioritize what to work on?
Metrics for analyzing the impact of DEPR work
Agenda
Past Agenda
Agenda (1/5/21)
Update / Review DRAFT - Koa Platform Release Notes
@Diana Huang to add DEPR tickets to the notes - can review with group next meeting
Site Configuration & White Label
update: it seems like the Open edX community isn’t interested in supporting this functionality
Can we talk to enterprise about removing these with assistance from DEPR?
We have code, templates, and data that all should be removed, but can be removed separately
talk to the service owners about removing site-specific code as well. does it make sense to deprecate this directly?
exclude: theming for now
Check in with Revenue team about https://openedx.atlassian.net/browse/DEPR-90
2021 Goals
White Label and Site Configuration code
Action Item: write a DEPR ticket to gauge community reactions to the deprecation, offer up eduNext model as an alternative - @Nimisha Asthagiri (Deactivated)
Action Item: communicate with enterprise team about future plans to deprecate white label and get buy in and support - @Diana Huang
Microfrontend/Frontend code
Koa has rudimentary support for MFEs
Lilac has plans to streamline MFE
How do we want to handle this removal of code?
ID verification might be a good place to start?
has a DEPR ticket
Learner profiles
start anything currently ‘Accepted’
Old Mongo
blocked by old mongo courses on edge
waiting for product to assess the state of edge
should be able to move forward in it
Phase out configuration?
could be a big project goal
looking into replacements
write up a project plan on this once we get further on it
XServer
In progress
EdxRestApiClient
review the list of services
focus on edx-platform, credentials, discovery - others are in the process of being deprecated, potentially
XModule → XBlock
DEPR pipeline
Can we provide a query or something to the eng directors regarding the engineering ops review?
Focus on it next quarter?
Agenda (1/19/21) -
XServer
This just needs a little bit of verification left
Just want to make sure we’re not breaking any courses unintentionally
import shims
reviewing the last few PR that are importing the wrong way
going to be put into #slash-n-burn channel for group to review
woohoo!
updates
XModule → XBlock conversions: XModule to XBlock ConversionUNDEFINED
Old Mongo
getting rid of courses on edx.org in Q3
looking at what is on edge
part of the TNL OGSPs, so looking good
white label + site configuration
still need to talk to enterprise about the future of this work
still used in other repos to split by sites
can we make a separate process for external deprecations? - where external teams are responsible for doing the deprecation work
configuration repo
dependent on getting devstack and other docker containers off of being reliant on this
Open edX looking for alternatives right not
BTR - hoping Lilac will be the last Open edX release to use configuration
Open edX is only half of the solution - internal edX usages like sandboxes will also have to move off of it
Agenda (2/2/21)
From Robert: I have a proposed slash-n-burn activity that goes along with our upcoming Toggle Doc-a-thon, but it concerns data clean-up and not code clean-up. I suspect that our Toggle State Report spreadsheet has many rows due old waffle data in our environments that is no longer used in code. The spreadsheet would be even more useful if this data could be cleaned up.
We’d need to confirm what data should be cleaned up.
We’d need to decide if clean-up would use manual deletion, or if we’d add a migration to hit all environments.
Could ARCHBOM be responsible for doing this cleanup?
Impact isn’t clear as of yet
Bring this back to ARCHBOM for discussion
How do we make the toggle report up to date and actionable?
Review potential DEPR work suggested by Robert in #slash-n-burn after the doc-a-thon
USE_MICROSITES: https://github.com/search?q=org%3Aedx+USE_MICROSITES&type=code
ENABLE_STUDENT_NOTES: https://github.com/search?q=org%3Aedx+ENABLE_STUDENT_NOTES&type=code
XBLOCK_FS_STORAGE_BUCKET -
unclear if this is okay to be removedWe should make a DEPR ticket for this setting to ensure no external XBlocks are using
Look at new DEPR-140 for UserProfile.allow_certificates:
Reach out to Justin to remind him of the DEPR process around announcement
Also ensure best practices around DB migrations are followed
EdxRestApi update - https://openedx.atlassian.net/browse/DEPR-21
did an example one with e2e-tests
should we disable
credentials PR is a good example for what this looks like:
https://openedx.atlassian.net/browse/DEPR-108
Follow up with Marco about this work - @Sarina Canelake (Do Not Use) (Deactivated)
Account pages and Learner Profile pages - https://openedx.atlassian.net/browse/DEPR-17
Unblocked - can be worked on by the team
Ecommerce - https://openedx.atlassian.net/browse/DEPR-42
Maybe hold off until decision made
https://openedx.atlassian.net/browse/DEPR-120
Check in with Simon about this
Everything else: ping reporters to ask them to send out announcements to the community to move the process forward
Hackathon
Solicit suggestions from the Mobile team about DEPR related things
Repos that can be DEPR-ed?
Agenda (2/16/21)
Hackathon readout
need to refactor the configuration and usage of edx-enterprise’s api client to use the single edx-enterprise user
need a review for course-discovery’s refactoring
Check in on existing streams
some things moved into accepted
pycontracts DEPR ticket to be written - @Diana Huang
Hack(?)
Agenda (3/2/21)
Check-In: https://openedx.atlassian.net/browse/DEPR-142
needs review: https://github.com/edx/edx-platform/pull/26772
@Diana Huang needs to announce pycontracts DEPR ticket
https://openedx.atlassian.net/browse/DEPR-58 update
still being held up by Edge content
okay to make it safe(?)
need to handle the edge cases
handle edge cases where course information is no longer available
will make it possible to ‘retire’ or ‘delete’ courses
https://openedx.atlassian.net/browse/DEPR-124
will be removed at the end of the week
XModule → XBlock conversion continuing
no DEPR ticket because backwards compatible
getting closer! actively worked on by OpenCraft
after this, still need to remove all the underlying peak/plateau confusion in the modulestore code
https://openedx.atlassian.net/browse/DEPR-150
need to make a decision about django admin login before announcement
@Feanil Patel (Deactivated) to do this
Agenda (3/16/21)
Last XModule removed from edx-platform!
Set up pinata celebration?
“One idea: When we retired Vertica recently, we each wrote Vertica tributes that were poems/odes/prose/limericks - and bbeggs read them all. That could be expanded to add PPT/Sheets presentations? And maybe fanfiction? Songs?” - Data Engineering
Round robin reading of odes
Mail everyone in the company one (1) piece of candy
Origami pinatas that people can make it at home
Ask Usman to stream himself destroying a pinata
Send him a jacket at least!
Send something out digitally for people to do themselves
@Sarina Canelake (Do Not Use) (Deactivated) to organize this.
Removal dates for DEPR tickets: can we just say this is the acceptance date?
Yup, we can do this
@Diana Huang to make these changes to the OEP
https://openedx.atlassian.net/browse/DEPR-108
In progress, getting close
https://openedx.atlassian.net/browse/DEPR-123
Community is not comfortable with replacing front ends with MFEs
No Tutor plugins or single-install support yet
Planning for Lilac to ship with a few MFEs as a sort of test case
Can we measure the metric of DEPR work?
Community contributions - @Sarina Canelake (Do Not Use) (Deactivated)
Contributions to front end code from anyone
Recouped cost of upgrade - @Jeremy Bowman (Deactivated) @Feanil Patel (Deactivated)
Lines of code(?)
Repos(?)
Dependencies
Rough estimate of this work
Coverage metrics
Code complexity - @Dave Ormsbee (Deactivated)
Run automated tools before and after
Measure happiness with code before and after
Could use 2 question survey to see how people
Flavor complexity
Developer effectiveness
Multiple implementations is confusing
Import tree size of edx-platform @Michael Terry (Deactivated)
Automated tools for this
proxy for code comprehensibility
How many other things do you need to understand to understand this file
Lines of code @Diana Huang
Talk to Ned about this - instrumented for OSPR
Meta discussion
How do we track non-DEPR work for the group?
Use wiki with checkboxes for now
Should we make another project?
Github projects
Agenda (3/30/21) -
Discuss metrics discovery
lines of code metric
not available easily through existing Github or Jira interfaces
still need to discuss with Ned about OSPRs
recouped cost of upgrade
discovery doc: https://docs.google.com/document/d/1GqzWcrm3rY8v9W6bqXa0hXBJk7VKx8mSGxKETwzeNjU/edit?usp=sharing
10 dev weeks/year per small service that we need to support
e.g. notifier, XServer
next up:
small python package/library that we need to support
JS package/library that we need to support
maybe turn into a spreadsheet later?
Focus on fewer metrics
Cost of upgrade
Lines of code - will give us a multiplier for cost of upgrade as well
Code complexity - is there a quick win we can get here?
Remove XQueue Watcher? Only 1 Partner uses it: MIT only for 7.QBWx and 6.00x courses. What is the purpose of this feature? Can it be either deprecated or deemed to be useful for other partners?
caused problems with the 20.04 upgrade
if we deprecate this, can we also deprecate XQueue?
need to make sure this is a separate DEPR effort
we need a followup discussion (maybe with TNL? Ed? Feanil? Peter Pinch?) about what we need to do to get rid of this
This seems like a reasonable way of handling ‘scope creep’ for DEPR work
For anything larger, we would want to create a separate DEPR ticket for the larger thing getting removed.
Community involvement - core committer commitment/participation
FYI - had a meeting with how we can invite core committers to what are currently internal-only meetings
DEPR working group is a good possible venue for inviting them
need to make sure that we have appropriate understanding about what’s internal and what’s not
maybe one meeting a month will be open to the community
Agenda (5/14/21) -
Shoppingcart removal
Final removal of shoppingcart djangoapp
DEPR metrics
what about the intangible effects of deprecation work and tech debt?
developer effectiveness
reliance on institutional memory
developer sanity/retention
AJ's question about DEPR project access
Look at Jira permissions on the DEPR project and verify they work correctly
New Jira vs. Old Jira confusion
Now we know where the permissions tab is
Agenda (6/8/21) -
Scheduling/recruitment: what works for people?
Should we recruit new people?
We have projects in flight, and without members, they don’t get done.
Review of DEPR board
fixed states for a few tickets
Hacking
focusing on removing libraries to reduce Django 3.2 upgrade effort
Agenda (6/22/21) -
Review Django 3.2 upgrade DEPR work
Hacking
pycontracts
ajax_handler work
Agenda (7/6/21) -
Devstack Deprecations from Arch-BOM
sync
NFS
context: https://github.com/edx/devstack/pull/465#issuecomment-872483489
It doesn’t seem like the community needs it
The slow filesystem backend (osxfs) that this was added to work around was replaced by the (usually) faster gRPC-FUSE in Docker Desktop Community 2.4.0.0 on 2020-09-30
https://docs.docker.com/docker-for-mac/previous-versions/#docker-desktop-community-2400
https://github.com/docker/for-mac/issues/1592#issuecomment-864453273
https://github.com/docker/for-mac/issues/5402
Add this as a subtask/part of DEPR ticket with this as an explanation
This is definitely superior to NFS, and all docker installations in the last 9 months should already have it by default.
pycontracts
now gone from edx-platform!
Review of DEPR board
https://openedx.atlassian.net/browse/DEPR-108 - should see if we can move forward on this
https://openedx.atlassian.net/browse/DEPR-59 - should be able to hack on this if we have time for it
Reach out to community about block storage tickets
DEPR working group can support what is needed to get these into Accepted
Could we make a blended project out of cleaning up DEPR tickets without resources?
Our current preferred providers are at capacity, could make a post to Discourse about who is interested in taking on work.
Agenda (7/20/3221) -
Devstack DEPR tickets next steps
Can Arch-BOM take on this work?
Will put on the Arch-BOM team’s backlog
Check in on making more DEPR blended projects
can we check in with preferred providers?
can we make a DEPR blended project
process: https://docs.google.com/document/d/1evXQEJcIIvs0AQ27ca7RMongZaVQNq0Th2RdyFV6hDY/edit
add specific tickets to this project
can do a lot of smaller tickets
candidates:
find, deprecate, and remove deprecated toggles
DEPR group can find and provide the list of toggles
ask for a fixed number of hours to work on something (500 is proposed)
fill that time with DEPR work that we would like done in a blended fashion
use DEPR time to review PRs
Stats on ticket creation - who is making them?
Agenda (8/3/21) -
DEPR Blended
@Diana Huang still needs to write a brief
Will collab with @Sarina Canelake (Do Not Use) (Deactivated) on this brief
Possibility of pitching different briefs to different providers
Give a ranking to the providers of projects
Within EdxRestApiClient project, we could just use contagion as a ranking criteria
What do we want to get rid of before the next Open edX release?
other candidate possibilities:
site configuration code in edx-platform - also needs a DEPR ticket
https://openedx.atlassian.net/browse/DEPR-153
blocker is that NewRelic access to see the link. is there a way to get them this data without needing NewRelic? need to check with legal.
Taking advantage of the updates to engineering pathways
Can we recruit?
Discussion amongst the engineering managers about the practical implications of this change upcoming
Next engineering all-hands
What could we give to a newer person that would not be too intimidating?
Toggles removal was an option
https://openedx.atlassian.net/browse/DEPR-151 - these are just aliases
https://openedx.atlassian.net/browse/DEPR-53 - hopefully small
We could also do a mentored pair-programming exercise to walk them through the issues
Agenda (8/17/21)
Blended Brief(s)
separate ones for site configuration + general bucket of DEPR work
briefs in progress, getting close to being in shape to be submitted
Engineering All-Hands
have some time to ask for recruits
how should we sell it?
learn about unfamiliar parts of the codebase
delete things so you never have to work on them
get to meet/work people in other parts of the organization, including more senior folks
Agenda (8/31/21)
Welcome to the new folks!
Blended project updates
site configuration
general DEPR work
Can use the Paragon Blended Brief as an example
https://openedx.atlassian.net/browse/DEPR-163
Help define how the community can continue to use entrance exams while removing it from edx-platform
Next Steps
Reach out to Ken/TNL first, to make sure we extend the Acceptance Date @Diana Huang
Find out what other blockers exist for Learning MFE
proctoring?
If there are others, we can delay until they are dealt with
Reach out to Appsembler to see if they can extract this from edx-platform @Nimisha Asthagiri (Deactivated)
Reach out to other preferred providers in #core-committers @Nimisha Asthagiri (Deactivated)
Elasticsearch vs OpenSearch @Diana Huang
propose OpenSearch as a replacement for Elasticsearch
would like to write a DEPR ticket to get input from community early
Interests and Contributions of New Members
configuration removal (SRE assistance) - https://openedx.atlassian.net/browse/DEPR-122
ecommerce removal/replacement (Purchase/Revenue assistance)
Agenda (9/28/21) -
Ecommerce removal/replacement
ticket created
very early in cycle - wanted to get community feedback early
focused on moving from Cybersource to Stripe
creating a new app to help route information between processors(?)
django-payments library might be useful
Other options: https://djangopackages.org/grids/g/payment-processing/
no community feedback yet
OpenSearch DEPR ticket
Early feedback from the community:
Need to announce in Slack/on Discuss, etc for further feedback
Remove Jenkins jobs
We are moving things to Github actions
Can’t use the Github-hosted runners, need to spin up self-hosted runners with SRE support
RequireJS
When can we move us off of Require to Webpack
May need to be a FEDx-led project to help fill in expertise gaps
Can we make this a Blended dev project?
@Diana Huang reach out to the FEDx group about next steps
Hacking time
Agenda (10/26/21) -
DEPR ticket for Elasticsearch announced
Good question: can we get rid of ES/ES-alikes from Open edX altogether?
It’s a resource hog on installations which is more obvious to the community than it is to http://edx.org
We don’t see it because AWS handles it for us
Proposed updates to DEPR OEP
Agenda (11/9/21) -
Discovery around removing ES from our services
Insights/Analytics already did this analysis:Do We Need Elasticsearch in Analytics Data API and Insights?UNDEFINED
Ecommerce update, maybe?
DEPR in the time of non-profits
Making DEPR meetings public
Learning how to be public first vs. private first is going to be different
We (2U) probably won’t own this in the future, but would still have to be driven by the folks as part of this working group and at 2U.
Would like to see more involvement from the community in DEPR process, proposing, etc.
Hopes and dreams: Can we get more visibility into DEPR and attention from the internal folks? Should we communicate these internally as well as externally?
Agenda (11/23/21) -
https://github.com/jazkarta/edx-jsme - Nobody really maintains this anymore and we’d like to deprecate it, Peter Pinch has offered to help
Can talk to MIT when we open this up more directly
edx-zoom also being deprecated, can we bundle the work of deprecating them together?
This might be more tricky due to the JS inserted into every capa problem.
lms/djangoapps/dashboard has only contained 2 ADRs for about a year after its removal, can those be removed now?
Answer seems like ‘yes’ - @Michael Terry (Deactivated) to open PR
Blended Projects
‘Bucket’ project needs to find contact to move onto the next step
Talk to Marco about this
‘Site Configuration’ project was owned by Sarina, who moved over to TCRIL
Would need to find out from her what the status of this is.
Steps to open this up to the community
Discourse thread
Posts in Open edX Slack
Move meeting to Zoom
Create a new calendar for DEPR and can send invite to the calendar
Ensure wiki page is visible to community
Mention in the contributors meeting (happens every two weeks)
Agenda (12/7/21) -
First public meeting!
What can we work on next?
Review DEPR board
Make sure https://openedx.atlassian.net/wiki/spaces/~151054611/pages/2442395936 gets moved to a public space so community can access it and add to https://openedx.atlassian.net/browse/DEPR-108
Talk with FedX folks/djoy about timeline around non-MFE frontends
https://openedx.atlassian.net/browse/DEPR-163 - can we investigate how other Open Source projects handle this use case (i.e. large site/maintainers don’t use it anymore but others do)?
https://github.com/jazkarta/edx-jsme - What support can we provide to get this removed?
only used by one edX course
need to write up a DEPR ticket and announce it - @Peter Pinch