Versions Compared

Old Version 76

changes.mady.by.user Jeremy Bowman

Saved on

compared with

New Version Current

changes.mady.by.user Binod Pant

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • [ideation] (David J) What do we need to change in Tutor to make it minimally useful for edX developers to start using?

  • [quest] (David J) What is paver used for today? Corollary, what is paver?

    • Paver is a python library for creating tasks to be executed from the command line

    • Runs unit tests, building static assets, maintenance tasks, in edx-platform.  Never got used anywhere else.

    • Still technically maintained.  Started moving away from it and running the underlying commands directly.  

    • We did not build paver.

      • There's two parts of this:  

        • We didn't make paver, it's someone else's thing.  

        • We did write a lot of code to run things under paver.  It may look like we made it because we wrote so much on top of it.

    • Chain of thought: we need a task runner, make files are default, messy and shell based, want to do something in python, lets use paver.  Wait, this is complicated, lets go back to using Makefiles.

    • Makefiles and paver give you a way of saying "this command needs these other commands first"

      • Django management commands don't do that.  

      • The appeal was complex dependent-task management ("Only run this thing once even if several commands need it")

      • Problem with Make: needs to be implemented as shell scripts

    • Paver lets you use python files instead of shell scripts.

    • When the project was first starting out, we wanted anyone to be able to run it anywhere.  A lot of initial goodwill - as the reality of that set in, we decided supporting windows is hard and not valuable for the amount of investment we'd have to do to keep it working.  Usage of paver stemmed out of that, but then getting back out became too big for any team to take on.

  • [quest] Enterprise team is looking to best leverage feature toggles abilities to do more flexible (per customer or per subscription) toggles of features.A lot of these are done via config models right now. I wanted to get a base understanding of what best practices are in this area (edx-toggles read gave me words like: Waffle, Django setting, SettingToggle). Also, want to brainstorm on how can frontends best leverage these settings? Prior art? link to edx-toggles

2021-12-01

  • [quest/ideation] (Diana) - Devstack data project - who has a good use case that we should tackle?

    • Have a prototype that will load data into devstack

    • Arch-BOM were hunting around for a team with a good use case

    • Reach out to #arch-bom if you have a use case

  • [Ideation] (Justin Lapierre) How to set up a course/configuration in Devstack for QA in a reliable, reproducible way 

  • [ideation] (David J) As the balance of edX devs to core contributors shifts in the future, how do we think that affects the risk profile of our CI/CD pipelines deploying off of master, and what might need to change? :eyes:

    • "Support not changing things until there's a problem"

    • We may not be the only people deploying from master - what if _we_ break someone else's systems?

    • SWG perspective - how do we do security releases reliably?

    • How do we know we're in a more risky situation?

      • Added complexity of pre-empting this problem is very expensive, so wait on the RCAs

    • "Eventual trouble is probably inevitable", we don't know what the problem will be, but we may be able to game out how we might deal with the problem.  There's no code change we can't roll back (putting aside malicious stuff)  Data migrations are scary because they can be 100% destructive and irreversible.  And/or wildly unperformant.

      • We have point-in-time recovery to the minute (!!!)

    • Does GitHub have a canonical solution to what edx-platform-private solves today?

      • "Temporary private forks"

        • Jinder said Feanil or others looked into this early on and it wasn’t ready.

        • May still be hard for GoCD to deploy off of private forks.

    • Idea: "migration files require an extra approval" via CODEOWNERS

  • [inform] (Adam B) - Spreading this info more widely: If you want to add users with granular permissions to ecommerce you can now do so through app-permissions, e.g. like this

  • [ideation/analysis] (David J) In spirit, over time edX employees could be thought of as core contributors themselves.  The CC program has a time commitment (20 hrs/month).  Could edX engineers/others be CCs?  What would that look like for product delivery teams?

...

  • ++++[ideation] (Jeremy) How should we handle dependencies that are little used/maintained by others?  Should we be more actively attempting to switch to alternatives or remove usage?

    • (djoy) What happened to the idea of having a comprehensive index of the packages we rely on, their licenses, etc?  I recall we had a push to figure that out a while back.

    • (nim) There is a tradeoff today since owning teams are not feeling the pains of managing and upgrading their dependencies. There’s an opportunity to surface this back to the teams or find a better way to help teams be accountable.

  • +++[quest] (djoy) Curious about any updates on the hooks extension framework, event bus, splitting LMS and studio, etc.  Things happening!?

    • Hooks - API for Django App Backend Plugins

      • Building on top of Django Signals

      • Focused on providing APIs for backend plugins

      • Targeting completion before Maple is released

      • Recent PR in edx-platform to be reviewed adds initial events that follow this pattern

    • Frontend Plugin Framework

      • Initial focus - iFrame-based plugins in MFEs

      • Next step

        • Find a use case: Could be Discussions in Learning MFE

        • How does this dovetail with LTI?

        • How do we define and document a plugin API?

      • Starting with a fundamental requirement of having a security sandbox around plugins - this means iframes.

      • Explored plugins via module federation

        • Proved complex

        • Impossible to put a sandbox around the code.

        • For plugins specifically, doesn’t have many uses that iframe plugins can’t support

    • Event Bus - Inter-service Communications

      • Working on POC

      • Draft OEP in progress

      • Robert, ChrisP, and Feanil as mini-team on this.

      • Is there any database-level replication work happening as part of the proof-of-concept?

        •  O’Reilly book proposes that the event stream itself is the source of truth.

          • Which O’Reilly book are we talkin’ about?  I see a few related to event driven architecture, I think.

      • What’s an initial use case the POC may look at?

        • Had started looking at Grades events as a possibility.

      • What are the use cases the POC is trying to solve?

      • Note: #event-bus slack channel recently created.

    • xAPI/Caliper

      • Edly is targeting Maple release cut for v1 of this.

      • Intended to be a foundation for integrating data across EdTech organizational boundaries.

    • Boundaries

      • Splitting LMS and Studio

        • Arch-BOM making changes to have Studio login through OAuth

      • Content Theme Architecture

  • ++[ned] wiki space ownership? Good idea, or great idea?

...

  • (3) *** [inform] (Jeremy) Upgrades in preparation for Maple:

  • (6) ****** [ideation] (Nim) Documentation strategy - in prep for hackathon and follow-ups from discussion in #institutional_knowledge

    • OEP-19 Principles

      • Distinguish between temporal versus permanent information

      • Co-location (docs and code)

      • Versioning

      • Ongoing Maintenance

    • Open questions

      • 1 - Discoverability

        • Idea: hierarchy of docs

        •  An independent search tool that searches across all things.

          • We’ve investigated this in hackathons, with Elastic Enterprise Search being a leading contender at the time, but IT approval is tricky due to the security/privacy ramifications

          • We have a Google Custom Search, but it’s a bit wonky: https://docs.edx.org/search  

            • Only searches public docs, and misses some of those if we forget to add them to the widget’s configuration

            • It is still not publicized from docs, which makes it difficult to learn from.

        • Centralizing our docs is probably best if pan-search tool isn’t feasible. Thinking about docs as a tree structure, there should ideally only be 1 “top node”, as in, you go to confluence, and everything is there (this doesn’t include code comments obvi). Searching across several “top nodes” (google docs, confluence, and so on) hurts discovery  ← opinion piece, but *shrug*

          • One root only makes sense if you have one kind of audience?

      • 2 - Non-technical docs ?  Follow the same standards as technical docs or something different?

        • I think we cannot ask other functions to follow Eng doc processes unless there is a clear reason for them to switch away from what they’re doing. Why would product managers start writing rST?

        • The Spec/Approach process seems to be working well for product -- if we could get those in Confluence (for discoverability) then that we be a win.

      • 3 - Tools: Confluence, Google, GitHub, Jira

        • Confluence/Google/Jira: Use these tools strictly for docs that are internally-oriented, temporal, or both.

        • Many of our doc decisions are intended to work around Confluence limitations (poor search, limited concurrent editing, etc.); may be worth considering a different wiki system

        • I think there’s some general guidance about “when to use which tool” but it might be useful to give clearer guidance and alignment so we have less fractured docs. 

        • Every squad maintains an up-to-date homepage with links to their own relevant internal/temporal docs

      • 4 - Access: Open edX versus edX.org, Org-specific-structure access (Security, Squad-specific, etc)

        • I think it could also be a valuable exercise to user-map, figure out who uses our documentation and what they need out of it.

          • E.g. Open edX, edX internal developers, students, course authors. These are already sort of broken up but having a really strong idea of what they need and where they go to access it could really help clear up the “charter” for each piece of documentation.

        • We currently allow http://edx.org specific notes in Open edX docs that others may learn from. Maybe we could not only make this explicitly allowed in the OEP, but acknowledge that it doesn’t have to be limited to http://edx.org . Maybe we have Core Orgs?

        • Idea: for http://edx.org specific technical decisions that would have been an OEP, create a separate closed-sourced repo for http://edx.org OEP-wide decisions.

          • This way, the other benefits of GitHub can apply.

      • 5 - Ownership of documentation as a framework

        • Each doc should have a stated owning team

        • Each team should have a list of all docs they own

        • There should be guidelines on what doc ownership entails

    • Change Management

      • Adoption

        • by other functions as well (Product, UX, etc) since cross-functional docs also need decisions.

    • ACTION Nathan, Ned, Feanil - propose a “just-enough” doc strategy for us before the summer’s hackathon so we have an aligned direction to drive during the hackathon.

  • (5) ***** [quest] (Kyle) Do we foresee edX being on a “level field” with other community members (OpenCraft, RacoonGang, et al) in terms of Open edX core committer rights? Specifically, do we think edX employees and, say, OpenCraft employees will have the same requirements to become CCs?

    • Nim: One of the principles from the Core Committer program:

      • Hold an equal bar for both edX engineers and community engineers, in the long-term. In the future, for instance, edX engineers might earn merge rights just as other contributors to the platform.

    • Nim: For extensions to the core, owning squads as admins of their own repos, can make autonomous decisions on merge rights.

    •  ACTION : Follow up on the B(oundary) part of BEES.

      • Kyle

      • Dave O

      • David Joy (may end up focused on one of the E’s)

      • Nim 

      • Adam Blackwell if it relates to SRE work or just to learn more about boundaries

...

  • [inform] Aperture has set up black auto-formatting on Credentials (our first open source repo using it), so it now matches the other repos we own (coaching and demographics). Positive feelings all around.👋

    • Check in CI, with a Make command

    • Can we create a pre-commit hook?

      • pre-commit hooks aren’t shared, but it could be set up locally

    • Can we create a pre-push hook?

      • possibly, ORA or gradebook may be using it.

  • [analysis?] I think it’d be helpful to have an our “app/code structure” formalized. There are plenty of norms we use, but I feel like a ref without a rulebook when I try to enforce them. 🤪 👋

    • Examples:

      • Apps contain their own rest API instead of it being it’s own app

      • The app’s Rest API should be in app.rest_apiand app.api should be the Python API

      • Apps should use from .models import X vs from long.path.app.models import X ?

    • Context: I love the having a “one right way to do things”, but I assume people will have strong feelings about this.

  • [quest] Which component libraries can/should be reused across MFEs? (asked by JJ who won’t be at this meeting) 🤪 1️⃣ 👋

    • Context: frontend-component-header-edx and frontend-component-footer-edx repos are used by a couple of the older MFEs but not by all. Yet we want consistent header/footers across all/a large portion of our pages…

    • Bonus question: who’s responsible for maintaining these components + updating them with new designs? Example: UX asked Purchase squad to use an updated footer design on the order history page. Prospectus rolled their own version, but frontend-app-ecommerce (the order history page) uses the shared frontend-component-footer-edx repo. Is the Purchase squad responsible for updating the footer design in the shared repo AND upgrading it on the MFE? Relevant ticket: https://openedx.atlassian.net/browse/REV-2073

  • [question] Alex: Do other themes/scrum teams maintain a distinct “engineering roadmap”? How do you maintain it, share it, get product buy-in, etc.? 🤪🤪 🤹‍♀️ 1️⃣ 👋

    • Both Programs and Enterprise have theme-wide engineering roadmaps

    • Current State

      • Independent Arch working groups don’t work with each other

      • Tech roadmaps include both long-term tech improvements/strategies as well as short-term pain

      • Current Wins by ORA team

        • ORA will be replatforming into React

        • Separate “fake” JIRA-sprint for tech initiatives, from which things can be pulled in.

        • Make the case to their PM: e.g., readability, security, etc.

      • Current Wins by Arch-BOM

    • Squad-specific roadmaps:

    • Aperture has a Quality of Life epic

      • We add things that we think need to be worked on

      • At sprint start, our product ask devs what QoL tickets feel important and we pull them in.

  • [question] How do service owners know what services depend on their services? - Adam 😎

    • Today: using Django user table to know whether a service depends on another (existence of the other service’s user).

    • Idea: New Relic should have a view for this.

  • [ask] If a service has a blocking dependency on another service, and your depending service goes down, we want to have a more graceful degradation experience.

  • ACTION Ned Batchelder (Deactivated): Double-check if adr-notifications in #edx includes private repos. If not, what would it take to add them to the edX slack channel?

    • DONE: adr-notifications doesn’t include private repos.

    • Ned Batchelder (Deactivated) If not, what would it take to add them to the edX slack channel?

    • DONE: The configuration for the notification bot is separate from the bot code. The current configuration is here: https://github.com/edx/pr_watcher_configuration/blob/master/config.yml . There’s a “notify_for_private_repos” setting which is not currently being used. We could run another instance of the bot that includes private repos and only notifies internally.

...