Arch Hours: 2022
- Jeremy Bowman (Deactivated)
- Kelly Buchanan
- Ned Batchelder (Deactivated)
- Robert Raposa
Meeting Expectations
Why?
Provide an opportunity for generative discussion and ideas.
Foster comradery through technical curiosity and geekdom.
Who?
Open to all edX-ers and Arbisoft-ers
What?
At times, these informal discussions result in follow-up action and beneficial change in our technology or in our organization. While this is not a decision-making body, these serendipitous discussions spark ideas that may result in ADRs/OEPs and tickets on team backlogs.
At times, it serves as a form of informal office hours to ask live technical questions of the archeological collective.
At times, we have pre-planned deep-dive topics that folks propose to gather wide-input or to answer questions.
At times, we have hosted special guests (internal and external to edX) on specialized topics.
When?
Not lunch hour in ET timezone: With Covid remote work, "Arch Lunch" has evolved into “Arch Hour” in order to accommodate various home/life situations during lunch time.
How? Live Co-Editing
To circumvent Confluence’s limitations with the maximum number of concurrent editors:
during the hour together, we capture topics and take notes at https://docs.google.com/document/d/18TmQf3GllPDfjR7WKiMIhR2eqsbwPi1h3Ojdb6yDYCY/edit.
after the hour, we move those notes to this page.
Why not just stick with keeping the notes in the Google doc?
Google docs are not as discoverable.
Google docs don’t notify observers of future edits.
Google doc comments don’t notify all observers.
How? Structure
Please enter your proposed topics for discussion.
When we use Lean Coffee Style (link1, link2), we vote on which topics the group wants to discuss and time-box the discussion to 10 or 15mns → 5mns (if re-voted) → 5mns (if re-voted).
Prefix your topic with your intention so we are clear on what outcome you are striving from the discussion. Examples:
[inform] You are simply seeking to inform the group of this item. You may field clarifying questions from the group on your inform, but not seeking further discussion at this time.
[ideation] You are seeking divergent and wide perspectives from this group. In this brainstorming mode, all ideas are accepted, without critical analysis.
It may be helpful to clarify whether you’d like to ideate on the problem space or the solution space.
[analysis] You are asking the group to help you poke holes in your idea/topic/plan/etc.
[quest] You are seeking information/responses to a question you have.
2022-12-21
[Phil] [discuss] edx-cookiecutter & auto-adding LMS id from JWT to User Django model in non-LMS new services
Consensus:
Let’s add the lms_user_id in by default: PR + ADR
Let’s consider in the future how to reduce the number of identifiers, especially considering future efforts of unifying identity at 2U
Enterprise may have a model for this in how they stub users if they are added to subscriptions before they exist in the LMS.
Raw discussion notes:
Purchase squad, migrating ecommerce to 2U pre-existing ecommerce - “Titan”
Confusion about canonical user identifiers - LMS user ID
Pie or Exams do this thing about auto-adding LMS user ID - should we add this to the cookie cutter? Should new services automatically have the LMS user ID in their user model?
Well, maybe not all of them need it… but many may eventually need it?
John: Side note: Maybe we could set the id of the user in the new service to be the same as the lms_user_id?
Phil: I didn’t know we could do this!
Chris D: What about conflicts?
John N: There is only one user table that creates IDs
John, Robert: Seconded
Robert: We should have docs in the cookiecutter about this information
Robert: On the older services we didn’t have this for a long time. We were re-using an assorted variety of user identifiers across services. Users were and many times still are being created in LMS by different services.
History: Ecommerce was one of the first repos where we were trying to get the lms_user_id holistically added to all calls to/from the repo & LMS
David: Does Enterprise has any use cases of user imports?
John: We have a stub record we create if a user doesn’t pre-exist in LMS
John: Makes sense to have lms_user_id in the user model. Maybe a future thought is to reduce our total number of ids.
Robert: In the LMS, we do have the concept of external IDs.
Chris: We have global identity as well.
John: Maybe we have options to map it in the future.
[Robert] (quest) Arch Monthly Stand-up used to provide me some info about what others are up to. I know we had thoughts about an async replacement, but right now I feel like I just don’t get this info.
Do others feel they are getting this info? Where can I tap in?
There’s an L&P Scrum of Scrums that covers some of this for managers
Or, do we need some replacement?
BOM teams try to keep track of what to announce, does this need to be a more widely done practice?
Are demo/sharing time meetings common in teams?
2022-12-14
[Feanil/Ned] Announcements
Think about conference talk ideas over break! Open edX Conference - 2023: Call for Proposals
[Feanil] General overview of how things are going at 2U?
[Andy] report on LTI tool actual vs. specified or expected behavior
Unique identifiers
PII sharing
2022-12-07
[2U internal] [Ned] (discuss) Brainstorm things 2U should tell people at the Open edX conference: https://docs.google.com/document/d/1nBW_uS7KSjFNq1K_sjkv8IliadcUiDc06HqCo4DIjas/edit
[2U internal] [Ned] (inform) Open edX lunch+learn Weds 14th. What questions don’t people even know they should ask? https://docs.google.com/document/d/1EpQ8TP3P38F5QPRF8IKx8NTISMFnCmKG-ZMXjW4YUiM/edit
[2U internal] [Ned] Is it OK that we don’t have a common town square?
We use the #tech-dev-edx for 2U
Docs: https://2u-internal.atlassian.net/wiki/spaces/AT/pages/16385625/How+We+Announce
[todo] Add sre@edx.org to some announcements
[idea] Use blogs
[question] Should we create a new Google group?
Maybe not, it may hide information from recipients
What would it be called?
Who would maintain it?
[2U internal] [Andy] tales of allowed programs
Ngrok
Github local testing of webhooks
Sharing a dev environment from
2022-11-30
[Ben W] What does the http->https forwarding?
[Robert] Cloudflare probably for http->https. Also, an answer to a separate question, Google TagManager is often where random scripts are dropped on the page.
2022-11-23
Low attendance due to Thanksgiving-related PTO. There was some continuation of discussions about XBlocks, iframes, and CSS conflicts, but notes weren’t taken.
2022-11-16
[inform] (Jeremy) Updated draft of Development Environment Vision is ready for review
[quest] Jeff Witt 1 min: Use of !important in CSS – OK to use, or to be avoided? Consensus seems to be that it’d be best to avoid it. Uncertain if there’s any substantial a11y angle on this guideline.
[discussion] Ned: OEP-55 Maintainership: monitoring issues, PR SLAs
We’ve picked repos for the pilot that are likely to do well at this, but what happens when it’s expanded to repos owned by overwhelmed teams?
[John] Do we need someone like Natalia to help teams keep track of this?
[Andy] It’ll probably increase the pressure to catch up with the maintenance backlog in various repos
[Jeremy] I suspect that much of the need for a project manager arises from immature processes around software maintenance and sustainability, we should also take steps to address that.
[Ned] Pilot Phase 2
[Andy] We have processes for tracking OSPRs, but really not for GitHub Issues yet. How do we make sure these actually get considered when prioritizing? (Given that many of our product managers/owners live primarily in Jira.)
[John] We could improve scheduling of automated upgrade PRs.
[Andy] Some teams are already doing this, at least for the Python upgrade PRs.
Much of OSPR handling is currently being dealt with in per-team on-call processes, which works but may not be the ideal approach.
[Andy] If you have a product-mandated backlog, fix that first. Needs to be a conversation that factors in maintenance needs.
[John] Having more advance notice that PRs will be coming (and why) really helps.
2022-11-09
[ideation] (Beggs) - OAS (OpenApi), Rest API standards and API client/SDK generation
GitHub - openedx/api-doc-tools: Tools for writing and generating API documentation for edX REST APIs
https://2u-internal.atlassian.net/wiki/spaces/IM/pages/18973040/Consumer+Driven+Contract+Testing (and child pages, for Pact)
Django Packages : drf api documentation (drf-spectacular may be worth switching to from drf-yasg, see GitHub - axnsan12/drf-yasg: Automated generation of real Swagger/OpenAPI 2.0 schemas from Django REST Framework code. )
[ideation] (Beggs) - Standardized/Convergent core education objects for all of 2u (enrollment, course, grades, course completion, etc…)
[Jeremy] Global identity project and workshops
[Jeremy] L&P Eng Leads conversations around DDD universal language and/or unified data model
[inform] (Ned) Open edX CFP
[quest] (Jeremy) - Senior engineers & business context awareness
Many engineers lack at least context around user demographics
Also lacking good context on specific priorities for the current and upcoming quarters (thanks re-orgs)
Engineering managers in too many meetings, senior engineers in not enough
https://github.com/orgs/edx/projects/15/views/1 (Roadmap of Platform Core Teams)
Business model summary doc could be useful, especially for onboarding
2022-11-02
[Jeff] Are all XBlocks in the Learning MFE kept in iframes?
[Jeremy] Any useful info in https://openedx.atlassian.net/wiki/spaces/AC/pages/1890681313
[Jeremy] Or in https://github.com/openedx/frontend-app-learning/blob/master/docs/decisions/0002-courseware-page-decisions.md ?
An entire unit (which may consist of multiple XBlocks) is rendered in the iframe
[Jeremy] An update in https://github.com/openedx/frontend-app-learning/blob/master/docs/decisions/0009-courseware-api-direction.md indicates that the iframe rendering isn’t on the shortlist of things to change
[Robert] [quest] What process should we use to review and commit (or abandon) to various architectural docs?
Is the OEP process the right process?
From a resourcing perspective, how do we handle our current situation, rather than the old chief-architect model.
Here are some of the now-questionable docs: https://openedx.atlassian.net/wiki/spaces/AC/pages/921895082
Note: as an example, DDD is listed as one of the principles, but is that still the case?
Ideas
Break down existing docs into smaller OEPs to try to get them over the line.
Start with less controversial parts of the Manifesto, or less controversial DDD domains (rather than all domains).
Use Architectural Coordination Working Group to find resources in trying to push some of this work over the line.
For DDD, architects driving OEPs would need to include Product.
2022-10-26
Skipped due to low attendance
2022-10-19
[Jeremy] High-level development environment objectives
No need to debug code updating problems
Fast to set up a new dev environment
Don’t need to carefully preserve manually set up testing data
Good support for debugging and observability
Consistent between services
Able to run reasonable subsets of the full Open edX ecosystem of services
Defaults to feature flags currently active in production
Comes with data needed to quickly test most features
[Adam] [quest] I'd like to discuss with this group and Simon to better understand the plan for moving to Open Search
E.g. When will we know that OpenSearch will be the thing for edx-platform?
How do we know that discovery and forums won’t have any issues
Do we have a plan to remove elasticsearch from edx-platform?
Indexing courses and course teams
What is the cost of delay of staying ?
What is the
Notes: 200 ES searches per month
Adam: I also need to leave shortly after 9:30
John supports less Elasticsearch, RDS is pretty good
Need to push pre-Olive for non-AWS
AWS will support
Current SRE work: https://2u-internal.atlassian.net/browse/ISRE-1280
Infinity needs the data in the new stage cluster
Who owns discovery? Vanguards
DEPR is also looking into stopping our use of elasticsearch
[Jeremy] Can we get away from requiring thorough owning team review for maintenance, bug fix, and small feature enhancement PRs? What would have to change to make that happen?
Plugins/libraries need to have been tested in the things they’re installed in
Make test suites more reflective of actual behavior in production deployment
Make the changes unused in edx-platform
Address issues raised in previous RCAs - trailing slash consistency, database migration linting
Shorten the time from merging to detecting problems in production
Canary deployments?
Shrink the size of edx-platform (small problem can bring down a large chunk of production)
Automatically deploy a test environment that exercises the change
2022-10-12
[inform] (Ben W) FWG/Opencraft/RacoonGang Theming conflict. Working with groups to try and consolidate how we co-ordinate work around the platform between working groups and get them to talk to each other.
We ended up with parallel meetings: 2U-focused and Open edX community focused
Not much communication happened between these parallel groups
Trying to fit all front end stuff into one series of meetings ends up at poor signal to noise ratio
Need a clear forum for this coordination
(Ned) Concerned about defaulting to a meeting as the primary forum for this: conflicts, time zones, etc.
(Chris) Should we use the Open edX roadmap for this kind of coordination?
(Ben) Trying to reconcile architectural initiatives being driven by multiple organizations in the same project sounds terrifying
(John) This sounds like a flaw/scalability problem with our architecture and process that needs to be fixed
(Andy) We need to get better at sending more redundant communications the larger a project is
[quest] (Jeremy B) Developer Experience - reasonable focus for this meeting? Arch-BOM is pivoting(?) to focus on this
For a loose definition of DX, perhaps
Any examples or resources we should learn from?
(Ben) Standardized debugging/troubleshooting tools
(Chris) It feels like some aspects of DX start to cross back into architecture
(Chris) Would be good to get a status update on development environment efforts
(Ben) How can we make “thing not in platform” easier (for a new python API)
(Ben) How can we make “New MFE” easier (observability, config, etc)
(Hamzah) A “newsletter” of changes and features would be helpful.
[inform] (Ben W) FedX exists again. What this means. What our focuses will be.
2022-10-05
[inform] (Ned) We need hackathon organizers, please volunteer
[quest] (Jeremy) Hacktoberfest - do we want to accept contributions this year?
Easy way to get T-shirts for developers
But it’s not clear how much else we get out of this; usually a few vaguely useful contributions, a few mild wastes of time
[Ned] We have enough problems with our contribution pipeline as is, may not be a great idea to pile more into the backlog
[Jeremy] We do have a bunch of GitHub Issues for edx-platform pytest warning fixes that we could tag for participants
Let’s activate selectively for things where there are useful issues open and maintainers are willing
[quest] (Jeremy) How important/useful to people think type checking would be?
[Ned] First we’d need to fix our existing linting
We’d need a policy, one reasonable example could be “you may add type hints, but you don’t have to. If you do, the linting must not break”
Communicate said policy
Hold off on any big push for test-generated type hints or other comprehensive annotations
[discuss] (Diana) What do we need to do to make sure there’s not much disruption from Slack migration?
Migrate existing channels
Update integrations
Handle shared channels
There’s a lot written about this, few people have had time to read it all. And it sounds like there are at least a few corner cases that the docs and process don’t cover yet.
Emoji transfers (Matt Hughes seems to be working on this)
[discuss] (Ned) Links to private wikis from public wiki. Allowed/disallowed?
Feanil: it’s fine as long as it’s clear that it’s a private link and that it was understood by the author that it’s private.
[Jeremy] Is it worth wrapping them in conditional content blocks to make it explicit and avoid distracting other readers?
[Feanil] How about a table at the bottom of the page for links to each org’s private related context?
[inform] (Ned) Kelly is trying to formalize “public workstreams”: https://edx-internal.slack.com/archives/CDA7GMJ4B/p1664910103145889 (private 2U link)
[discuss] Max’s impressions of FedBom PR flow
[Jeremy] Gave historical context
A key problem is that teams feel that the risk of breakage is high enough that it’s only worth merging code directly related to current product management priorities. This isn’t good for long-term product quality and maintenance efforts, and we need to find ways to fix it.
[Feanil] Michelle Philbrick has been nominated for new core committer role to help manage PR workflow, evolution of what Natalia has done in the past
[Feanil] Grimoire does some PR tracking work
[inform/quest] (Jeremy) Arch-BOM -> Developer Experience
If you have any suggestions on improvements that should be prioritized, please let us know
2022-09-28
[Ned/inform] Open Source Process working group: https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/19467639/Open+Source+Process+Working+Group (private)
#openedx-internal in edX or 2U slack (private)
[Ned] Forking Strategies doc in progress: https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/155746369/Forking+Strategies (private)
[Jeff] Does this also cover the case where we need to fork an external dependency to fix an a11y issue?
[Ned] Not yet, but it probably should
[Jeremy] Potentially related: https://openedx.atlassian.net/wiki/spaces/AC/pages/3036972032
[Jeff/quest] Do we have a Dates API, for extensions?
Idea is that we should have some mechanism in the platform to facilitate people scheduling time to work together on a course
Things like this: https://www.flow.club/ and https://focusme.com/
[Dave] There’s support for retrieving key dates about the course, but not adding dates
[ideation] (Jeremy) Frontend security vulnerability handling
We get dependabot alerts about security vulnerabilities in dependencies.
Would be nice to just upgrade things (hopefully automatically)
Fed-BOM is working to get upgrade PRs like this assigned to owning teams.
[Alex] opines that teams may be missing a more formal on-call process, through which these upgrades could be actualized.
[Andy] A big part of the problem is that our frontend test suites are insufficient to catch even fairly major problems before deployment
This is not really a frontend unique problem, it hits all PRs from outside the team
[Feanil/question] What kind of testing maturity do we feel we need?
Better mocking and Test Data
More contract testing
Educating more developers on how to do this
Recording by Dawoud? Jeremy will follow up to discuss the possibility
BTW, he’s giving a DjangoCon US talk this year: https://2022.djangocon.us/talks/building-microservice-architecture-for/
Questions
Can you test backend rendered pages with contract testing?
Adding tests specifically for issues that broke Prod.
Record context on the bugs that escaped to production in a more public way so the community can better understand what broke and how.
[Ned/question] Hackathon?
[Jeremy] We need organizers, please get in touch if interested
2022-09-21
[Ned] What information would be helpful for improving PR review flow, either inside or outside of 2U?
[Ben] Tickets! (Jira) Hard to accurately prioritize among other work.
[John] Get PMs to review them on arrival?
[Jeremy] There are columns in the Squads tab of the ownership spreadsheet to specify how each team wants to be notified of review requests.
[Jeremy] https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/76808270/Cross-Squad+PR+Reviews proposes guidelines for improving this, feel free to suggest enhancements
[Andy] Even pretty innocuous PRs have caused major problems recently, making people very reluctant to review and merge incoming PRs from outside the team
[Andy] Having a dedicated concierge for tracking incoming PR review requests to each team would be useful.
[Jeremy] There’s an auto-formatter for tox.ini files; would people find value in trying it? https://pypi.org/project/tox-ini-fmt/
Tox-ini-fmt deletes all comments!! https://github.com/tox-dev/tox-ini-fmt/issues/42
I want one for GitHub actions