/
2024-02-08 Meeting notes

2024-02-08 Meeting notes

 Date

Feb 8, 2024

 Participants

  • @Feanil Patel

  • @Felipe Montoya

  • @Michelle Philbrick

  • @Kyle McCormick

  • @Sarina Canelake

  • @Tobias Macey

  • @Navin Karkera

  • @Jeremy Ristau

  • @Robert Raposa

  • @Tim Krones

  • @Maksim Sokolskiy

  • @Xavier Antoviaque

  • @Max Frank

 Goals

  •  

 Discussion topics

Time

Item

Presenter

Notes

Time

Item

Presenter

Notes

 

Review TODOs from Last time

@Feanil Patel

 

Ubuntu Upgrade

@Tobias Macey

MIT already running using Debian Buster instead of Ubuntu and using standard python Docker Images

  • Reduces usage of other infra like the deadsnakes PPA.

  • Using the external codejail service and building that against the Ubuntu container.

    • Using a sudoers profile inside the code jail service image.



Node 18 → 20 Upgrade

@Feanil Patel

Looks like we’ll have to accelerate this upgrade to keep things functional. @Adolfo Brandes can you provide some context here?

Context:

  • The upgrade from Node 16 to 18 took a long time, and a lot of coordination. We still have time until Node 18 is EOL (April 2025), but some packages have started to drop support for it already, leading to complications when upgrading related dependencies.

  • Renovate has already started to open 18 → 20 upgrade PRs.

  • To get in front of it, we should elect somebody to start the discovery and coordination work, and set a tentative target: Sumac.

  • Tutor only supports one version of Node in production for MFEs

    • Can we change something in the tutor plugin to bring up a second container for some of the MFEs?

      • @Régis B. people are concerned that a PR to run 2 images would be to clunky to be accepted by the tutor team. Do you have thoughts on how we could incrementally upgrade MFEs without having to do a big bang upgrade of all of MFEs.

      • We’re going to have more MFEs in the future and doing a big bang upgrade can be more costly.

      • Note: In the future we will have the same problem with Piral so we may be stuck with this.

 

Moving NPM workspaces

@Adolfo Brandes

Turns out it’s a pretty manual process. @Brian Smith and now @Yagnesh Nayi have been contributing PRs, but as usual, the more contributors, the merrier!

Main ticket (see referenced issues towards the bottom of the feed): https://github.com/openedx/axim-engineering/issues/23

 

Maintenance burden of org-specific code

@Adolfo Brandes

In going through the above (in particular, when upgrading frontend-app-course-authoring dependencies), we found that the Video Upload feature is not realistically usable by the community. Not only that, it’s not feasible to run it locally for the purpose of, for example, fixing unit tests. We proposed that in such situations, the unit tests be disabled / made non-blocking as a workaround, and long-term, the feature should be pluggified, deprecated, or put in the docs and other work to make it usable by the community.

 

  • We’re aware that we don’t have a single plugin API that we’re going with yet.

  • Can the tests be run elsewhere?

    • How do we run the tests sometimes?

      • This is in Jest, there is some capability for optionally running tests.

  • How can 2U know that 2U specific tests were disabled?

    •  

 

Ruby Upgrade in Comment Service

@Maksim Sokolskiy

What’s the plan with the ruby upgrade?

  • Feanil will prioritize finding a maintainer for this so we can get the Ruby upgrade going.

  •  

 

Central vs Distributed Tasks for Upgrades

 

Who will be in charge of the centralized tasks for upgrades?

  • This group

  • We’ll need help with both the creation of tasks and driving them forward with maintainers

  • Feanil will make board

  • Product roadmap will also have line items for main upgrade epics.

  • Difference between transparency and visibility

    • A board is useful, but comms is still important.

    • Ristau would be happy to help bolster some of the communications and visibilty steps moving forward.

Previous Action Items

 Action items

@Feanil Patel Add a link to the meeting notes to the calendar event.
@Felipe Montoya to connect Feanil with edunext contact who can help maintain ecommerce for now.
@Felipe Montoya to email Feanil a list of repos that edunext is interested in.
@Adolfo Brandes ticket up enabling node 20 tests on all existing released MFEs and post a call to action in the forums.
@Robert Raposa to come back with any process that could help us easily find the disabled tests, specifically with Jest.
@Feanil Patel to create a maintance project board to track all outstanding maintenance work.
@Jeremy Ristau to review existing 2U upgrade processes and comms and update/hand-off to the community.
Anyone can provide feedback to Jeremy via Slack to help improve the process.
@Feanil Patel put up cs-comment-service for ownership and be clear about the upgrade that needs to happen.

 Decisions

 

edx-platform Maintenance Meeting Notes

 Participants

  • @Feanil Patel

  • @Kyle McCormick

  • @Tobias Macey

  • @Jeremy Ristau

  • @Robert Raposa

Discussion Topics

Time

Item

Presenter

Notes

Time

Item

Presenter

Notes

 

Historic edx-platform maintenance state

@Kyle McCormick

We had a central team that would route to experts within 2U. We think that was a good system but we would want to build that with CCs instead of a 2U internal ownership delegation.

  • Michelle was historically doing the routing within edx-platform to 2U.

  • More recently the BOM team tried to take over and that was badly timed.

  • Be mindful about the difference between the alerting setup within 2U which will remain internal and the routing for PR reviews which will become public.

  • Should we be switching to CODEOWNERS instead?

    • CODEOWNERS is not super visible and does not get info to the owners quickly.

    • CODEOWNERS currently auto assigns PRs

 

Impending Redwood Upgrades

 

  • More important than getting the full ownership mapping.

  • For now focus on the upgrades we need for Redwood.

    • Python 3.8->3.10/11/12

    • Mongo 4->6

  • Feanil will work on making tickets for all edx-platform python packages

  • Who will make the PRs?

    • Arbi-BOM used tools to create issues

    • We think they have tooling to update the matrix

  •  

 

3.11 or 3.12?

 

  • If some repos are already running 3.11 let’s aim for 3.11 otherwise let’s get to 3.12.

 

where will the issues go?

 

Feanil will make it

Give him a search query and he’ll ge them on there

 Action items

@Jeremy Ristau to make sure that the internal audit of ownership at 2U will end with an update to CODEOWNERS file to make that mapping public.
@Feanil Patel Ticket running 3.12 testing on all python repos in the openedx org that are in the release with a priority for edx-platform.
@Robert Raposa @Jeremy Ristau will reach out to Arbi-BOM to make the github matrix change PRs for exsting python repos. Also, find out what Python upgrade state is?
@Kyle McCormick Will look into Python 3.9+ dependency on libsass

Decisions

  1. There should be a public routing system/table for routing PRs within edx-platform
  2. We’ll start by putting the mapping into the CODEOWNERS file.

 

Transcript and Recording

Maintenance Working Group Meeting (2024-02-08 08:35 GMT-5) - Transcript

Attendees

Adolfo Brandes, Feanil Patel, Feanil Patel's Presentation, Felipe Montoya, Jeremy Ristau, Kyle McCormick, Maksim Sokolskiy, Max Frank, Michelle Philbrick, Navin Karkera, Robert Raposa, Sarina Canelake, Tim Krones, Tobias Macey, Xavier Antoviaque, Yagnesh nayi

Transcript

This editable transcript was computer generated and might contain errors. People can also change the text after it was created.

Feanil Patel: Okay.

Feanil Patel: Hi everybody. Welcome to the second meeting of the maintenance working group.

Feanil Patel: We've got a couple of different things to cover. I'm going to start to do from last time. Jeremy do you have new information on u up the Ubuntu upgrade. and the edx platform tests infrastructure

Jeremy Ristau: I asked about it. I'm not totally sure. I would call it an update per se. I have confirmation that previous you want to upgrades or dealt with as a collaboration between internal teams inside of 2u so there's infrastructure upgrades. We got to go to SRE for that but then there are code updates those can be done and have been done by the bomb teams historically…

Feanil Patel: Yeah.

Jeremy Ristau: if tests or test infrastructure needed to be changed that would be done waiting. Obviously. All those teams are heavily Consolidated now, so I get any information about what would need to change. We don't know that because we haven't looked into it, but I also don't have confirmation that anyone to upgrade would actually be prioritized by to you right now. So.

Feanil Patel: okay.

Feanil Patel: right

Jeremy Ristau: I think I can say I know what we've done historically, but I don't have any Clarity on what we'll do going forward.

Feanil Patel: there's sort of two places where I think that. We sort of need help is I don't think we need necessarily to you to do an Ubuntu upgrade but the infrastructure that runs the tests is still I think maintained by to you. So either helping us get that updated so we can test two things that are more which is the mango upgrade in the Ubuntu upgrade itself the mango upgrade I think being the more urgent of the two and hopefully the smaller lift.

Feanil Patel: Maybe we can Target that and understand sort of what it would take to get that over the line.

Jeremy Ristau: And are you referring specifically to the Cyprus test infrastructure or something else?

Feanil Patel: No, no, so the GitHub actions Runners for the edx platform tests are created and managed by to you at the moment.

Jeremy Ristau: Okay, and

Feanil Patel: and the version of Mongo on there is controlled by building that container somewhere Onto You infrastructure. So in order for that version of Mongo to change currently we need to use help.

Jeremy Ristau: Okay, so I can definitely confirm that the Mongo 6.0 upgrade is in our list of upgrades to do. It hasn't been fully prioritized yet…

Feanil Patel: Okay.

Jeremy Ristau: because I think we just decided recently to skip Mongo 5 and…

Feanil Patel: Yeah.

Jeremy Ristau: go straight to Mongo 6 instead rolling upgrades.

Feanil Patel: Yeah.

Jeremy Ristau: So I think we're sort of in the phase of planning for it, but if Help is needed to upgrade Mongo. That's something we definitely will offer.

Feanil Patel: Yeah, I mean in particular the thing I think we need help with is the current test Runners. I think only have Mongo 4 on them. And so if they had both installed or both.

Feanil Patel: that's a couple more people if they had both installed or if there were two set of Runners so that we could test with both that would I think identify a lot of issues and let us give us a lot more confidence. So I don't know how big a lift that is, but that I think is the thing to prioritize.

Robert Raposa: and finale I'm just curious I don't know the config for the runners that you're discussing is that private is that okay?

Feanil Patel: I think yeah, so historically I know to you has been managing that so I don't know the full details, but I think the runners themselves are run on sort of two you containerization infrastructure somewhere and then the config for that. I believe that's part of why arkbomb had admin access on adex platform was to manage that config.

00:05:00

Feanil Patel: that said I think that I looked at that code recently and I think that it can become less complicated by using their GitHub actions to set up Mongo that I used in some of the other tests, but it's hard to set them up on the Ubuntu machines that we're currently using for edx platform for the unit tests. So I think there's a way off of this if that's a thing that's desirable, but we need to sort of think about that. before cup

Feanil Patel: we can't hear you.

Kyle McCormick: Bye Cool, the tests are only run on to infrastructure.

Feanil Patel: Yeah.

Adolfo Brandes: That's good.

Kyle McCormick: If they're against the open edx or edx if they're in the open adx or edx private Forks of that If you open a PR against your personal Fork of X platform. it will run the same tests on GitHub actions like Really donors.

Feanil Patel: Okay,

Kyle McCormick: I know that was done for tests time reasons you could Get the test time down by running the test themselves. I don't know if there are more differences. So that is something we should check out because maybe we just switch to the GitHub action Runner version.

Feanil Patel: Yeah, I looked at that and it looked like there was a bunch of stuff installed by default on the edx platform Runner. So I wonder if there's different I might have not looked at all of the GitHub actions workflow files and maybe they're separate workflows that run depending on which is available. yeah, okay, so

Feanil Patel: Yeah.

Feanil Patel: cool Felipe do you have any updates Jeremy? I'm gonna mark this done for now.

Feanil Patel: unless you want to sort of do more follow-up, but I think that there's maybe a different open question about the Mongo to Mongo six update which we can sort of have follow-up conversations on

Tobias Macey: Just as a side note to the Ubuntu conversation for the deployment that I'm running. We're currently actually using the Upstream python containers running against Debian Buster and that's been stable for us just as a anecdote not necessarily advocating for that but in case that's useful to anyone.

Adolfo Brandes: That's interesting. I ought to bring this up to hajis. So we don't necessarily need. Ubuntu

Tobias Macey: Correct. Yeah,…

Adolfo Brandes: Okay.

Tobias Macey: we were able to build our images using Debian Buster. We're not using every feature of edx app. So, I don't know if there are elements of coverage that we're missing in that but we've been able to successfully build and run our instances and production for a number of months and Debbie and Buster from the Upstream python container image.

Adolfo Brandes: And What does that give you over using Ubuntu longer support? What?

Tobias Macey: It's just more in line with the rest of our infrastructure. So we use Debbie and as the base Ami for all of our other applications, and so it's just consistency. Yeah, just not coupled to the Ubuntu ecosystem and…

Adolfo Brandes: Okay.

Tobias Macey: then also because we're using the Upstream python Docker images. It just reduces the number of steps required to build the image. So we don't have to pull in the dead snakes BPA, etc, etc.

Adolfo Brandes: Yeah, that's valuable info.

Adolfo Brandes: Thank you.

Feanil Patel: It sounds good. Did tell me did you guys get coach jail to work with that setup for edex platform?

Tobias Macey: So we are actually using the external code jail service. And I think we're building that against Debian as well.

Feanil Patel: got a

Tobias Macey: Let me find that.

Tobias Macey: it looks like that we're using the Ubuntu container for I don't know exactly why we didn't go with Debian for that. But I think we just at the time just did a wholesale copy paste of What other folks are doing?

Feanil Patel: and then

00:10:00

Feanil Patel: that catchment and for that you just have a code jail or you have an app armor profile associated with that code jail. image

Tobias Macey: Yep. Looks like we're using sudoers actually.

Tobias Macey: Yeah, so we have a pseudoers file this that we're using inside that image.

Feanil Patel: Okay, awesome. Let's see. Yeah, Felipe. Do you want to check them in with you for a second on the commerce testing? Did you guys get a chance to talk about that?

Felipe Montoya: Yes, so I talked to the teams in an Erics that are using that and one larger team would rather get rid of commerce completely but they are using Enterprise and the time we briefly spoke out if Enterprise was to continue more forward without a Commerce how that look like and they rather like that vision. Try to help on the Enterprise side to remove agencies on the comments.

Feanil Patel: Okay.

Felipe Montoya: The modern game could devote some of our amount of equipment when we're putting it two hours a week or something to just keeping the lights on mode. So maintenance tests that they operates in completely recover and that's it. But at least not without the customer that they work for. Also committing to eCommerce they couldn't take more resources in that. But something's more like towards the big then definitely they can do it and just require.

Feanil Patel: I think we will need that where the moment so maybe you could put me in touch with them or have them post something in the forums. Whoever the right technical contact is over there.

Felipe Montoya: Yeah of those.

Feanil Patel: awesome

Felipe Montoya: or both actually Yeah.

Feanil Patel: Probably both eventually.

Feanil Patel: cool

Feanil Patel: Yeah, and I think definitely Jenna and I would both be interested in talking more about what it would take to. Extract the dependency on commerce from Enterprise if that is. A thing you guys want to help? support and pursue

Feanil Patel: All right, the next two things on here are updating the maintenance spreadsheet with a new column. How did a quick run of the old pilots and put that data in there? I have not yet gone through the catalog info files and started automatic automating that in there yet. So that's still outstanding and then I have posted an update to discourse about the change in requirements we discussed last time but I've not yet hosted a new.

Feanil Patel: A new list of repo seeking maintenance because I've been waiting on some other people to provide me lists, and I think I have most of that now, so hopefully we'll be able to do that soon.

Feanil Patel: So that's Felipe go for it.

Felipe Montoya: Yeah, so if we're ready when we've been discussing the whole weekends what we can do in terms of Maintenance. Should I update you directly on the reposphere to maintain their maintenance industry both or wait for the post? And then right? As a responsible.

Feanil Patel: why don't you send me the list that you guys are interested in so I can cross-reference where there's multiple people and where there's not and then I'll start posting here's a list of things that we're looking for. So it's not the bombard Everything at Once although I'm happy to change my sort of tactic if people feel it would be better to just say here's the full list and please volunteer, so I'm not sure about which to do so I could use some support some. thoughts on that

Felipe Montoya: Sure, I live in you with reset. We are interested in and

Feanil Patel: Okay, awesome. Thank you.

00:15:00

Feanil Patel: Okay, let's see. That's good.

Feanil Patel: Okay, sounds good. Thanks, very

Feanil Patel: That's just a bit.

Feanil Patel: Next so we covered the Ubuntu upgrade. Is there anything else anybody wants to say on the Ubuntu upgrade?

Feanil Patel: So the node 18 to 20 upgrade an Adolfo raise this I believe yesterday or the day before we've been talking about it a couple of times but to summarize I think I'm gonna give it a shot Adolfo and then you tell me if I missed anything to summarize we are. Currently running node 18, which is technically not end of life. But the node ecosystem is basically over it and starting to upgrade things to node 20 and requiring things to have no 20 newer versions of packages. We are depending on are starting to drop support for node 18, which may require us to upgrade to node 20 sooner rather than later. I miss anything Adolfo.

Adolfo Brandes: No, that's about it.

Adolfo Brandes: and as I mentioned there we have a long time quote unquote Until it's end of life. But that really is just sumac right…

Feanil Patel: Okay.

Adolfo Brandes: because the one after that he is already end of life. So this is too soon for Redwood,…

Feanil Patel: right

Adolfo Brandes: I think. maybe we can get in a couple of them if he's that actually supports node 20, but as I mentioned in a slack thread because tutor only supports one version of notes in production. during this transitional phase all mfps are going to have to support node 18 and 20.

Feanil Patel: functions

Adolfo Brandes: It's the same thing that happened during the 16 to 18 upgrade.

Feanil Patel: is

Adolfo Brandes: So the unit tests have to have a test Matrix for both versions. in the MF fees

Feanil Patel: It possible to change something in tutor to just bring up two images temporarily like in the plugin for mfe's. Which I assume already has a list of MFS that it's loading just have a second container.

Adolfo Brandes: it's possible but It's really clunky. and I doubt Jesus is going to accept a PR that does that

Feanil Patel: I mean Okay.

Feanil Patel: That sounds like a conversation worth having.

Adolfo Brandes: Yeah, but as I mean, I'm also maintainer of tutor MFE which is where this would happen. and I actually think This should be on the mfis,…

Feanil Patel: but

Adolfo Brandes: right? it's

Adolfo Brandes: it needs to be done at some point. right

Feanil Patel: Yeah, yeah, I don't think I'm disputing that. I think.

Feanil Patel: If it's a timing issue. I think being able to gracefully upgrade some of the mfes without having to block on all of the mfps is really valuable.

Adolfo Brandes: No, I agree.

Feanil Patel: And so if our development environment can't support anything but a big bang that to me is a weakness of our development environment.

Adolfo Brandes: But the development environment if you run mfe's individually can support the different versions.

Feanil Patel: Sure, our communities.

Adolfo Brandes: but in production But in production,…

Feanil Patel: Yeah, yeah. whatever we say is the official version should be flexible enough to be able to let us upgrade some things and…

Adolfo Brandes: yeah, I know it.

Feanil Patel: not other things I think because

Adolfo Brandes: Yeah. But it's a complicated conversation…

Kyle McCormick: an MFE can support

Adolfo Brandes: because then you lose some of the benefits of the Xavier single image for all them plus when we move on to a pyro world.

Adolfo Brandes: That's just going to be the case. Sometimes all of my fees are gonna have to be upgraded at once at least all the ones that are in supported by the shell. Maybe not. I don't know. But this is a big conversation.

00:20:00

Feanil Patel: Yeah, let's get it started. Yeah.

Kyle McCormick: Adolfo

Adolfo Brandes: Yeah. Go ahead.

Kyle McCormick: 10 an MFE support both note 18 and 20 at the same time. are they?

Adolfo Brandes: yeah, yeah, that's

Kyle McCormick: So could we start testing specific mfes, running tests on? Okay, so whether or…

Adolfo Brandes: That's a great first action item,…

Kyle McCormick: you can do it we should do that because

Adolfo Brandes: but start enabling no 18 tests or…

Feanil Patel: No 25th.

Adolfo Brandes: not 20 times

Feanil Patel: Adolfo do you think for all the mfe's can you just pick it up doing that? It sounds like there's already PRS for that.

Adolfo Brandes: Yeah, sure.

Sarina Canelake: Yeah, something that I don't feel I've heard brought up. But just as another thing to consider about upgrades is we're gonna have to keep doing these upgrades and we might be getting more mfes. And so the more we're stuck in that big bang situation. It's just going to get harder and harder we're gonna have to keep dealing with this over and over so I think it's important that we try and look at all the solutions and even if there's up front cost to one solution if it makes the next few years of upgrades easier, I would say we should go with something like that.

Adolfo Brandes: That's a good point. Yeah.

Feanil Patel: Yeah.

Adolfo Brandes: It is something that it's in the back of my mind with the move to spiral and how that's going to impact things. regarding in particular because right now the gatekeeper is tutor right like having to do all at once sort of…

Feanil Patel: Sure.

Adolfo Brandes: but it will have a higher level actually lower level gatekeeper going forward worth investigating how that's going to

Tobias Macey: So I missed what you said was the upcoming gatekeeper Beyond tutor.

Adolfo Brandes: viral, …

Feanil Patel: payroll

Adolfo Brandes: it's where we're actually Moving forward faster with that now and in that world, it's a single web app a single Spa right? it's not multiple sbas anymore. And that will introduce certain constraints on polymer fees or loaded built on and so forth. We're not sure for example, if we'll be able to build nmfe with nodes, 18 where the rest are built with no 20 and That's still work.

Feanil Patel: right

Adolfo Brandes: I don't know.