edx-platform Maintenance  Sub-Group (2024-04-18 09:03 GMT-4) - Transcript

Attendees

Awais Qureshi, Feanil Patel, Feanil Patel's Presentation, Felipe Montoya, Jeremy Ristau, Kyle McCormick, Maksim Sokolskiy, Maria Grimaldi, Michelle Philbrick, Muhammad Qasim Gulzar, Piotr Surowiec, Tim Krones

Transcript

This editable transcript was computer generated and might contain errors. People can also change the text after it was created.

Jeremy Ristau: Logged out every 10 hours now.

Feanil Patel: Not that I'm aware of. I haven't had logout issues.

Jeremy Ristau: Okay.

Feanil Patel: So I don't know if it's something on this side. but

Kyle McCormick: Hey guys, SSO again from your Google account.

Jeremy Ristau: I have no idea. Yes.

Feanil Patel: if you're looking in with Google to you account, there might be some things your ideas so that prevent it from being longer something there's nothing as far as I know on the open edx Wiki said that causes you to log out of every 10 hours.

Jeremy Ristau: Yeah.

Jeremy Ristau: Fantastic. Yeah, I'm getting pushed through. MFA to sign in so that sounds right.

Feanil Patel: But yeah, yeah to you it for the win I think.

Jeremy Ristau: It's confusing because I log in with my http://edx.org email address,…

Feanil Patel: Right, right.

Jeremy Ristau: but it

Feanil Patel: What's going all the way around?

Feanil Patel: All right, so let's get going. I just wanted to check in on a couple of different things. I think all the previous to do items are mostly to do with this meeting so that we can check in with them now. which is

Feanil Patel: you have the two you coordinating the three of the 311 upgrades that we can drop three eight support after Redwood. Is there anything new happening there?

Kyle McCormick: I didn't reach out to them that's on me. I'll do that for next week.

Feanil Patel: That sounds good. Yeah, and…

Jeremy Ristau: Is there anything there Beyond upgrading the 311?

Feanil Patel: I think Jeremy's

Feanil Patel: So it'll be 311 for edx platform. It'll probably be three 12 for most other things like edx notes already should be able to run 312. So if you can run that update.

Jeremy Ristau: Yeah, I just mean three eight specific.

Feanil Patel: Yeah. No, I think it should just be a bumping the python version however that needs to happen.

Jeremy Ristau: No. we'll have to have requirements files for 311 and 312 to deal with numpy and stuff like that, but in terms of the work that's happening right now to push all Ideas to 311 or…

Feanil Patel: Yeah.

Jeremy Ristau: 312 depending on whether…

Feanil Patel: Yeah.

Jeremy Ristau: what would take there shouldn't be anything beyond that to deal with dropping support for three eight, right?

Feanil Patel: Yeah, that's right. We just need to make sure that that update is gonna happen reasonably quickly because running both the three eight and three 11 tests and as we're going to talk about the Mongo 4 and the Mongo 7 test means 64 test Suites per PR on edx platform.

Jeremy Ristau: I mean From yeah,…

Feanil Patel: 

Jeremy Ristau: just from my perspective

Jeremy Ristau: RV bomb is fully working on python upgrades, so I don't think there should be any problem that.

Jeremy Ristau: And they're taking the same approach that you mentioned finial of. If there's a numpy dependency,…

Feanil Patel: but yeah.

Jeremy Ristau: it's 311. And if not, it's 312.

Feanil Patel: Yeah, and the summer has been coordinating with me. So I appreciate that as well. I think he's going to start shipping some of the stuff that he was curious about…

Jeremy Ristau: Thanks.

Feanil Patel: which one of us was taking so I think we're in good shape.

Jeremy Ristau: yeah, you know how

Kyle McCormick: And so some of Sydney is a good person to read tattoo for that.

Feanil Patel: Yeah, yeah.

Jeremy Ristau: yeah or so on but yeah, it was almost perfectly fine.

Jeremy Ristau: You can probably call that done.

Feanil Patel: Sounds good.

Feanil Patel: I really wish this task report let you check things up, but there's no feature where it's a roll-up that also lets you update in Confluence at the moment.

Jeremy Ristau: Great.

Feanil Patel: Which is extremely annoying so I'll get that afterwards.

Feanil Patel: Yeah, so real quick coming back. I guess the other things are.

Feanil Patel: I think Kyle already sent out the FYI about ideas that we aren't prioritizing for three 12. I think everybody's seen it so we could probably Mark that as done. The codegel upgrade pull request I reviewed this morning. So that's in really good shape.

Feanil Patel: The upshot is I think to you Jeremy is that we won't break anything on with the free Redwood cut but that there will be a upgrade you'll have to manage with your stakeholders for switching the sandbox python version from 3A to 311.

00:05:00

Feanil Patel: but it will be easier to do than it has ever been to Switch and also roll back, which was not easily possible before.

Jeremy Ristau: Yeah.

Feanil Patel: So I think a really good improvements in that space. I haven't written up this maintaining dock yet. Just because The upgrades have kind of taken over most of my time, but I'll try to get to that after the Redwood cut is my plan.

Jeremy Ristau: is the code you'll upgrade expected to be in Redwood or is it happening after?

Feanil Patel: no, so it'll be in Redwood because three eight will have gone out of support before we cut sumac so we want to make sure that there'll be a Quinn stop text file which will be python 38 compatible and we'll have a SIM link to it from the existing Pi 38 text file and then there will be a redwood dot text file which will be python 311 with the new Sci-Fi at least the new one that is 11 and…

Jeremy Ristau: cool

Feanil Patel: 8 compatible. And then there's a bump from there to the one that's 12 compatible which will happen in sumac.

Jeremy Ristau: so I think what I mean is the risk associated with pushing this in has to happen before the Redwood releases cut.

Feanil Patel: The risk associated with pushing this confused. I don't think I understand what you're saying.

Jeremy Ristau: right now, it's a PR in that PR is not merged. There is risk associated with merging a code jlpr.

Feanil Patel: Yeah.

Feanil Patel: No, no, so the that's merged is going to not make changes to the existing text file. That you guys are using for the sandbox requirements. So those were like the requirements that are currently getting installed. Remain the same. There are just new files that are getting updated which things should start moving to and tutor will move to probably for the default.

Feanil Patel: But that's a jump that won't be automatic. because there's no

Jeremy Ristau: The okay.

Kyle McCormick: to put it another way coachell will be.

Jeremy Ristau: So go ahead come.

Kyle McCormick: Pi 311 compatible just like X platform in time for Redwood, but you won't have to make that jump in time for Redwood.

Feanil Patel: Yeah, and merging that PR won't break anything today.

Jeremy Ristau: Okay, because it isn't changing the name of the existing file. it isn't replacing that file with the different file.

Feanil Patel: It is but we're simulinking the name old name so that you guys don't have to have a breaking. change Yeah.

Jeremy Ristau: I didn't say it was expected to break anything. And so there's a risk of Yeah.

Feanil Patel: I say yes, it's Now yeah,…

Jeremy Ristau: And we have to take that risk before Redwood. Right. Yeah.

Feanil Patel: yes. Yes we do. Yes, but

Jeremy Ristau: I would love to now that we have a maintenance working group. Talk about spreading this risk out over the release. window way more than 72 hours for the car.

Feanil Patel: Yeah. right right yeah,…

Jeremy Ristau: Yeah, okay.

Feanil Patel: I'm gonna put it on here just

Kyle McCormick: full

Jeremy Ristau: Yeah.

Feanil Patel: said yeah, I was gonna do it either So I'm glad you're excited to do that also.

Jeremy Ristau: Awesome it just like, timing situations with

Feanil Patel: Yeah.

Feanil Patel: No, I think historically we waited till the releases to do a bunch of upgrades and that makes no sense to me. it just yeah,…

Jeremy Ristau: yeah, it's painful.

Feanil Patel: I think part of the reason is that there's a cost to running both versions. And so there's an attempt to minimize that cost but I think there are other ways of solving this problem.

Jeremy Ristau: next one

Feanil Patel: Yeah, so real quick the manga of prayer. Do you have any updates on that? Because right now we are running the four four and the Seven tests, but My plan is to not run the 4/4 tests against python 311 just we have 48. Shards instead of 64 shards and testing isn't gonna get completely bogged down and queued up for edx platform.

Feanil Patel: So there's a risk there that as we move forward and drop python 3/8 support. We are not testing with mongo4 anymore. and if you guys haven't upgraded we might introduce issues that aren't being tested anymore before they go to production for you.

00:10:00

Jeremy Ristau: Yeah, I mean, I'm not a part of the Mongo upgrade so. that

Feanil Patel: Yeah. Yeah. This is more of you're the only person that can tell and I would love to for you to share that information with the people who care because I don't want to take down your database anymore than you want us to take down your database, but I also don't want to wait forever for somebody who's not participating in the community to do an upgrade.

Jeremy Ristau: Do you know the person to the new contact for SRE or…

Feanil Patel: 

Jeremy Ristau: these things? Okay.

Jeremy Ristau: So this is the main. edx SRE team member left

Feanil Patel: Okay. awesome Yeah.

Jeremy Ristau: my main I mean Pretty much the only one.

Jeremy Ristau: I don't have a slack handle.

Feanil Patel: And Nadine.

Jeremy Ristau: open at X but

Jeremy Ristau: you should be able to Ping him in to you room and stuff I'll make sure he's in there just zip up that loose end right here.

Feanil Patel: Okay.

Feanil Patel: Yeah, it looks like he's not in here right now. So you might need to.

Feanil Patel: Yeah. Yeah if you Adam.

Jeremy Ristau: I can't add him weirdly.

Feanil Patel: Okay.

Jeremy Ristau: But I will see what I can do for sure.

Feanil Patel: Yeah, I think it's more I'm trying to balance not breaking people with giving them enough time to do the upgrade with I don't want to run multiple versions of mango testing forever.

Jeremy Ristau: Yeah, I totally understand the reasoning behind it. But You…

Feanil Patel: I'll make sure it's to the people…

Feanil Patel: who can do something about it.

Jeremy Ristau: Yeah, you can get lost in translation when I play middle man you.

Feanil Patel: Yeah, that's good. Okay, I see him in there. So I will chat with him.

Kyle McCormick: What else like to connect with the name about to you having a private settings file.

Feanil Patel: Yeah. That sounds good.

Kyle McCormick: I'm realizing now that I don't think they have one. Which makes the river PR that I put up and…

Feanil Patel: No, they're just using production of

Kyle McCormick: thought would be simple not actually simple.

Feanil Patel: Okay, let's talk about that afterwards Maybe.

Feanil Patel: then it's about is the major manga upgrade things. We'll check in with all at an item here. for both of us Kyle

Feanil Patel: python upgrade is going pretty well.

Feanil Patel: edx platform my sort of test Branch I did add three 11 testing for all of the GitHub workflows

Feanil Patel: so the only thing that failed was one of the scripts requirements files, which I think I just need to untangle a little bit and I think it's like a back ports in a script. So it's like nothing in the core platform is failing on that pull request now, so I think as we're gonna merge a bunch of the test fixes probably today and I'll announce that since that one's kind of a slightly major. Change I haven't been announcing most of the python dependency upgrades just because most of them not much changed or things changed test in the library. but for this one, there's major test changes and a couple of major sort of like actual code changes. So I'll announce that when I ship that out hopefully later today.

Jeremy Ristau: Is there a running list of the dependency changes? I think I'm aligned with you on not needing to announce them, but I wonder if a change log is. appropriate for it

Feanil Patel: Yeah, when I make those changes I am updating the relevant tickets and linking them to the relevant PRS. Let me just grab use the

Feanil Patel: So everything is going on here.

00:15:00

Feanil Patel: And the change there and it's usually like I've switched to now basically one PR per change, so it's pretty easy to get in the history as well. So

Jeremy Ristau: Just a second.

Jeremy Ristau: back

Feanil Patel: Yeah, so those have been going. I'm gonna merge those tests and other sort of fixes to get all the tests to Green in 311 shortly, and then I'm thinking I need to rearrange some of the rest of that stuff now that Kyle's PR is really close to Landing so that it's based on top of his stuff.

Feanil Patel: I think that means a back Port there's a handful of changes that need to happen to unpain sci-fi and numpy. For the new files that exist. So we're gonna have to figure that out. But yeah, I think we're getting close. We're getting to the point where we're now running into each other because there's only so many things left to do to get X platform up and running which is good.

Feanil Patel: There's still a number of

Jeremy Ristau: And where do you see that? As people run into each other. It doesn't help to have that many people involved anymore.

Feanil Patel: Muhammad

Feanil Patel: Yeah.

Jeremy Ristau: Who do you see remaining?

Feanil Patel: I think to add to get it landed over the line, it'll probably be me and Kyle who sort of helped do that. I think there's plenty of Library changes that still need to done where I think the Arab team is still super useful work that they're doing. And I think we've been mostly trying to coordinate in either WG maintenance or CCX platform. Could use some feedback on which of those two things make more sense to other people, but they're kind of both. So I've been trying to just share as much as possible and either

Jeremy Ristau: Okay.

Jeremy Ristau: Yeah, I guess my question would be more like at what point in time should our be bomb turn its focus to specific python upgrading rather than open edx python upgrading to prepare for the 3/8 drop

Feanil Patel: Yeah I think that the other services still need to be there's core course Discovery and credentials have open PRS but nobody's reviewed them and I don't know what the state of those are. So once that it's platform is out of the way. That's the place I would focus them and…

Jeremy Ristau: Yeah, if yours exist for yeah,…

Feanil Patel: then after that.

Jeremy Ristau: we're waiting for the owners to review them.

Feanil Patel: Yeah, exactly. Yeah, I think for both course Discovery has a draft one and…

Jeremy Ristau: but Okay.

Feanil Patel: credentials maybe has a draft one also, but I don't know I haven't got a chance to look at them. So I don't know sort of like how mature they are. But that's the place.

Jeremy Ristau: Yeah.

Feanil Patel: I think that's the place to prioritize next after. the edx platform changes

Feanil Patel: and then I think we have the prioritized list of services to sort of work down. in

Jeremy Ristau: Yeah. This is what people have been signing up for. one by yeah

Feanil Patel: Yeah, so the people have been signing up for the edx platform subset and you can see where 36 of 65 packages have been updated and merged. most of them are assigned and just need to in some cases. It's that the person who's done the work can't do the releases. So I've been trying to go and do the releases and sort of unblocking those Small, all the bits of chore that are surrounding a release. I need to go.

Jeremy Ristau: Yep.

Feanil Patel: I think at the end of this I will make some notes about things that we need to automate to make everybody's let's a little easier. after that it is, credentials. Which is blocked by other work, but I don't know. What work.

Feanil Patel: This makes no sense.

Jeremy Ristau: Sorry was that a passing comment like a prioritization situation or an actual blocker like there's different issues.

Feanil Patel: so it looks like there's this credentials one and I just bumped into it. I just jumped into it and it said this is blocked on the docker file upgrade in tutor which doesn't make sense because you should be able to land this master change without Landing the talk of the tutor change.

00:20:00

Feanil Patel: So this is the prioritize list I would focus on after an ex platform. And then these things afterwards. I don't know if you've been communicating with Enterprise and if they have a plan or if they're just gonna deal with these things later.

Feanil Patel: but

Jeremy Ristau: this needs to land before 3/8 is dropped not before Redwood.

Feanil Patel: and I know Ecommerce is what's up?

Feanil Patel: Yeah, yeah. I think the prioritize list is the stuff that needs to land before Redwood and the unprioritized list needs to land before October. So they have a lot more time.

Jeremy Ristau: Or before Kyle wants to remove support for three.

Feanil Patel: Yeah, but yeah,…

Feanil Patel: yeah, one of those two things.

Jeremy Ristau: Which is really plus three.

Feanil Patel: Yeah, yeah.

Kyle McCormick: I think we definitely want that Kyle love and support merged in to those lower repos before Pi 38 goes end of life. Otherwise, we should be talking about why we can't maintain them and whether they should be in the open edx organization.

Feanil Patel: Right, which our conversations ongoing around the Enterprise things anyway, so it'll probably just surface some of the things that we already talked about elsewhere. right, so

Kyle McCormick: we did have someone volunteer from Italy to help with Ecommerce. Let me link that and just

Kyle McCormick: when I find out when I get in the notes, but there's a couple folks said they'd help with that.

Feanil Patel: Yeah.

Feanil Patel: Yeah, an agendas has a couple of folks that they would too. I just have not had a time to connect them to the work that needs to be done.

Feanil Patel: Yeah, that sounds good. I put an item in the discussion real quick. The note upgrade. I think Brian still working on getting through the test now that he can reproduce it locally so still in progress.

Feanil Patel: but that one we don't need We're already late on the 18 upgrade for edx platform. So I don't think it makes sense to hold up Redwood for it, but we will try to land that as soon as possible on platform. And hopefully by sumac we'll have actually gone all the way to 20. So we might end up being at 16 for Redwood and then bumping to versions for sumac.

Feanil Patel: And I think we will need it for students before 16 goes NWI for 18 goes out of life. 18 because 16 is I think already end of life.

Kyle McCormick: So we're saying that we might just accept that we have an EOL node version and edx platform for Redwood.

Feanil Patel: I think yeah,…

Feanil Patel: I think we'll have to

Kyle McCormick: Because of…

Kyle McCormick: how hard the upgrade is.

Feanil Patel: Yeah, This upgrade has taught us a lot about the places where we have coverage for testing in edx platform.

Feanil Patel: But I think that we won't unless Brian it's extremely volatile. he might figure it out today or he might figure it out two weeks from now and I have no idea and I don't know if there'll be another thing that pops up after that. So we just don't have enough I think up front testability of edx platform front end. To be confident that it won't break things because of how complex it is at the moment.

Feanil Patel: All right, and then the next thing I wanted to talk about is

Feanil Patel: ax improvements is probably going to start working on removing edxpecific packages from edx platform.

Feanil Patel: So this is kind of like a inform an FYI. this is specifically. packages where the code still lives in the edx GitHub org but is installed by default in the open edx platform Repository. Which is I think four or five things.

00:25:00

Feanil Patel: That are sort of related to raise and other sort of edx specific tooling. So all of that will need coordination. This is the sort of pre coordination of inform that this is definitely happening. So make sure there's room for dealing with this in whatever schedules you guys are making.

Feanil Patel: and we can

Jeremy Ristau: Okay, so from that perspective. Going to start. is like how do we know what that means?

Feanil Patel: so they're going to start making the PRS and investigating what removals need to happen and all of that will turn into pull requests. I think sometime after the Redwood cut. So we're talking, early May time frame. At which point I think. We won't merge them without sort of doing the communication and planning of when is that? Okay and what the alternatives are? So I think this is for May and June is when that work there should be space for some of that work. If that helps.

Jeremy Ristau: Yeah, I mean I will say that is completely full for us. June might be something…

Feanil Patel: Yeah.

Jeremy Ristau: where we can start to look into that. But the data dog work is just filled up the completely.

Feanil Patel: That sounds good. Yeah, and we'll do our best to work with what some of those might be really easy ones and are small some of those might be more complex, but I think we won't know until they kind of get in there and start pulling them out and see what fails. but Yeah,…

Jeremy Ristau: Muhammad

Feanil Patel: I think. As much as we can we want to start dropping some of those things are making those optional.

Feanil Patel: I'm hoping that for a lot of them because if they're behind feature Flags,…

Kyle McCormick: platform

Feanil Patel: we can at least just drop the requirement and add them to the private requirements to you, which should be an easy change for you guys to make. But for things where it also requires code changes or other things, it'll be more complex. But yeah, we don't know enough about it yet to give you a tighter timeline.

Jeremy Ristau: platform Thank you.

Feanil Patel: Okay.

Kyle McCormick: Still I think this might dovetail with the settings file thing because I imagine there are some. of those packages in the edx org that need hooks In the second calendar to work.

Feanil Patel: Right, right. Just the Yeah.

Kyle McCormick: Okay.

Feanil Patel: So I'll make a Kyle make a note for you and I to check in.

Feanil Patel: Yeah, just so we have that everybody welcome.

Feanil Patel: this is actually nice. This will be a great General conversation to have with everybody. That's spreading Up release risk.

Feanil Patel: And we'll give folks another minute. Not too much longer.

Feanil Patel: And everybody who had to use was in the other meeting, so we mostly went through them. I'll recap you all what we talked about in the next platform meeting real quick. And then we'll get going we chatted about the manga upgrade and The python upgrade notice platform is in pretty good shape at this point. I think we're getting close to Landing all the changes on the platform side the lries there's a bunch of libraries that still need to be upgraded but most of them have people working on them. So just a matter of merging and releasing those and updating the requirements in edx platform. So that's going really well.

Feanil Patel: One of the things that we're going to be working on with the X improvements team at axim over May and June after the Redwood cut is removing some of the Packages out of the requirements in edx platform. So these are specifically packages where the code lives in the edx GitHub organ not in the open edxca network.

00:30:00

Feanil Patel: And then the first thing I want to talk about today is about our sort of upgrade Cadence and spreading out some of this upgrade related risk across more time than just the couple weeks before the release. I think there's sort of two components to this. and also let me Please put yourself in here and help take notes.

Feanil Patel: So yeah, I think.

Feanil Patel: We have a lot of knowledge I think about what things are going to be end of life, but I don't think we're using that to plan. as much as we could be So between end-of-life dates for the sort of Baseline. items and the end of life stuff for edx platform

Feanil Patel: the maintenance windows

Feanil Patel: What is it Support Windows? Yeah.

Feanil Patel: so if we look at our support Windows

Feanil Patel: I think between this and we actually have a bunch of Redis were good throw but elasticsearch. We're still on an extremely old version and we need to resolve this and hoping that the melee a search. related Spike that is happening right now will yield good information and we'll hopefully be able to invest in that elastic search upgrade and Define what it should look like one of the problems with the elasticsearch upgrade has been that there hasn't been cleared sort of decision making around what should happen to the various places that use less search. So I'm hoping that we can move to Malia assertion sort of Define what that work looks like.

Feanil Patel: And I need to update this. We're on Ruby three three now. so All right, because Ned used to update this. That's why it's not up to date.

Feanil Patel: But yeah, I think I would love to hear people's thoughts about. Spreading out the releases further. I think that rather than trying to do them. So close to the open and next releases trying to get updates in sort of sooner to when they are available makes more sense to me.

Jeremy Ristau: I mean, I'll just say I agree because I brought up this topic earlier. obviously as an instance that deploys off of Master being able to spread these out across the time window is just easier to plan for Less risk in any one particular month or week or quarter or whatever?