Summary

Project Updates and Initiatives

• John Swope is preparing to release a micro platform in open beta and working on open-source licensing.

• AI projects are being developed for academic validation and feedback, saving faculty time.

• Sarina Canelake discusses code automation for student submissions and its benefits.

Documentation and Pull Requests

• Sarina Canelake and John Swope discuss linking OX documentation and organizing course components.

• Sarina has open pull requests for documentation updates and seeks feedback from attendees.

• Maintenance tables and hackathon documentation fixes are being addressed.

Testing and Review Processes

• Discussion on establishing a system where the person fixing an issue is not the one testing it.

• Sarina and John discuss reviewing pull requests and ensuring documentation accuracy.

• The need for a script to check maintenance table entries is highlighted.

Collaboration and Communication

• Sarina plans to collaborate with Chelsea on aligning testing processes with documentation.

• Ana Gomez and others discuss the importance of involving more people in the testing process.

• Sarina emphasizes distributed ownership within the working group to empower the community.

Conference and Future Plans

• Sarina plans to improve LTI integration documentation before a conference talk.

• Discussion on identifying contributors for documentation roles at the conference.

• Plans for a testathon and further hackathon initiatives are mentioned.

Documentors WG: Working Session - 2025/04/16 09:58 EDT - Transcript

Attendees

Ana Gomez, John Swope, Peter Pinch, Sarina Canelake

Transcript

Sarina Canelake: What have you been up to?

John Swope: What have I been up to? I am getting ready to release the latest on micro platform as in open beta so folks can start signing up and…

Sarina Canelake: Nice.

John Swope: 

John Swope: using it. and I'm trying to get through some of the licensing stuff so I can release it open source. which I think it'll be done pretty soon. That's been taking up a lot of time. And then I've got a decent number of people that have got me working on projects around what I would call this concept like validation or AI co-assessment. And it's basically this idea that there's a whole lot of stuff we do in academia where people submit things. So students will submit something and…

Sarina Canelake: Right. Right. Right.

John Swope: it's scored according to a rubric often. and 80% of that rubric is kind of like did you follow the instructions? is it the right length? Is it the right structure? Did you cite your sources etc. you don't need to have a academic grade, those kind of things. and not only that, but you want to give the student a fair warning of, hey, this thing is missing.  And so there's a decent amount of projects that involve having AI do that work both as a feedback machine for students before they submit so they know they're submitting something that's quality and also saving the faculty time where they say okay the stuff that is low order thinking I don't have to spend all my time doing basically. Yeah.

Sarina Canelake: Not writing the same damn feedback for every single submission.

John Swope: And it takes different forms. it's not just like, " I'm submitting an essay." we're doing one with a school that's trying to place their students into the right English course. So, we're actually preassessing their reading and…

John Swope: writing and giving them a suggestion and they can disagree, but we're, doing that preassessment. students submitting portfolios. So, that's been something that's been popular with folks that I talked to.

Sarina Canelake: That's really cool.

Sarina Canelake: Yeah. I mean,…

John Swope: Yeah. Yeah.

Sarina Canelake: it kind of reminds me of what we got with a lot of code automation. when I was a TA, we just had put in code automation so students could at least check that their submissions ran the unit tests, because they were supposed to write their own tests, but then they could submit.

Sarina Canelake: I think the way we set it up was you had three submissions and we ran secret tests and we just told you the output and because part of it is learning to write your own unit tests too. But it was good for students who would submit the first time and…

Sarina Canelake: be like wow I failed 100% of these tests I need to go back and really rework my code.

John Swope: Yeah. Yeah.

John Swope: Yeah. Mhm.

Sarina Canelake: So that saved us a lot of time and helped us grades we already knew in advance what tests had passed and what tests had failed and we could really focus in on giving the right feedback. So yeah that makes a ton of sense. as far as docs go I have some open PRs.

00:05:00

Sarina Canelake: I don't know if you want to look at them together.

Peter Pinch: Hello.

John Swope: Yeah. …

John Swope: I would be interested in doing that in this time.

Sarina Canelake: Hey,…

John Swope: Hey, Peter.

Sarina Canelake: Peter reviewed one of my PRs and I kind of wanted you to take a look at the other one.

Sarina Canelake: Let me just share my screen real quick. So this one is just adding the also tables. I tried to do my best at adding also tables that were mostly just linking all the ox documentation together. but for some of them you'll see the course assets, right?

John Swope: Yeah, sure. Yeah.

Sarina Canelake: I did link to the course assets reference even though that's going to refer to everything in studio cuz so where I could link to a definition of something that we're talking about I tried to do that is taxes yet,…

John Swope: Yeah, OX is a bit of a different beast than educators because if I remember, is just one block of all the migrated OX content. there's not really navigation within that.

Sarina Canelake: but I figured this was a lower lift than splitting things out into diet taxes was just at least linking all the articles together.

John Swope: Did you just kind of eyeball like, okay, these five articles belong with short description.

Sarina Canelake: Yeah, for the most part I kind of did the same ones because I didn't know where people would land.

Sarina Canelake: The first one I just did is like…

John Swope: Yeah. M.

Sarina Canelake: what is Ox because I was doing some searching on the site and if you search for certain things you'd get the OX stocks in the first three results and I wanted to have a see also for all the OX docs that was just like what am I looking at in case landed on that page inadvertently.

John Swope: Mhm. Yeah. Yeah.

Sarina Canelake: So the kind of like what this would be these first three what is open what is an ox what's an example of an ox course and getting started and then other ones for the course assets I added the course assets but also what is the course asset policy

John Swope: Yeah. Is there a way to get to OX from the homepage?

John Swope: I'm looking through right now.

Sarina Canelake: Yeah. …

Sarina Canelake: and not from the main homepage, but if you go to the ed that's a quick start. If you go to the educator's home, we do have a card for Ox.

John Swope: Yeah. Okay. I think it's a good first pass.

Sarina Canelake: So I don't know if there was something more that you wanted me to do for these CLSOs or if this is a good enough first pass.

John Swope: I think that so we do have some organization around OX So if you go to the Ox homepage like we have policies, organizing course components.

Sarina Canelake: Right. Okay.

John Swope: So the way that we kind of I think imagine this behavior-based navigation is someone is saying okay I want to do an action and so I want to build course components with ox. So, I think having what do you call the basic three what is all X is good and then adding anything that's in the navigation section.

Sarina Canelake: Yeah.

John Swope: So they can be like okay I'll click on discussion next I'll click on video doesn't work quite as well as ox as I go through this but that's what we did for educators where it works I think pretty well because it's very behavior-based. so I think that this commit can go through.

John Swope: I can look at the PR and then maybe if you want to open an issue around updating the navigation updating the CL also links to also include whatever else is in that navigation section. Does that make sense?

Sarina Canelake: Yeah, I mean I can just add that in this PR I think.

John Swope: And…

Sarina Canelake: Yeah,…

John Swope: then the exercise is too huge. So I think that wouldn't make sense. But Okay. Okay.

Sarina Canelake: For the course structure overview I did link to the components and activities table of contents. So It would just bring you directly to I think this section.

Sarina Canelake: So that you'd just be able to see the whole list of what we have any other. course link to math expression input as well as the math expression input XML. So I did try to for articles that were a little bit more specific to something like the math expression input just be sure to look this is…

00:10:00

John Swope: Mhm. Okay.

Sarina Canelake: what it is and here are some of the references for the XML and yeah okay I'll take another pass at this and see if there are things in the table of contents

Sarina Canelake: that would be good to link to for some of these articles. and then I have a couple of other poll requests. So, I did a poll request where let's see, what did I do for this one? I had the diet taxis tags, the maintenance tables, failed some of them that I knew had tickets.  And then there was a pile of OX stacks that were just like placeholder to be written and I'm like I don't think we'll ever write these.

Sarina Canelake: And Peter took a look at this and I think, had a question So, I'm going to see also tables into this branch and then merge this branch.

John Swope: Yeah. Yeah.

John Swope: Great.

Sarina Canelake: And then I did some pull requests to just do hackathon docs fixes.

Sarina Canelake: I split them up by commits and I addressed a bunch of issues. I don't know what extent these need to be reviewed. Jennifer Aana's bug reports were amazing.

John Swope: Mhm. yeah,…

Sarina Canelake: And for Jennifer's I followed her her lead but I did go on a sandbox and test that all the steps worked.

John Swope: I think just as a sis first is great. I mean as a system I think that someone reports the issue,…

John Swope: you fix the issue, someone else should test and pass the issue. So how will that work in So you fixed them.

Sarina Canelake: Okay. Right.

Sarina Canelake: So there was you can review it commit by commit.

John Swope: And you did pass them looks like. So at the bottom there.

Sarina Canelake: Yeah. Because I went through and tested it. So for this one the failure was simply it's like this works…

John Swope: No. Yeah. Yeah.

Sarina Canelake: but you need to enable LTI components was the feedback. So can you add that as a first instruction? So that was a relatively easy fix. we didn't have any documentation in the course researcher role so I added references to the course data researcher role so this is a longer commit.

John Swope: Yeah. Yeah.

Sarina Canelake: But then added I mean this is how I actually just described the role because that's all it is. It's just allows you to see a single tab on the document.

John Swope: Yeah. Yeah.

Sarina Canelake: So I did pass these because I addressed the failures.

John Swope: And we don't have a system. So, I think that's I mean I think if we try to lay out a real system for this someone opens an issue someone fixes an issue and they make a PR if they do that so let's say so let's walk through it for a second so let's say that you fix an issue you create a PR that fixes an issue and…

John Swope: then you say John can you test this would you create that in the PR or…

Sarina Canelake: Yeah. …

John Swope: in the issue cuz I can't add to your PR. I don't know be kind of weird, I have to pull your PR run.

Sarina Canelake: so you don't have to add.

John Swope: That's Yep.

Sarina Canelake: So, this is the best thing to do is when you go into the files view, you press this plus to make a comment and then this button here to add a suggestion. You can just type whatever you want to add you didn't like header bar. You could say I just want this to be the header. Just fix the sentence there and…

John Swope: Yeah. Yeah.

Sarina Canelake: add the comment and it'll allow me to just apply your changes.

John Swope: That's good to know. but then also if I say I want to pass something in the maintenance table,…

John Swope: I guess I could add that as a comment. could I like if I want to pass something in your PR?

00:15:00

Sarina Canelake: Yeah. Would you So,…

Sarina Canelake: I added pass to all of the documents I reviewed. I did already add a pass.

John Swope: Yeah. Mhm.

Sarina Canelake: If you wanted to say add your name,…

Peter Pinch: Have an excellent

Sarina Canelake: you could just make a comment, but then you'd have to adjust the table, too. Yeah.

John Swope: Yeah, it's interesting. I can play with that.

Sarina Canelake: I mean, Mhm.

John Swope: Cuz I mean it would just be nice to have a system at least officially listen if it's you and me and stuff and not a big deal but at least officially …

John Swope: where the person who's fixing is not the person who's testing.

Peter Pinch: I mean the…

Peter Pinch: if you don't agree with the fix, if you're reviewing the PR and you leave comments on the PR or you don't approve it, that's not going to get merged.  So if you're reviewing Serena's PR, it's not going to get merged until you are done reviewing it and say that it's okay. Yeah.

John Swope: So, my review would involve testing because I mean often if I'm reviewing a bunch of documents, I'm not testing all those documents generally,…

John Swope: but we're saying in this case where a fix has been made, that's The review is like this test should work.

Peter Pinch: I mean, if I were reviewing this and I see that Serena's changed the fail to a pass, I would assume that that means that I'm agreeing that it's now passing.

John Swope: Okay,…

Sarina Canelake: Yeah. …

John Swope: that's simpler. All right, I'm happy with that.

Sarina Canelake: so I don't know if somebody wants to take some time to review both of these PRs that have some hackathon fixes. they're kind of big because I just lumped a bunch of fixes together. yeah.

John Swope: Yep. I'll take this one.

John Swope: This is a 103. Mhm.

Sarina Canelake: And then I also have one that's 104.

Sarina Canelake: I think Usama approved it, but it was unclear to me if he just read through it or if he actually tested it. I don't know. Anna, do you have any time to review 104?

Ana Gomez: Yes, I can do it. Sorry for the delay and how missing I have been these last weeks.

Ana Gomez: Right now I'm in a car attending to the meeting because that's how life has been crazy these days. So yes, I can do that. what is the number of the PR?

Sarina Canelake: I'm just tagging you in it,…

Sarina Canelake: so hopefully you'll get an email.

Ana Gomez: Yeah. Yes. Yes. Yes. I will review it.

Sarina Canelake: Okay, thank you. All right, that gets those guys reviewed. yeah,…

John Swope: Cool. …

Sarina Canelake: so those are the ones that I have.

Sarina Canelake: We have somebody who is adding emoji for us. So, I'm excited to work with them. They're brand new contributors. cool. and…

John Swope: we don't have any report on test coverage for these docs.  Now that we have this maintenance table, I mean, it's not the same but now that we have this maintenance table, we don't have any report of how many have been passed and how many are failing at this point, right? Yeah. Yeah.

Sarina Canelake: how many maintenance tables are just empty? Yeah.

Sarina Canelake: I could take a look at writing a script that would parse all of the files and maybe we could just instead of running it maybe on every PR,…

Sarina Canelake: we run it on a cron job.

John Swope: Yeah, I can take that on.

John Swope: I wasn't asking to put something on your plate. it's something that I wanted to do with those maintenance tables in the beginning and now that we have some of them ready and…

Sarina Canelake: So, I think all the docs should have maintenance tables now once I merge the ox1. Yes. So I think…

John Swope: have them. But are they empty? Yeah. Yeah.

Sarina Canelake: what your script would want to do is look at the first entry in the maintenance table and say is it empty?

Sarina Canelake: Is it fail? Yeah.

John Swope: Yeah. Yep.

John Swope: Agreed. I didn't hear and I need to sync up with LRA too. First of all, apparently LRA said she tested her email to me was that she tested everything and I said I don't see everything. So I'll sync up with her and see what she says.

Sarina Canelake: So, I was noticed when I was fixing some of Lear's issues, the maintenance table wasn't filled out with the fail.

John Swope: All right.

Sarina Canelake: I will also say…

John Swope: I will

Sarina Canelake: if you can give her the feedback,…

Sarina Canelake: she's not providing the full path to the file that's broken. So, it's sometimes it it's just annoying to try and find where that file is. and some of her responses simply the instructions don't match what I see. And I'm like, okay. …

John Swope: Okay. Yeah.

John Swope: Yeah. Yeah. Yeah.

00:20:00

Sarina Canelake: some more descriptions and some screenshots of what she does see would be really helpful. that kind of helps me narrow in. a number of her things are unfortunately just she didn't have the feature flag enabled and so that I'm catching that in the docs, too.

Sarina Canelake: I'm being like, "Okay, you need to have your system administrator enable this feature flag." if that was what LRA came up with testing, I was like, "Okay, clearly we need some documentation on just that some sort of setting or…

Sarina Canelake: plugin needs to be enabled.

John Swope: Right. Yeah.

John Swope: And it would be nice I think in the docs like to be clear whenever there is a feature flag.

Peter Pinch: Yeah.

John Swope: Okay.

Sarina Canelake: So for all of the ones she's failed in that way, I've added the note in a callout box. It's like have your system administrator enable this flag or install in. for notes, you have to install a plugin for example. Yeah.

John Swope: But yeah, taking both those full path screenshots and then I guess I can talk about the flags and stuff, but I don't think she'll understand.

Sarina Canelake: No, I actually think it was fine to just be like, I can't get this to work.

John Swope: Yeah. Yep.

Sarina Canelake: Then that was enough for me to be let me figure out what the flag is and then I did the additional work once I got the flags turned on to actually update the screenshots in the dock and stuff. yeah how do we want to go about getting more docs tested? I think the docs hackathon was really successful.

Sarina Canelake: TEK is coming up. So, it could be a good opportunity maybe when Tik is kind of stabilized to do another hackathon or…

Sarina Canelake: something to try and get some more docs updated but with peak instructions. yeah.

John Swope: I mean as part of the release cycle,…

John Swope: we do have that testing sheet which is separate and…

Sarina Canelake: Yeah. Yeah,…

John Swope: 

John Swope: that's separate from our docs. But I think that has in my opinion always been the holy grail of this to sync up that testing sheet with our testing docs because they should say the same thing arguably.

Sarina Canelake: for sure.

John Swope: So I talked about this a long time ago with not a long time, a few months ago with Alice Neo and someone else.

John Swope: I think it was going to be the release coordinator. and they said, "Sure, next time a release comes along, let's sync up again and basically update the sheet to point to our docs as opposed to kind of, oneline instructions." so not about that.

Sarina Canelake: Have you ever talked with Chelsea?

John Swope: No. Does she work on releases or…

Sarina Canelake: Yeah, Chelsea's in the process this week of getting our tests looking better. she's the product leazison for the release.

John Swope: Okay.

Sarina Canelake: So, I would follow up with her because I think she's currently in process of updating a lot of the tests.

John Swope: I will reach out to her right now. Yeah, maybe she was on that, but let's see. no. She was on that document.

Sarina Canelake: Yeah, I'm not sure that it might be good to flag it anyway.

John Swope: So, this is in February with code cut off in early April.

Sarina Canelake: Yeah. Yeah.  She's

John Swope: It would make sense to sync anywhere from to mid It's April 16th. yes, that was the last email. All right. Is Slack usually good for her? All right.

Ana Gomez: Yes, also Maho from ADX, she's working really close to the BTR group and all the releases process. So I think she can also be really helpful for this process to bring all the maintenance tables and synchronize all the documentation process with the testing process to bring more force and strengths to this initiative.

Sarina Canelake: Yeah, I think I would Yeah,…

Ana Gomez: I talked with her and she was really interested. Yes.

Sarina Canelake: Mah's got a lot on her plate for the release. I think that starting with the people who are writing the tests would be Chelsea and Alice Neo is probably the right place to start. because they're in the process. I don't think Maho is actually writing the product tests. Yeah. Yeah.

Ana Gomez: No, no, she's more in the process of bringing people to the test process.  So, no, not for her writing the process, but more for her inviting more people to the test process and having everybody in the same page

00:25:00

Sarina Canelake: And Chelsea's running a testathon during our product working group on May 6th.

Sarina Canelake: So, we're hoping we'll get a lot of people to help test then and that could be a good opportunity to get some of the docs filled out, too. Perfect.

John Swope: All right, I just ping Chelsea.

Sarina Canelake: So yeah, getting that as the part of the release process would be great. I would love to. I'm gonna try and fix some more hackathon issues this weekend.

Sarina Canelake: And also I just don't have time this week, Peter. I did see your Ox course, so you were also working on a Saturday.

Peter Pinch: Yep. Whatever.

Peter Pinch: Yeah. Yeah.

Sarina Canelake: It was terrible weather. I was like, I'm just gonna sit with my kittens, write some docs.

Peter Pinch: It's definitely procrastinating in my case it was procrastinating something else I didn't want to do. It's great.

Sarina Canelake: Yeah. Yeah.

Sarina Canelake: Yeah. I totally get it. my kittens were stoked that I was just sitting on the couch and not moving.

Peter Pinch: Somebody gets to enjoy it.

Sarina Canelake: Yeah, exactly. Somebody got a benefit from it. All right. does anybody else have anything they want to talk about?

John Swope: This group doesn't have a group lead,…

John Swope: You're kind of deacto Serena, but x doesn't like being leads on these working groups.

Sarina Canelake: Yeah, I mean that's,…

Sarina Canelake: I think kind of as a philosophy thing like Axim doesn't want to be in charge of too much because we want the community to feel empowered to

Sarina Canelake: own things and…

Sarina Canelake: I don't want it to be the case where it's like Serena's from Axom so whatever she says is right that's not true whatever I say is my opinion it doesn't mean it's not challengeable so I think that's kind of where our perspective comes in I don't mind running this group but I kind of feel like it's maybe more of a distributed ownership we all show up to these meetings you and Anna run the hackathon I think…

John Swope: Yeah. Yep.

John Swope: Yeah. yeah.

Sarina Canelake: if we all take a little bit of ownership for various pieces. that will be sufficient.

John Swope: I wonder if the conference will be a good time to just identify folks who are maybe looking for things and maybe have contributed documentation and want to take on a role just to I don't know the notes and tracking and things like that.

Sarina Canelake: Yeah, for sure.

John Swope: there. Mhm.

Sarina Canelake: And getting more people to just be comfortable.

Sarina Canelake: I really like the documentation you guys wrote on just how to file an issue about the documentation. I think that was super helpful during the hackathon and we got so many people who just filed issues and It's not really a problem for someone who knows how to write the docs to follow up on those and…

John Swope: Okay, And nothing else for me right now. I have two tasks coming out of this script for maintenance coverage and then working with Chelsea on the release process and aligning that with docs.

Sarina Canelake: looking at my PR.

John Swope: Yes. NPR 103.

Sarina Canelake: Yeah. Awesome.

John Swope: Okay. Peter,…

Sarina Canelake: All right.

Sarina Canelake: thanks everybody and have a great rest of your day.

John Swope: MIT is not under any fire yet for being woke or stuff. …

Peter Pinch: it hasn't hit the papers.

Peter Pinch: …

John Swope: it's Yeah.

Peter Pinch: but there's a lot of nervous glances and that kind of stuff. it doesn't help that they're changing for existing contracts they're changing the overhead rate.

John Swope: Yeah. Yep.

Peter Pinch: So everything that we do for NIH or Department of Energy now so if you don't want the nuclear reactor to go critical we got to do that at half the rate now.

John Swope: Yeah, I did.

John Swope: I didn't understand it all, but I did hear a little about that. But been thinking about you guys…

Peter Pinch: Yeah. Yeah.

John Swope: because everyone else is in the news. I'm like, I don't know. they skipped MIT makes computers.

Peter Pinch: I mean we share with everybody right?

00:30:00

John Swope: Yeah. All right.

Peter Pinch: …

John Swope: Cool.

Sarina Canelake: I tangential.

Sarina Canelake: This is related to Peter, I reached out to Caleb and Meredith and Mary Ellen. They're giving a talk on using Canvas with open edex at the conference and I really would love to get all of our LTI integration docs into tiptop shape prior to that doc that talk. then yeah,…

Peter Pinch: you mean on the unsupported feature?

Sarina Canelake: the one that says it's unsupported that you're g your team is giving a talk on.

Sarina Canelake: I haven't heard back from them. Granted, I sent the email Monday evening.

Sarina Canelake: But if they don't reply to me in a week. Is there anybody else I could add to that thread?

Peter Pinch: I mean you could add me a lot of the tech work on that so I mean the original work was done by Harvard by people…

Sarina Canelake: Okay. Okay. Mhm.

Peter Pinch: 

Peter Pinch: who aren't there anymore. But keeping it alive and adding the features that they're going to talk about was done by Opencraft. So in terms of documentation it might be Caleb's probably the best person. but there's a part of it about the operational part that OpenCraft may have already made something.

Peter Pinch: I'm sure they wrote something for us. I would have to put my hands on.

Sarina Canelake: Yeah, I think…

Sarina Canelake: if there's any documentation on your side that could be dug up or a contact at open craft. And I think there's two parts of the documentation. There's the instructor facing and then the technical facing.

Peter Pinch: Yeah, exactly.

Sarina Canelake: And so getting both pieces into shape.

Sarina Canelake: I think it would be great if at the conference at the end their slides could be like and hey here's the documentation that we know works okay yeah and…

Peter Pinch: Yeah,…

Peter Pinch: totally agree with that. And, yeah, I'll try to help. It's good to know that you asked.

Sarina Canelake: I did work with Mary Ellen on some other docs and I really enjoy working with her so absolutely…

Peter Pinch: Yeah. I mean, they're all great. Mary Ellen's probably the busiest, but that sounds like Mary Allen.

Sarina Canelake: but we had a great time

Sarina Canelake: we fixed up three docs, but she was like, "This is these docs bother me the most. I want to fix them. Let's do it." …

Peter Pinch: That's one of the things I'll be honest.

Sarina Canelake: so yeah, with any other documentation that team finds that they think is wrong, they can always reach out to me and I can get it fixed. They don't have to I told Mary, Just put what you want fixed in the Google doc and I will fix it for you. Yeah. I Yeah.

Peter Pinch: I think there's also just probably some reticence be because it's not a supported feature. whatever that meant, that was a decision that was made by people at edex 10 years ago. Mhm.

Sarina Canelake: And I think if it works and there's documentation and this is the other thing I will test it on our sandboxes.

Sarina Canelake: And given that loop, I think we could always say …

Sarina Canelake: this feature isn't in wide use or something like that that indicates,…

Peter Pinch: Yeah. I mean,…

Peter Pinch: you just qualify it somehow.

Sarina Canelake: yeah, just adding some qualification. But it says it's unsupported and yet I know people are using it. So unsupported is the wrong word. Of course.

Peter Pinch: Yeah. the folks that you've talked to on our end probably don't feel comfortable changing that language. So if you encourage them and honestly saying that you'll test it in the sandbox at some point in the future is a great thing to offer and confidence building and I think would Caleb in particular knows what the value of that great.

Sarina Canelake: Okay, cool.

Sarina Canelake: if I don't hear back from them in a couple days, I don't want to push too hard, but I will reply and offer All right, everybody. Enjoy the rest of your day.

John Swope: Thanks.

Peter Pinch: Okay, thanks

Ana Gomez: Have a great day. Bye.

Meeting ended after 00:34:27 👋

This editable transcript was computer generated and might contain errors. People can also change the text after it was created.