[BD-19, BD-20] Elasticsearch & Haystack Upgrades

Owned by Nimisha Asthagiri (Deactivated)
Last updated: May 16, 2022 by Wes Mason [he / him / his] (Deactivated)Version comment
4 min read

 

Blended Project ID

BD-19, BD-20

edX Product Lead

@Nimisha Asthagiri (Deactivated)

edX Technical Primary

@Diana Huang (@Michael Terry (Deactivated) as secondary)

Provider

Raccoon Gang

Provider Lead

@Igor Degtiarov

Calendly Scheduling Link

calendly.com/nasthagiri 

Slack Channel

  • external-openedx-edx-raccoon-gang (on edX side)

  • edx-shared-raccoon-gang (on Open edX side)

  • shared_open-edx (on RG side)

Meeting Cadence / Preference

  • Cadence

    • First 2 weeks: Every week (in June)

    • Then 2x a week (in July)

    • Then 1x a week (in Aug)

  • Times: 10am-12pm, ET; 5pm-7pm RG Time

Action Item Tracking

Use slack first, can also add action items on project status pages or provider status page

Escalation Path

 

Relevant Discovery Documents

[BD 19-20] Technical Plan (Haystack Replacement, Elasticsearch Upgrade)

Status Update Frequency

weekly

Target Completion Date

August 2020

~Dev Hours Per Sprint Expected

20

Total Hours Used

BD-19 100% - 10/20

BD-20 100% - 1/12

Blended Project Epic
(Tracks pull requests )

  • https://openedx.atlassian.net/browse/BLENDED-188

  • https://openedx.atlassian.net/browse/BLENDED-189

Open edX Jira Epics

 

Open Action Items

Status Updates

Time Frame

Updates & Actions Needed

~Hours

Time Frame

Updates & Actions Needed

~Hours

Dec 16, 2020 - Jan 12, 2021

We are our of the budget because of unknown unknowns.

Work done this sprint:

  • course-discovery released to the production

  • edx-analytics-piple – paused

Work in progress:

  • new issue for course-discovery still in progress

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 39 hours

BD-20
~475/475
100% - 1/12

Out of the budget - 37 hours.

Dec 2, 2020 - Dec 15, 2020

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 21 hours

BD-20
~472/475
99% - 12/15

Nov 18, 2020 - Dec 1, 2020

Work done this sprint:

  • course-discovery Improvement work: several PRs were proposed

  • edx-analytics-piple travisCI was fixed.

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 25 hours

BD-20
~451/475
95% - 12/1

Nov 4, 2020 - Nov 17, 2020

The extended (in September) scope was corrected on the budget that add + 75h to projects. BD-20 volume was updated to 475h

Work done this sprint:

  • course-discovery Improvement work is near the finish line.

  • PR for edx-analytics-piple was prepared.

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 35 hours

BD-19
100% - 10/20

BD-20
~426/475
89% -11/17

 

Oct 21, 2020 - Nov 3, 2020

Work done this sprint:

  • course-discovery Improvement work is in progress, several new fixes were proposed that addresses found issues.

  • PR for cs_comments_service was deployed on Prod!

Work in progress:

Support for both projects during manual testing, live session testing, etc.

Note:

We didn’t have precise efforts splitting between the projects. All-time was divided equally between projects, but since ВD-19 is full all efforts are tracked to BD-20.

We’ve spent in sprint 54 hours

BD-19
~280/280
100% - 10/20

BD-20
~391/400
98% -11/3

Oct 7, 2020 - Oct 20, 2020

Work done this sprint:

  • course-discovery has been passed several test sessions. Several fixes were proposed.

  • PR for cs_comments_service was merged new fixes were proposed as well.

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 42 hours

BD-19
~280/280
100% - 10/20

BD-20
~337/400
84% -10/20

Sep 23, 2020 - Oct 6, 2020

Work done this sprint:

  • PR for edx-analytics-pipeline was created, some minor improvements for coverage are required.

  • course-discovery has been passed several test sessions. Several fixes were proposed.

  • PR cs_comments_service is actively reviewed and tested by edx. Several fixes were proposed during the sprint.

Work in progress:

Support for both projects during manual testing, live session testing etc.

We’ve spent in sprint 92 hours

BD-19
~260/280
93% - 10/6

BD-20
~315/400
79% -10/6

Sep 9, 2020 - Sep 22, 2020

Work done this sprint:

  • Local PR for edx-analytics-pipeline was reviewed and approved.

  • Bugs were found in course-discovery, PR with fixes was created

  • Local deployment issues with cs_comments_service and edx-search were investigated and solved.

Work in progress:

Support for both projects during manual testing, live session testing etc.

  • BD - 20

    • ES upgrade - edx-analytics-pipeline – is ready to be forward to the upstream.

We’ve spent in sprint 39 hours

BD-19
~220/280
78% - 9/23

BD-20
~263/400
65% - 9/23

Aug 26, 2020 - Sep 8, 2020

Scopes of the projects was extended, details in the [BD 19-20] Technical Plan (Haystack Replacement, Elasticsearch Upgrade)

Work done this sprint:

  • The retrospective was provided to both blended projects.

  • Local PR for edx-analytics-pipeline was opened.

  • PR for course-discovery in upstream was created

  • Work during the review process on the upstream PRs for cs_comments_service, course-discovery, and edx-search.

Work in progress:

  • BD - 19

    • haystack replacement - in the test/review phase

  • BD - 20

    • ES upgrade - edx-analytics-pipeline – in the review phase

We’ve spent in sprint 46 hours

BD-19 (additional efforts were not estimated 280h is temporal value)
~204/280
73% - 9/8

BD-20
~240/400
60% - 9/8

Aug 12, 2020 - Aug 25, 2020

Work done this sprint:

  • Active review work and improvements in local PR for course-discovery

  • Local PR for cs_comments_service was approved

  • Create PR for cs_comments_service in upstream.

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery - in the review phase

  • BD - 20

    • ES upgrade - cs_comments_service – in the review phase

We’ve spent in sprint 60 hours

BD-19
~189/200
95% - 8/26

BD-20
~209/240
87% - 8/26

Jul 29, 2020 - Aug 11, 2020

Work done this sprint:

  • Created PRs for edx-search in upstream.

  • Create local WIP PR for course-discovery

  • Create local PR for cs_comments_service

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery - in final dev phase

  • BD - 20

    • ES upgrade - cs_comments_service – in the final testing phase

We’ve spent in sprint 104 hours

BD-19
~169/200
85% - 8/11

BD-20
~169/240
70% - 8/11

Jul 15, 2020 - Jul 28, 2020

Work done this sprint:

  • Created local PR for edx-search.

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery - in active dev phase

  • BD - 20

    • ES upgrade - cs_comments_service

We’ve spent in sprint 90 hours

BD-19
~117/200
58% - 7/28

BD-20
~117/240
48% - 7/28

Jul 1, 2020 - Jul 14, 2020

Work done this sprint:

  • Created PRs in edx-notes-api and edx-analytics-data-api libraries.

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery

  • BD - 20

    • ES upgrade - edx-search

We’ve spent in sprint 108 hours

BD-19
~72/200
36% - 7/14

BD-20
~72/240
30% - 7/14

Jun 16, 2020 - Jun 30, 2020

Work done this sprint:

  • Team kick-off and onboarding

  • Took first two libraries edx-notes-api and edx-analytics-data-api into upgrade work

  • Updated ES to latest

Work in progress:

  • BD - 19

    • haystack replacement - edx-notes-api

  • BD - 20

    • ES upgrade - edx-analytics-data-api

We’ve spent in sprint 36 hours

 

 

 

 

Relevant Jira Stories, Tasks, Bugs

This section will link to all the development stories, tasks, and bugs connected to the single (or multiple) epics used to track the work on the Open edX JIRA site.

key summary type created updated due assignee reporter priority status resolution
Refresh

Relevant Pull Requests linked to Blended Project Epic

This section will link to the edX internal Blended project epic, and contains all Pull Requests related to this Blended Project.

key summary type created updated due assignee reporter priority status resolution
Refresh

key summary type created updated due assignee reporter priority status resolution
Refresh