/
[BD-19, BD-20] Elasticsearch & Haystack Upgrades

[BD-19, BD-20] Elasticsearch & Haystack Upgrades

 

Blended Project ID

BD-19, BD-20

edX Product Lead

@Nimisha Asthagiri (Deactivated)

edX Technical Primary

@Diana Huang (@Michael Terry (Deactivated) as secondary)

Provider

Raccoon Gang

Provider Lead

@Igor Degtiarov

Calendly Scheduling Link

calendly.com/nasthagiri 

Slack Channel

  • external-openedx-edx-raccoon-gang (on edX side)

  • edx-shared-raccoon-gang (on Open edX side)

  • shared_open-edx (on RG side)

Meeting Cadence / Preference

  • Cadence

    • First 2 weeks: Every week (in June)

    • Then 2x a week (in July)

    • Then 1x a week (in Aug)

  • Times: 10am-12pm, ET; 5pm-7pm RG Time

Action Item Tracking

Use slack first, can also add action items on project status pages or provider status page

Escalation Path

 

Relevant Discovery Documents

https://openedx.atlassian.net/wiki/spaces/COMM/pages/1413810945

Status Update Frequency

weekly

Target Completion Date

August 2020

~Dev Hours Per Sprint Expected

20

Total Hours Used

BD-19 100% - 10/20

BD-20 100% - 1/12

Blended Project Epic
(Tracks pull requests )

Open edX Jira Epics

 

Open Action Items

Status Updates

Time Frame

Updates & Actions Needed

~Hours

Time Frame

Updates & Actions Needed

~Hours

Dec 16, 2020 - Jan 12, 2021

We are our of the budget because of unknown unknowns.

Work done this sprint:

  • course-discovery released to the production

  • edx-analytics-piple – paused

Work in progress:

  • new issue for course-discovery still in progress

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 39 hours

BD-20
~475/475
100% - 1/12

Out of the budget - 37 hours.

Dec 2, 2020 - Dec 15, 2020

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 21 hours

BD-20
~472/475
99% - 12/15

Nov 18, 2020 - Dec 1, 2020

Work done this sprint:

  • course-discovery Improvement work: several PRs were proposed

  • edx-analytics-piple travisCI was fixed.

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 25 hours

BD-20
~451/475
95% - 12/1

Nov 4, 2020 - Nov 17, 2020

The extended (in September) scope was corrected on the budget that add + 75h to projects. BD-20 volume was updated to 475h

Work done this sprint:

  • course-discovery Improvement work is near the finish line.

  • PR for edx-analytics-piple was prepared.

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 35 hours

BD-19
100% - 10/20

BD-20
~426/475
89% -11/17

 

Oct 21, 2020 - Nov 3, 2020

Work done this sprint:

  • course-discovery Improvement work is in progress, several new fixes were proposed that addresses found issues.

  • PR for cs_comments_service was deployed on Prod!

Work in progress:

Support for both projects during manual testing, live session testing, etc.

Note:

We didn’t have precise efforts splitting between the projects. All-time was divided equally between projects, but since ВD-19 is full all efforts are tracked to BD-20.

We’ve spent in sprint 54 hours

BD-19
~280/280
100% - 10/20

BD-20
~391/400
98% -11/3

Oct 7, 2020 - Oct 20, 2020

Work done this sprint:

  • course-discovery has been passed several test sessions. Several fixes were proposed.

  • PR for cs_comments_service was merged new fixes were proposed as well.

Work in progress:

Support for both projects during manual testing, live session testing, etc.

We’ve spent in sprint 42 hours

BD-19
~280/280
100% - 10/20

BD-20
~337/400
84% -10/20

Sep 23, 2020 - Oct 6, 2020

Work done this sprint:

  • PR for edx-analytics-pipeline was created, some minor improvements for coverage are required.

  • course-discovery has been passed several test sessions. Several fixes were proposed.

  • PR cs_comments_service is actively reviewed and tested by edx. Several fixes were proposed during the sprint.

Work in progress:

Support for both projects during manual testing, live session testing etc.

We’ve spent in sprint 92 hours

BD-19
~260/280
93% - 10/6

BD-20
~315/400
79% -10/6

Sep 9, 2020 - Sep 22, 2020

Work done this sprint:

  • Local PR for edx-analytics-pipeline was reviewed and approved.

  • Bugs were found in course-discovery, PR with fixes was created

  • Local deployment issues with cs_comments_service and edx-search were investigated and solved.

Work in progress:

Support for both projects during manual testing, live session testing etc.

  • BD - 20

    • ES upgrade - edx-analytics-pipeline – is ready to be forward to the upstream.

We’ve spent in sprint 39 hours

BD-19
~220/280
78% - 9/23

BD-20
~263/400
65% - 9/23

Aug 26, 2020 - Sep 8, 2020

Scopes of the projects was extended, details in the https://openedx.atlassian.net/wiki/spaces/COMM/pages/1413810945

Work done this sprint:

  • The retrospective was provided to both blended projects.

  • Local PR for edx-analytics-pipeline was opened.

  • PR for course-discovery in upstream was created

  • Work during the review process on the upstream PRs for cs_comments_service, course-discovery, and edx-search.

Work in progress:

  • BD - 19

    • haystack replacement - in the test/review phase

  • BD - 20

    • ES upgrade - edx-analytics-pipeline – in the review phase

We’ve spent in sprint 46 hours

BD-19 (additional efforts were not estimated 280h is temporal value)
~204/280
73% - 9/8

BD-20
~240/400
60% - 9/8

Aug 12, 2020 - Aug 25, 2020

Work done this sprint:

  • Active review work and improvements in local PR for course-discovery

  • Local PR for cs_comments_service was approved

  • Create PR for cs_comments_service in upstream.

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery - in the review phase

  • BD - 20

    • ES upgrade - cs_comments_service – in the review phase

We’ve spent in sprint 60 hours

BD-19
~189/200
95% - 8/26

BD-20
~209/240
87% - 8/26

Jul 29, 2020 - Aug 11, 2020

Work done this sprint:

  • Created PRs for edx-search in upstream.

  • Create local WIP PR for course-discovery

  • Create local PR for cs_comments_service

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery - in final dev phase

  • BD - 20

    • ES upgrade - cs_comments_service – in the final testing phase

We’ve spent in sprint 104 hours

BD-19
~169/200
85% - 8/11

BD-20
~169/240
70% - 8/11

Jul 15, 2020 - Jul 28, 2020

Work done this sprint:

  • Created local PR for edx-search.

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery - in active dev phase

  • BD - 20

    • ES upgrade - cs_comments_service

We’ve spent in sprint 90 hours

BD-19
~117/200
58% - 7/28

BD-20
~117/240
48% - 7/28

Jul 1, 2020 - Jul 14, 2020

Work done this sprint:

  • Created PRs in edx-notes-api and edx-analytics-data-api libraries.

Work in progress:

  • BD - 19

    • haystack replacement - course-discovery

  • BD - 20

    • ES upgrade - edx-search

We’ve spent in sprint 108 hours

BD-19
~72/200
36% - 7/14

BD-20
~72/240
30% - 7/14

Jun 16, 2020 - Jun 30, 2020

Work done this sprint:

  • Team kick-off and onboarding

  • Took first two libraries edx-notes-api and edx-analytics-data-api into upgrade work

  • Updated ES to latest

Work in progress:

  • BD - 19

    • haystack replacement - edx-notes-api

  • BD - 20

    • ES upgrade - edx-analytics-data-api

We’ve spent in sprint 36 hours