[Proposal] Content Search Powered by Meilisearch

Content Search Powered by Meilisearch

Overview

The Open edX platform is enhancing its search functionality by implementing Meilisearch, a fast and memory-efficient search engine. This project aims to improve search performance, reduce resource usage, and provide a more developer-friendly experience.

Problem

The current search functionality in the Open edX platform, primarily powered by Elasticsearch, is resource-intensive, lacks proper multi-tenancy support, and has licensing issues that complicate its usage for some operators.

Key Use Cases

• As a course author, I need to be able to efficiently search for course materials to streamline content management and updates.

• As an instructor, I need to be able to quickly find relevant forum posts and course content to provide timely assistance to learners.

• As a learner, I need to be able to search course materials and forum posts effectively to enhance my learning experience and find answers quickly.

Supporting Market Data

• Feedback from developers indicates that the current Elasticsearch implementation is complex and difficult to maintain.

• Analysis of resource usage shows that Elasticsearch often consumes more than a gigabyte of memory even for small search operations, which is inefficient.

Is Meilisearch a viable upgrade alternative to OpenSearch?

Auto-suggest course content on search (Meilisearch-compatible)

Proposed Solution

We propose implementing Meilisearch as a replacement for Elasticsearch in the Open edX platform. Meilisearch offers significant benefits such as lower memory usage, faster performance, strong support for multi-tenancy, and a simpler API. The new solution will involve creating a generalized search API that can support various search engines, including Elasticsearch, Meilisearch, and Algolia.

UX/UI Designs

We have implemented course search functionallity using openedx-search-api there are no significat changes in UI/UX.

Open

Pull Requests

https://github.com/openedx/frontend-app-learning/pull/1433

https://github.com/openedx/edx-platform/pull/35177

https://github.com/qasimgulzar/openedx-search-api/pull/8

Other Approaches Considered

1. Continuing with Elasticsearch:

   • Why it won't work: Licensing changes, high resource usage, and poor multi-tenancy support make it unsuitable for long-term use.

2. Using Algolia exclusively:

   • Why it won't work: While Algolia is powerful, it may not be as cost-effective as Meilisearch for all use cases.

Competitive Research

• Canvas: Utilizes Elasticsearch but faces similar issues with resource usage and complexity.

• Moodle: Employs a mix of search technologies with varying degrees of efficiency and usability.

• Coursera: Uses proprietary search solutions tailored to their platform, which are not easily adaptable to Open edX.

Plan for Long-Term Ownership/Maintainership

• The new search API and Meilisearch integration will be maintained by the Edly development team.

• Documentation and training will be provided to ensure that community developers can contribute to and maintain the system.

Open Questions for Rollout/Releases

1. How will existing Elasticsearch indexes be migrated to Meilisearch?

2. What is the timeline for deprecating Elasticsearch support?

3. How will we handle potential disruptions during the transition period?

Considerations for Basic Implementation Requirements

• Unit Tests Written: Comprehensive unit tests for the new search API and Meilisearch integration.

• Docs Updated: Documentation will be updated to reflect the new search functionality and API.

• I18n/L10n Requirements: Support for internationalization and localization will be built into the new search API.

• Deprecation Plan: A clear plan for phasing out the old Elasticsearch-based search functionality.

• Acceptance Criteria Tests: Rigorous acceptance tests to validate the new search functionality against user requirements.