Lightweight catalog implementation via extensions
This proposal is an early response to the proposed DEPR of course-discovery.
Specifically from this comment, the new catalog app:
new catalog app: should this proposal take off, I believe we should put the core primitives like programs or catalog and write the most basic implementation possible. Then leave every bit of product development to plugins. I don't think that requirements for courses or anything so specific should be implemented in core. Mostly a models and calls to extension points should live there.
Roughly, when looking at the bare minimum requirements, out team looked at our current projects and the existing implementations of the same or very similar features in:
enterprise-catalog/blob/master/enterprise_catalog/apps/catalog/models.py
academic-innovation/platform-plugin-course-series/blob/basic-series-support/course_series/models.py
With this in mind, we are proposing a bare implementation that includes only:
FlexibleCatalogModel
course_runs: as manyToMany to CourseOverview
query_string: a string to match course_runs dinamically
slug: url friendly
name: for humans
get_course_runs: the method that either does fetches the course_runs
directly or that uses the query string for the dynamic
asignment. Implemented through filters so that
search: a method that does flexible search. This would be implemented via
filters so that the default implementation can be swapped. I would
also vouch for shipping with a mysql only implementation and one with
meilisearch
model specific openedx-filters: before meaningful changes are made
model specific openedx-events: after the catalog changes in a meaningful way
API REST:
CRUD on the FlexibleCatalog
the API must allow to set the course_runs or the query
the API to list the catalogs should use get_course_runs so that filters are respected
Search API that uses the search method so that filters are respected
Python API
get_catalogs: a public api method that includes filters to do access control
get_course_runs: for a catalog without exposing the model (for stability for those that can manage to only use this)
exposing the search api
CRUD on the Catalog
Naturally including the accompanying:
- docs
- serializers
- forms
- admin pages
- QA suite
- …
Comments
Thanks for the proposal @Felipe Montoya.
@Braden MacDonald, I mostly agree with your comments, I am just lost on a few details.
+1 on the
typefield.I understand removing both the fields would shift the responsibility to the plugin’s
get_course_runsimplementations. And it is upto them to whatever extra fields they want to add to the inherited model. However, I have hard time visualizing the default implementation. What would the default implementation ofget_course_runsreturn? Would this model be an Abstract model and is only usable via inheritance? Would it be possible to use this model and create simple catalogs, without developing plugins?No notes on performance. But your #5 got me thinking, if we would be better served to have something more generic like
get_catalog_items. Where the default implementation supports only course runs or courses (whichever is simpler) and plugins can implement complex things like Programs, Pathways or whatever and swap the default serialize with their own implementation. Adding to the “Program” point, I don’t think it is something that the core model should ever concern itself with. Because even the simplest Array of courses can have a variety of implementations. For eg., a program which gives off a specific cert for each course in it, a program which is a collection of courses all completed (sequentially vs non-sequentially) to get a single cert. So, it is best left to plugins to implement them the way they see fit.
Thanks Braden, you raise some very interesting points.
1. & 2. I like a lot more the #2 way of having a bare minimum model that defines `get_course_runs()` in a very basic way and it expects other classes to inherit and extend both the method and the fields required for the config fields. I think this would do away with the type field if implemented correctly. I’ll put some time away to play around with implementing code and get a feel for the dynamics.
3. I agree returning a QuerySet is the way to go there. Constructing the query does not seem like a big deal.
4. I was thinking of search as something that passes the search string into filters and thus allows you to build search using any backend (meilisearch et all) or even get the queryset from `get_course_runs` and filter from it. It would be as flexible as someone is willing to make filter steps for it.
5. I haven’t looked at programs yet.
Hi, @Felipe Montoya thanks for opening this conversation.
For one of our installations, having a catalogue feature as part of the core edx-platform and not as part of other (Enterprise) implementations is essential and useful. We are currently building an internal solution for Catalogues as we cannot wait until the platform supports an out-of-the-box catalogue and we are not willing to use Enterprise Catalogues. We are developing a lightweight implementation based on a model to store the main entity (Catalogues) and a many-to-many relationship with course overview; we also have a model called available_courses to handle the case where you want to be able to select a specific set of courses to be in a catalogue instead of all the courses in the course overview table and to handle the lifecycle of courses when a particular course or courses are deleted or deprecated so that they can be removed from all catalogues.
I like the idea of having different types of catalogues and the fact that it will be up to the operator to use the default one which will be I think course_runs or create an extension like query_string or course_number (For Programs?). In practice, I see that this Catalogue app will only have one model, which will be FlexibleCatalogModel with a type field to link the catalogue and the catalogue methodology used. However, I'm still thinking about the actual implementation so this is what I think it will look like:
class CatalogueCourseRuns(models.Model) or (FlexibleCatalogModel) in case that is an abstract model:
catalogue_id = models.ForeignKey(FlexibleCatalogModel, ...)
courses = models.ManyToManyField(CourseOverview or AvailableCourses)
class CatalogueQueryString(models.Model) or (FlexibleCatalogModel) in case that is an abstract model:
catalogue_id = models.ForeignKey(FlexibleCatalogModel, ...)
query_string = models.CharField(...)If the above implementation is correct, I believe get_catalog_items (from FlexibleCatalogModel) will receive the ID or UUID of the catalogue and check the type and depending on the type will retrieve the items from the appropriate implementation, if this is still correct, I believe get_catalog_items will need to call a standard method on the catalogue extension models (course_runs or query_string) to retrieve the items, for this, it will need to know the Python module path of the catalogue extension model to call this standard method, right?
If all of the above statements are correct do you all think:
1. Would we need a model to store the catalogue type and the Python module path? So that operators can define X number of catalogue types and their Python path.
2. Will FlexibleCatalogModel be an abstract model? Since an installation could have different types of catalogues, how would the platform know which one to call?
3. Is it worth having available_courses as part of the course_runs catalogue type? Or can it be handled by the Filters implementation?
4. Will the implementation of Filters be part of each of the ‘standard’ methods for retrieving courses in the extension catalogue models, or will it be part of get_catalog_items?
After I started using InheritanceManager points 1 and 2 were solved. No need to add a type field, django-model-utils solves it when calling `select_subclasses`.
About 3) having available_courses looks like something that not everybody will need. With the goal of keeping the base implementation lightweight and allowing for heavy customization at the extending plugins I would favor that we keep this out of the base models. I tried to implement this in the custom_catalog plugin from the concept and it was easy to keep it separated. As you mentioned, we could make this a filter, but it would again mean that we will maintain support for an interface that is more complex than it needs to be for most.
On 4) I do expect that filters be part of the standard models. Specially in the get_course_runs and search methods so that basic extensions can be acomplished by creating filter pipelines only. For things that go beyond those two this proposal would require that you create your own subclass model.
I started a project to test the concepts outlined here and here it is: https://github.com/eduNEXT/flex-catalog-concept
To go directly to the models: https://github.com/eduNEXT/flex-catalog-concept/blob/main/flex_catalog/flex_catalog/models.py
I’m still missing some method to single out which class should be used to load the instance of the catalog as right now the base class could load any of the records and return the basic catalog.
I’ll probably need to implement the type field and use that to introspect the proper model.
I leave some screenshots here:
The base models:
One catalog of each type in the admin model of the base class:
the admin models for the two implementations (fixed and dynamic):
Nice, @Felipe Montoya.
I think:
The
FlexibleCatalogModelshould also have a “type” field, so that you can have several different types of catalog (e.g. dynamic vs. static) each managed by different plugins. This also lets us have a clean way to provide a default implementation (e.g. when “type = default”, always just usecourse_runs, ignorequery_string) that can be used alongside custom catalog types.It may be better to remove
course_runsandquery_stringentirely and just let inherited/one-to-one models define those. For example, a common requirement would be to have a static list of course sets (“catalog courses” like “Math 100”, not course runs like “Math 100 2024-Winter”), and if we only provide a many-to-many link to CourseOverview, we can’t do that even with this supposedly “flexible” model. (Note that get_course_runs() would still return runs in this example, but the list is defined in terms of catalog courses / course sets, not runs. e.g. it might return only the most recent run or some other criteria.) I think we also want to avoid anyone using thecourse_runsandquery_stringfields other than for implementing a new catalog type, and removing them from the base model is a great way to enforce that.I think in 80% of use cases, we can improve performance by requiring that
get_course_runsreturn a QuerySet of CourseOverview, which can then be used in further queries. The downside is that if a search engine is used, a query like “SELECT c FROM course_overview WHERE c.id IN (1, 5, 7, …)” must be constructed, but I think that’s fine.What does the
searchmethod do? Keyword search? Would it also support filtering like filter by org, filter by tag, filter by year, or more?Have you thought about how this could look like with programs?