Courseware StudentModule (CSM) Call Catalog

Below I'm cataloging each and every read/write access to the courseware StudentModule (CSM) Django model. It's essential to know of each access due to our pending work to encapsulate all CSM access behind an interface. The interface backend will support both the present Django ORM and a Cassandra backend to better handle student traffic at scale. Dave Ormsbee has authored this page to capture existing CSM query patterns - the catalog below looks at all code in edx-platform and catalogs access in each file therein.

lms/djangoapps/courseware/model_data.py

FieldDataCache is defined in this file - it's a cache of Django ORM objects from CSM. The object are read into the cache and then modified / written from the cache.

Reads

Call Chain	Query	R/W?	Maps To New Query
`FieldDataCache.__init__() / FieldDataCache.add_descriptor_descendants()` `FieldDataCache.add_descriptors_to_cache()` `FieldDataCache._retrieve_fields()`	return self._chunked_query( StudentModule, 'module_state_key__in', self._all_usage_ids(descriptors), course_id=self.course_id, student=self.user.pk, ) Ultimately, an objects.filter() call.	R	001 / 002

Writes

Call Chain	Query	R/W?	Maps To New Query
FieldDataCache.find_or_create() DjangoKeyValueStore.set_many()	`field_object = StudentModule(` `course_id=self.course_id,` `student_id=key.user_id,` `module_state_key=key.block_scope_id,` `state=json.dumps({}),` `module_type=key.block_scope_id.block_type,` `)`	W	001 / 002
`DjangoKeyValueStore.set_many()`	`StudentModule.save()`	W	001 / 002
`DjangoKeyValueStore.delete()`	`StudentModule.save()`	W	001 / 002

lms/djangoapps/class_dashboard/dashboard_data.py

Reads

Call Chain	Query	R/W?	Maps To New Query
`get_problem_grade_distribution()`	`db_query = models.StudentModule.objects.filter(` `course_id__exact=course_id,` `grade__isnull=False,` `module_type__exact="problem",` `).values(` `'module_state_key', 'grade', 'max_grade'` `).annotate(` `count_grade=Count('grade')` `)`	R	010
`get_sequential_open_distrib()`	`db_query = models.StudentModule.objects.filter(` `course_id__exact=course_id,` `module_type__exact="sequential",` `).values(` `'module_state_key'` `).annotate(` `count_sequential=Count('module_state_key')` `)`	R	015
`get_problem_set_grade_distrib()`	`db_query = models.StudentModule.objects.filter(` `course_id__exact=course_id,` `grade__isnull=False,` `module_type__exact="problem",` `module_state_key__in=problem_set,` `).values(` `'module_state_key',` `'grade',` `'max_grade',` `).annotate(` `count_grade=Count('grade')` `).order_by(` `'module_state_key', 'grade'` `)`	R	012
`get_students_opened_subsection()`	`students = models.StudentModule.objects.select_related('student').filter(` `module_state_key__exact=module_state_key,` `module_type__exact='sequential',` `).values(` `'student__username', 'student__profile__name'` `).order_by(` `'student__profile__name'` `)`	R	013
`get_students_problem_grades()`	`students = models.StudentModule.objects.select_related('student').filter(` `module_state_key=module_state_key,` `module_type__exact='problem',` `grade__isnull=False,` `).values(` `'student__username', 'student__profile__name', 'grade', 'max_grade'` `).order_by(` `'student__profile__name'` `)`	R	011

lms/djangoapps/courseware/grades.py

Reads

Call Chain Query R/W? Maps to New Query

answer_distributions(course_key)

for module in StudentModule.all_submitted_problems_read_only(course_key):

...which is:

queryset = cls.objects.filter(
course_id=course_id,
module_type='problem',
grade__isnull=False
)
if "read_replica" in settings.DATABASES:
return queryset.using("read_replica")
else:
return queryset

R

006

_grade(student, request, course, keep_raw_scores)

should_grade_section = StudentModule.objects.filter(
student=student,
module_state_key__in=[
descriptor.location for descriptor in section['xmoduledescriptors']
]
).exists()

R

---

get_score(course_id, user, problem_descriptor,

module_creator, scores_cache=None)

student_module = StudentModule.objects.get(
student=user,
course_id=course_id,
module_state_key=problem_descriptor.location
)

R

008

common/djangoapps/xmodule_modifiers.py

This access behaves poorly - it accesses the CSM table via direct SQL!

Call Chain Query R/W? Maps to New Query

grade_histogram()

SELECT courseware_studentmodule.grade,
COUNT(courseware_studentmodule.student_id)
FROM courseware_studentmodule
WHERE courseware_studentmodule.module_id=%s
GROUP BY courseware_studentmodule.grade

R

---

lms/djangoapps/courseware/entrance_exams.py

Determines a user's entrance exam score via direct CSM access.

Call Chain	Query	R/W?	Maps to New Query
`_calculate_entrance_exam_score()`	`student_modules = StudentModule.objects.filter(` `student=user,` `course_id=course_descriptor.id,` `module_state_key__in=exam_module_ids,` `)`	R	002

lms/djangoapps/courseware/views.py

Call Chain Query R/W? Maps to New Query

submission_history(request, course_id,

student_username, location)

student_module = StudentModule.objects.get(
course_id=course_key,
module_state_key=usage_key,
student_id=student.id
)
history_entries = StudentModuleHistory.objects.filter(
student_module=student_module
).order_by('-id')

# If no history records exist, let's force a save to get history started.
if not history_entries:
student_module.save()
history_entries = StudentModuleHistory.objects.filter(
student_module=student_module
).order_by('-id')

R/W

003

BUT CSMH will likely become just another

column family. Possibly one like:

  Row key

<course|student|module> =>

 Column Name       Column Value

 <timestamp> : <JSON with version|created|state|grade|max_grade>

lms/djangoapps/courseware/management/commands/clean_history.py

Django management command that clears CSM history for a particular student module.

Call Chain	Query	R/W?	Maps to New Query
`get_last_student_module_id()`	`SELECT max(student_module_id) FROM courseware_studentmodulehistory`	R	If TTL used, this isn't necessary.
`get_history_for_student_modules()`	`SELECT id, created FROM courseware_studentmodulehistory` `WHERE student_module_id = %s` `ORDER BY created, id`	R	(course, student, module, timestamp)
`delete_history()`	`DELETE FROM courseware_studentmodulehistory` `WHERE id IN ({ids})`	W	We'll use TTL instead of explicit delete?

lms/djangoapps/courseware/management/commands/regrade_partial.py

One-off Django management command.

Call Chain	Query	Maps to New Query
`fix_studentmodules()`	`modules = StudentModule.objects.filter(modified__gt='2013-03-07 20:18:00',` `created__lt='2013-03-08 15:45:00',` `state__contains='"npoints": 0.')`	None - will be deleted.

lms/djangoapps/courseware/management/commands/remove_input_state.py

One-off Django management command.

Call Chain Query Maps to New Query

fix_studentmodules_in_list()

module = StudentModule.objects.get(id=student_module_id)

hist_modules = StudentModuleHistory.objects.filter(student_module_id=student_module_id)

None - will be deleted.

lms/djangoapps/courseware/management/commands/tests/test_clean_history.py

Call Chain	Query	Maps to New Query
`From many tests: write_history()`	`INSERT INTO courseware_studentmodulehistory` `(id, created, student_module_id)` `VALUES (%s, %s, %s)`	---
`From many tests: read_history()`	`SELECT id, created, student_module_id FROM courseware_studentmodulehistory`	---

lms/djangoapps/courseware/tests/test_model_data.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/courseware/tests/test_module_render.py

Two tests - test_xmodule_runtime_publish() & test_xmodule_runtime_publish_delete() - use direct StudentModule.objects() access.

lms/djangoapps/courseware/tests/test_submitting_problems.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/enrollment.py

Call Chain Query R/W? Maps to New Query

reset_student_attempts()

module_to_reset = StudentModule.objects.get(
student_id=student.id,
course_id=course_id,
module_state_key=module_state_key
)

Then .delete() or .save()

RW

001

lms/djangoapps/instructor/management/commands/openended_stats.py

Call Chain Query R/W? Maps to New Query

calculate_task_statistics()

student_modules = StudentModule.objects.filter(

module_state_key=location, student__in=students

).order_by('student')

R

007

(course, module, students)

lms/djangoapps/instructor/tests/test_api.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/tests/test_enrollment.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/tests/test_tools.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/views/legacy.py

Dumps CSV of problem reponses.

Call Chain Query R/W? Maps to New Query

instructor_dashboard()

smdat = StudentModule.objects.filter(
course_id=course_key,
module_state_key=module_state_key
)
smdat = smdat.order_by('student')

R

None - who needs this?

Provide another inteface for it.

lms/djangoapps/instructor_task/tasks_helper.py

Call Chain Query R/W? Maps to New Query

perform_module_state_update()

modules_to_update = StudentModule.objects.filter(

course_id=course_id, module_state_key__in=usage_keys

)

Also supports a further student and generic filter_fcn() filtering.

RW

001 / 004

BUT - support of generic filter_fcn()

probably not possible.

lms/djangoapps/instructor_task/tests/test_base.py

A test performs direct StudentModule.object() access.

lms/djangoapps/instructor_task/tests/test_tasks.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/psychometrics/models.py

Creates a model that joins with every single row in CSM. This functionality will likely not be possible with the CSM behind an interface!

Other associated files:

lms/djangoapps/psychometrics/psychoanalyze.py
lms/djangoapps/psychometrics/management/commands/init_psychometrics.py

src/edx-sga/edx_sga/sga.py

Staff-graded assignments XBlock.

Call Chain	Query	R/W?	Maps to New Query
`staff_grading_data()`	`module, _ = StudentModule.objects.get_or_create(` `course_id=self.course_id,` `module_state_key=self.location,` `student=user,` `defaults={` `'state': '{}',` `'module_type': self.category,` `})`	RW	---
`staff_upload_annotated()`	`module = StudentModule.objects.get(pk=request.params['module_id'])` `And finally, .save()`	RW	---
`staff_download_annotated()`	`module = StudentModule.objects.get(pk=request.params['module_id'])`	RW	---
`enter_grade()`	`module = StudentModule.objects.get(pk=request.params['module_id'])` `And finally, .save()`	RW	---
`remove_grade()`	`module = StudentModule.objects.get(pk=request.params['module_id'])` `And finally, .save()`	RW	---

src/edx-sga/edx_sga/tests.py

Tests of staff-graded assignments XBlock.

Several tests perform direct StudentModule.object() access.

src/edx-sga/edx_sga/management/commands/sga_migrate_submissions.py

Django management command that migrates existing SGA submissions for a course from old SGA implementation to newer version that uses the 'submissions' application.

Call Chain	Query	R/W?	Maps to New Query
	`student_modules = StudentModule.objects.filter(` `course_id=course.id).filter(` `module_state_key__contains='edx_sga')`	R	---