Courseware StudentModule (CSM) Call Catalog

Below I'm cataloging each and every read/write access to the courseware StudentModule (CSM) Django model. It's essential to know of each access due to our pending work to encapsulate all CSM access behind an interface. The interface backend will support both the present Django ORM and a Cassandra backend to better handle student traffic at scale. Dave Ormsbee has authored this page to capture existing CSM query patterns - the catalog below looks at all code in edx-platform and catalogs access in each file therein.

lms/djangoapps/courseware/model_data.py

FieldDataCache is defined in this file - it's a cache of Django ORM objects from CSM. The object are read into the cache and then modified / written from the cache.

Reads

Call Chain

Query

R/W?Maps To New Query

FieldDataCache.__init__() / FieldDataCache.add_descriptor_descendants()
FieldDataCache.add_descriptors_to_cache()
FieldDataCache._retrieve_fields()

return self._chunked_query(
StudentModule,
'module_state_key__in',
self._all_usage_ids(descriptors),
course_id=self.course_id,
student=self.user.pk,
)
Ultimately, an objects.filter() call.
R001 / 002

Writes

Call Chain

Query

R/W?Maps To New Query
FieldDataCache.find_or_create()
DjangoKeyValueStore.set_many()

field_object = StudentModule(
course_id=self.course_id,
student_id=key.user_id,
module_state_key=key.block_scope_id,
state=json.dumps({}),
module_type=key.block_scope_id.block_type,
)

W001 / 002
DjangoKeyValueStore.set_many()StudentModule.save()W001 / 002

DjangoKeyValueStore.delete()

StudentModule.save()

W001 / 002

lms/djangoapps/class_dashboard/dashboard_data.py

Reads

Call ChainQueryR/W?Maps To New Query

get_problem_grade_distribution()

db_query = models.StudentModule.objects.filter(
course_id__exact=course_id,
grade__isnull=False,
module_type__exact="problem",
).values(
'module_state_key', 'grade', 'max_grade'
).annotate(
count_grade=Count('grade')
)

R010

get_sequential_open_distrib()

db_query = models.StudentModule.objects.filter(
course_id__exact=course_id,
module_type__exact="sequential",
).values(
'module_state_key'
).annotate(
count_sequential=Count('module_state_key')
)

R015

get_problem_set_grade_distrib()

db_query = models.StudentModule.objects.filter(
course_id__exact=course_id,
grade__isnull=False,
module_type__exact="problem",
module_state_key__in=problem_set,
).values(
'module_state_key',
'grade',
'max_grade',
).annotate(
count_grade=Count('grade')
).order_by(
'module_state_key', 'grade'
)

R012

get_students_opened_subsection()

students = models.StudentModule.objects.select_related('student').filter(
module_state_key__exact=module_state_key,
module_type__exact='sequential',
).values(
'student__username', 'student__profile__name'
).order_by(
'student__profile__name'
)

R013

get_students_problem_grades()

students = models.StudentModule.objects.select_related('student').filter(
module_state_key=module_state_key,
module_type__exact='problem',
grade__isnull=False,
).values(
'student__username', 'student__profile__name', 'grade', 'max_grade'
).order_by(
'student__profile__name'
)

R011

lms/djangoapps/courseware/grades.py

Reads

Call ChainQueryR/W?Maps to New Query

answer_distributions(course_key)

for module in StudentModule.all_submitted_problems_read_only(course_key):

...which is:

queryset = cls.objects.filter(
course_id=course_id,
module_type='problem',
grade__isnull=False
)
if "read_replica" in settings.DATABASES:
return queryset.using("read_replica")
else:
return queryset

R006
_grade(student, request, course, keep_raw_scores)

should_grade_section = StudentModule.objects.filter(
student=student,
module_state_key__in=[
descriptor.location for descriptor in section['xmoduledescriptors']
]
).exists()

R

---

get_score(course_id, user, problem_descriptor,

module_creator, scores_cache=None)

student_module = StudentModule.objects.get(
student=user,
course_id=course_id,
module_state_key=problem_descriptor.location
)

R008

common/djangoapps/xmodule_modifiers.py

This access behaves poorly - it accesses the CSM table via direct SQL!

Call ChainQueryR/W?Maps to New Query
grade_histogram()

SELECT courseware_studentmodule.grade,
COUNT(courseware_studentmodule.student_id)
FROM courseware_studentmodule
WHERE courseware_studentmodule.module_id=%s
GROUP BY courseware_studentmodule.grade

R

---

 

lms/djangoapps/courseware/entrance_exams.py

Determines a user's entrance exam score via direct CSM access.

Call ChainQueryR/W?Maps to New Query

_calculate_entrance_exam_score()

student_modules = StudentModule.objects.filter(
student=user,
course_id=course_descriptor.id,
module_state_key__in=exam_module_ids,
)

R002 / 009

lms/djangoapps/courseware/views.py

Call ChainQueryR/W?Maps to New Query

submission_history(request, course_id,

student_username, location)

student_module = StudentModule.objects.get(
course_id=course_key,
module_state_key=usage_key,
student_id=student.id
)
history_entries = StudentModuleHistory.objects.filter(
student_module=student_module
).order_by('-id')

# If no history records exist, let's force a save to get history started.
if not history_entries:
student_module.save()
history_entries = StudentModuleHistory.objects.filter(
student_module=student_module
).order_by('-id')

R/W

003

BUT CSMH will likely become just another

column family. Possibly one like:

 

  Row key
<course|student|module> =>
 Column Name       Column Value
 <timestamp> : <JSON with version|created|state|grade|max_grade>

lms/djangoapps/courseware/management/commands/clean_history.py

Django management command that clears CSM history for a particular student module.

Call ChainQueryR/W?Maps to New Query
get_last_student_module_id()SELECT max(student_module_id) FROM courseware_studentmodulehistoryRIf TTL used, this isn't necessary.
get_history_for_student_modules()

SELECT id, created FROM courseware_studentmodulehistory
WHERE student_module_id = %s
ORDER BY created, id

R(course, student, module, timestamp)
delete_history()

DELETE FROM courseware_studentmodulehistory
WHERE id IN ({ids})

WWe'll use TTL instead of explicit delete?

lms/djangoapps/courseware/management/commands/regrade_partial.py

One-off Django management command.

Call ChainQueryMaps to New Query
fix_studentmodules()

modules = StudentModule.objects.filter(modified__gt='2013-03-07 20:18:00',
created__lt='2013-03-08 15:45:00',
state__contains='"npoints": 0.')

None - will be deleted.

lms/djangoapps/courseware/management/commands/remove_input_state.py

One-off Django management command.

Call ChainQueryMaps to New Query
fix_studentmodules_in_list()

module = StudentModule.objects.get(id=student_module_id)

hist_modules = StudentModuleHistory.objects.filter(student_module_id=student_module_id)

None - will be deleted.

lms/djangoapps/courseware/management/commands/tests/test_clean_history.py

Call ChainQueryMaps to New Query
From many tests: write_history()

INSERT INTO courseware_studentmodulehistory
(id, created, student_module_id)
VALUES (%s, %s, %s)

---
From many tests: read_history()

SELECT id, created, student_module_id FROM courseware_studentmodulehistory

---

lms/djangoapps/courseware/tests/test_model_data.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/courseware/tests/test_module_render.py

Two tests - test_xmodule_runtime_publish() & test_xmodule_runtime_publish_delete() - use direct StudentModule.objects() access.

lms/djangoapps/courseware/tests/test_submitting_problems.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/enrollment.py

Call ChainQueryR/W?Maps to New Query
reset_student_attempts()

module_to_reset = StudentModule.objects.get(
student_id=student.id,
course_id=course_id,
module_state_key=module_state_key
)

 Then .delete() or .save()

RW001

lms/djangoapps/instructor/management/commands/openended_stats.py

Call ChainQueryR/W?Maps to New Query
calculate_task_statistics()

student_modules = StudentModule.objects.filter(

module_state_key=location, student__in=students

).order_by('student')

R

007

(course, module, students)

lms/djangoapps/instructor/tests/test_api.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/tests/test_enrollment.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/tests/test_tools.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/views/legacy.py

Dumps CSV of problem reponses.

Call ChainQueryR/W?Maps to New Query
instructor_dashboard()

smdat = StudentModule.objects.filter(
course_id=course_key,
module_state_key=module_state_key
)
smdat = smdat.order_by('student')

R

None - who needs this?

Provide another inteface for it.


lms/djangoapps/instructor_task/tasks_helper.py

Call ChainQueryR/W?Maps to New Query
perform_module_state_update()

modules_to_update = StudentModule.objects.filter(

course_id=course_id, module_state_key__in=usage_keys

)

Also supports a further student and generic filter_fcn() filtering.

RW

001 / 004

BUT - support of generic filter_fcn()

probably not possible.

lms/djangoapps/instructor_task/tests/test_base.py

A test performs direct StudentModule.object() access.

lms/djangoapps/instructor_task/tests/test_tasks.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/psychometrics/models.py

Creates a model that joins with every single row in CSM. This functionality will likely not be possible with the CSM behind an interface!

Other associated files:

  • lms/djangoapps/psychometrics/psychoanalyze.py
  • lms/djangoapps/psychometrics/management/commands/init_psychometrics.py

src/edx-sga/edx_sga/sga.py

Staff-graded assignments XBlock.

Call ChainQueryR/W?Maps to New Query
staff_grading_data()

module, _ = StudentModule.objects.get_or_create(
course_id=self.course_id,
module_state_key=self.location,
student=user,
defaults={
'state': '{}',
'module_type': self.category,
})

RW---
staff_upload_annotated()

module = StudentModule.objects.get(pk=request.params['module_id'])

And finally, .save()

RW---
staff_download_annotated()

module = StudentModule.objects.get(pk=request.params['module_id'])

RW---
enter_grade()

module = StudentModule.objects.get(pk=request.params['module_id'])

And finally, .save()

RW---
remove_grade()

module = StudentModule.objects.get(pk=request.params['module_id'])

And finally, .save()

RW---

src/edx-sga/edx_sga/tests.py

Tests of staff-graded assignments XBlock.

Several tests perform direct StudentModule.object() access.

src/edx-sga/edx_sga/management/commands/sga_migrate_submissions.py

Django management command that migrates existing SGA submissions for a course from old SGA implementation to newer version that uses the 'submissions' application.

Call ChainQueryR/W?Maps to New Query
 

student_modules = StudentModule.objects.filter(
course_id=course.id).filter(
module_state_key__contains='edx_sga')

R---