Versions Compared

Old Version 6

changes.mady.by.user Julia Eskew

Saved on

compared with

New Version Current

changes.mady.by.user Kyle McCormick

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Below I'm cataloging each and every read/write access to the courseware StudentModule (CSM) Django model. It's essential to know of each access due to our pending work to encapsulate all CSM access behind an interface. The interface backend will support both the present Django ORM and a Cassandra backend to better handle student traffic at scale. Dave Ormsbee has authored this page to capture existing CSM query patterns - the catalog below looks at all code in edx-platform and catalogs access in each file therein.

lms/djangoapps/courseware/model_data.py

FieldDataCache is defined in this file - it's a cache of Django ORM objects from CSM. The object are read into the cache and then modified / written from the cache.

Reads

Call Chain

Query

R/W?Maps To New Query

FieldDataCache.__init__() / FieldDataCache.add_descriptor_descendants()
FieldDataCache.add_descriptors_to_cache()
FieldDataCache._retrieve_fields()

return self._chunked_query(
 StudentModule,
 'module_state_key__in',
 self._all_usage_ids(descriptors),
 course_id=self.course_id,
 student=self.user.pk,
)
Ultimately, an objects.filter() call.
R 001 / 002

Writes

DjangoKeyValueStore
Call Chain

Query

R/W?Maps To New Query
FieldDataCache.find_or_create()
DjangoKeyValueStore.set_many()

field_object = StudentModule(
course_id=self.course_id,
student_id=key.user_id,
module_state_key=key.block_scope_id,
state=json.dumps({}),
module_type=key.block_scope_id.block_type,
)

 W001 / 002
DjangoKeyValueStore.set_many()StudentModule.save()W 001 / 002

DjangoKeyValueStore.delete()

StudentModule.save()

W 001 / 002

lms/djangoapps/class_dashboard/dashboard_data.py

Reads

Call ChainQueryR/W?Maps To New Query

get_problem_grade_distribution()

db_query = models.StudentModule.objects.filter(
course_id__exact=course_id,
grade__isnull=False,
module_type__exact="problem",
).values(
'module_state_key', 'grade', 'max_grade'
).annotate(
count_grade=Count('grade')
)

R 010

get_sequential_open_distrib()

db_query = models.StudentModule.objects.filter(
course_id__exact=course_id,
module_type__exact="sequential",
).values(
'module_state_key'
).annotate(
count_sequential=Count('module_state_key')
)

R 015

get_problem_set_grade_distrib()

db_query = models.StudentModule.objects.filter(
course_id__exact=course_id,
grade__isnull=False,
module_type__exact="problem",
module_state_key__in=problem_set,
).values(
'module_state_key',
'grade',
'max_grade',
).annotate(
count_grade=Count('grade')
).order_by(
'module_state_key', 'grade'
)

 R012

get_students_opened_subsection()

students = models.StudentModule.objects.select_related('student').filter(
module_state_key__exact=module_state_key,
module_type__exact='sequential',
).values(
'student__username', 'student__profile__name'
).order_by(
'student__profile__name'
)

 R013

get_students_problem_grades()

students = models.StudentModule.objects.select_related('student').filter(
module_state_key=module_state_key,
module_type__exact='problem',
grade__isnull=False,
).values(
'student__username', 'student__profile__name', 'grade', 'max_grade'
).order_by(
'student__profile__name'
)

R 011

lms/djangoapps/courseware/grades.py

Reads

Call ChainQueryR/W?Maps to New Query

answer_distributions(course_key)

for module in StudentModule.all_submitted_problems_read_only(course_key):

...which is:

queryset = cls.objects.filter(
course_id=course_id,
module_type='problem',
grade__isnull=False
)
if "read_replica" in settings.DATABASES:
return queryset.using("read_replica")
else:
return queryset

R 006
_grade(student, request, course, keep_raw_scores)

should_grade_section = StudentModule.objects.filter(
student=student,
module_state_key__in=[
descriptor.location for descriptor in section['xmoduledescriptors']
]
).exists()

R 

---

get_score(course_id, user, problem_descriptor,

module_creator, scores_cache=None)

student_module = StudentModule.objects.get(
student=user,
course_id=course_id,
module_state_key=problem_descriptor.location
)

 R008

common/djangoapps/xmodule_modifiers.py

This access behaves poorly - it accesses the CSM table via direct SQL!

Call ChainQueryR/W?Maps to New Query
grade_histogram()

SELECT courseware_studentmodule.grade,
COUNT(courseware_studentmodule.student_id)
FROM courseware_studentmodule
WHERE courseware_studentmodule.module_id=%s
GROUP BY courseware_studentmodule.grade

R

---

 

lms/djangoapps/courseware/entrance_exams.py

Determines a user's entrance exam score via direct CSM access.

Call ChainQueryR/W?Maps to New Query

_calculate_entrance_exam_score()

student_modules = StudentModule.objects.filter(
student=user,
course_id=course_descriptor.id,
module_state_key__in=exam_module_ids,
)

 		R002 / 009

lms/djangoapps/courseware/views.py

Call ChainQueryR/W?Maps to New Query

submission_history(request, course_id,

student_username, location)

student_module = StudentModule.objects.get(
course_id=course_key,
module_state_key=usage_key,
student_id=student.id
)
history_entries = StudentModuleHistory.objects.filter(
student_module=student_module
).order_by('-id')

# If no history records exist, let's force a save to get history started.
if not history_entries:
student_module.save()
history_entries = StudentModuleHistory.objects.filter(
student_module=student_module
).order_by('-id')

 

lms/djangoapps/courseware/management/commands/clean_history.py

Django management command that clears CSM history for a particular student module.

Call ChainQueryMaps to New Query
get_last_student_module_id()SELECT max(student_module_id) FROM courseware_studentmodulehistory 
get_history_for_student_modules()

SELECT id, created FROM courseware_studentmodulehistory
WHERE student_module_id = %s
ORDER BY created, id

 
delete_history()

DELETE FROM courseware_studentmodulehistory
WHERE id IN ({ids})

 

lms/djangoapps/courseware/management/commands/regrade_partial.py

One-off Django management command.

Call ChainQueryMaps to New Query
fix_studentmodules()

modules = StudentModule.objects.filter(modified__gt='2013-03-07 20:18:00',
created__lt='2013-03-08 15:45:00',
state__contains='"npoints": 0.')

None - will be deleted.

lms/djangoapps/courseware/management/commands/remove_input_state.py

One-off Django management command.

None - will be deleted.
Call ChainQueryMaps to New Query
fix_studentmodules_in_list()

module = StudentModule.objects.get(id=student_module_id)

hist_modules = StudentModuleHistory.objects.filter(student_module_id=student_module_id)

R/W

003

BUT CSMH will likely become just another

column family. Possibly one like:

 

  Row key
<course|student|module> =>
 Column Name       Column Value
 <timestamp> : <JSON with version|created|state|grade|max_grade>

lms/djangoapps/courseware/management/commands

...

/

...

clean_history.py

Call ChainQueryMaps to New Query
From many tests: write_history()

INSERT INTO courseware_studentmodulehistory
(id, created, student_module_id)
VALUES (%s, %s, %s)

 
From many tests: read_history()

SELECT id, created, student_module_id FROM courseware_studentmodulehistory

 

lms/djangoapps/courseware/tests/test_model_data.py

Several tests perform direct StudentModule.object() access.Django management command that clears CSM history for a particular student module.

Call ChainQueryR/W?Maps to New Query
get_last_student_module_id()SELECT max(student_module_id) FROM courseware_studentmodulehistoryRIf TTL used, this isn't necessary.
get_history_for_student_modules()

SELECT id, created FROM courseware_studentmodulehistory
WHERE student_module_id = %s
ORDER BY created, id

R(course, student, module, timestamp)
delete_history()

DELETE FROM courseware_studentmodulehistory
WHERE id IN ({ids})

WWe'll use TTL instead of explicit delete?

lms/djangoapps/courseware/

...

management/commands/

...

regrade_

...

partial.py

Two tests - test_xmodule_runtime_publish() & test_xmodule_runtime_publish_delete() - use direct StudentModule.objects() access.One-off Django management command.

Call ChainQueryMaps to New Query
fix_studentmodules()

modules = StudentModule.objects.filter(modified__gt='2013-03-07 20:18:00',
created__lt='2013-03-08 15:45:00',
state__contains='"npoints": 0.')

None - will be deleted.

lms/djangoapps/courseware/management/

...

commands/

...

remove_

...

input_

...

state.py

Several tests perform direct StudentModule.object() access.

...

One-off Django management command.

Call ChainQueryMaps to New Query
resetfix_studentmodules_studentin_attemptslist()

module _to_reset = StudentModule.objects.get(
student_id=student._module_id,)

coursehist_id=course_id,
module_state_key=module_state_key
)

 Then .delete() or .save()

 

modules = StudentModuleHistory.objects.filter(student_module_id=student_module_id)

None - will be deleted.

lms/djangoapps/

...

courseware/management/commands/tests/

...

test_clean_

...

history.py

 
Call ChainQueryMaps to New Querycalculate_task_statistics()

student_modules = StudentModule.objects.filter(

module_state_key=location, student__in=students

).order_by('student')

QueryMaps to New Query
From many tests: write_history()

INSERT INTO courseware_studentmodulehistory
(id, created, student_module_id)
VALUES (%s, %s, %s)

---
From many tests: read_history()

SELECT id, created, student_module_id FROM courseware_studentmodulehistory

---

lms/djangoapps/courseware/tests/test_model_data.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/

...

courseware/tests/test_

...

module_render.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/tests/test_enrollment.py

Several tests perform direct StudentModule.object() accessTwo tests - test_xmodule_runtime_publish() & test_xmodule_runtime_publish_delete() - use direct StudentModule.objects() access.

lms/djangoapps/

...

courseware/tests/test_

...

submitting_problems.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/

...

enrollment.py

...

Call ChainQueryR/W?Maps to New Query
instructorreset_student_dashboardattempts()

smdat module_to_reset = StudentModule.objects.filterget(
student_id=student.id,
course_id=course_keyid,
module_state_key=module_state_key
) smdat = smdat.order_by('student')

 

 Then .delete() or .save()

RW001

lms/djangoapps/instructor/

...

management/commands/openended_stats.py

Call ChainQueryR/W?Maps to New Query
getcalculate_extendedtask_duestatistics(course, unit, student)

student_module modules = StudentModule.objects.getfilter( student_id=student.id,
course_id=course.id,
module_state_key=unit.location
)

 

set_due_date_extension(course, unit,

student, due_date)

student_module = StudentModule.objects.get(
student_id=student.id,
course_id=course.id,
module_state_key=node.location
)

Then .save()

or

student_module = StudentModule.objects.create(
student_id=student.id,
course_id=course.id,
module_state_key=node.location,
module_type=node.category
)

 
 dump_module_extensions(course, unit) query

module_state_key=location, student__in=students

).order_by('student')

R

007

(course, module, students)

lms/djangoapps/instructor/tests/test_api.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/tests/test_enrollment.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/tests/test_tools.py

Several tests perform direct StudentModule.object() access.

lms/djangoapps/instructor/views/legacy.py

Dumps CSV of problem reponses.

 
Call ChainQueryR/W?Maps to New Query
instructor_dashboard()

smdat = StudentModule.objects.filter(
course_id=course.id_key,
module_state_key=unit.location)

 dump_student_extensions(course, student)

query = StudentModule.objects.filter(
course_id=course.id,
student_id=student.id)

module_state_key
)
smdat = smdat.order_by('student')

R

None - who needs this?

Provide another inteface for it.


lms/djangoapps/instructor_task/tasks_helper.py

Call ChainQueryR/W?Maps to New Query
perform_module_state_update()

modules_to_update = StudentModule.objects.filter(

course_id=course_id, module_state_key__in=usage_keys

)

 

usage_keys

)

Also supports a further student and generic filter_fcn() filtering.

RW

001 / 004

BUT - support of generic filter_fcn()

probably not possible.

lms/djangoapps/instructor_task/tests/test_base.py

...

Staff-graded assignments XBlock.

Call ChainQueryR/W?Maps to New Query
staff_grading_data()

module, _ = StudentModule.objects.get_or_create(
course_id=self.course_id,
module_state_key=self.location,
student=user,
defaults={
'state': '{}',
'module_type': self.category,
})

RW ---
staff_upload_annotated()

module = StudentModule.objects.get(pk=request.params['module_id'])

And finally, .save()

RW ---
staff_download_annotated()

module = StudentModule.objects.get(pk=request.params['module_id'])

 RW---
enter_grade()

module = StudentModule.objects.get(pk=request.params['module_id'])

And finally, .save()

RW ---
remove_grade()

module = StudentModule.objects.get(pk=request.params['module_id'])

And finally, .save()

 RW---

src/edx-sga/edx_sga/tests.py

...

Django management command that migrates existing SGA submissions for a course from old SGA implementation to newer version that uses the 'submissions' application.

Call ChainQueryR/W?Maps to New Query
 

student_modules = StudentModule.objects.filter(
course_id=course.id).filter(
module_state_key__contains='edx_sga')

R ---