2021-05-28
Agenda
Follow up on slack message related to Caliper
Clarification request on Processor: Validator in OEP-26 proposal doc
Setting up periodic meeting/status sharing
Discussion on previous Caliper certification attempt
Discussion regarding scalability of solution (meaning, concerns, challenges etc.)
Discussion on ADR procedure
Notes
Preventing inadvertent PII leakage
Requires all new transformers to use
extract_subdict_by_keys
properlyHow to ensure this happens for all new event types?
Create a test harness - that is a base test for all events
Have an ADR
publicized on the README
explains our approach to this PII concern
links to the test
[Linter - automated parsing of code that checks for new event types and warns the developer]
Decision: Move forward with #1 and #2. Aamir will verify #1 with Zia.
IMS Certification
Membership is required for Test site access and slack access.
Will revisit once the rest of the framework is further along and we’ve added the course-completion event.
Scalability
#1 Performance retained with growth of users and events or #2 Adding more event types?
Answer: #1
1. Performance concerns
Granular code optimizations - discovered those through Python profiling
Doing heavy-lifting operation within the web request of a user/learner
Not block on responding to web requests
Web request
→ emits event → happens in the web process
→ transformation of event → ? is this happening in a separate celery process or in the original web process?
→ sending events to consumers → asynchronous job
Performance Testing: How to Test Performance
Load Testing
Heavy-weight - instead, prefer gradual rollout and monitoring in production
Profile Testing
Python Profile testing tool
2. Reliability and Resiliency concerns
sending events to consumers → asynchronous job
Handling errors when consumers are offline
Handling errors when our own backend system breaks temporarily
3. Latency concerns
Latency - meaning - how long does it take between a learner performs an action and the consumer receives the corresponding event?
We’ll want to optimize this at some point - but it doesn’t have to be part of v1.
For adaptive learning (in the future), we’ll want latency to be as minimal as possible.
Cadence: Weekly
ADRs
long-term compatibility and API support
To ensure have a strategy for not breaking our consumers when fields are added/removed/etc
Error handling
Notice silent failure if field doesn't exist in the edX event:
Validators
IMS certification testing → 1-2x a year
A nice-to-have - can revisit at another time
2021-06-02
Agenda
Discussion on solution for addressing accidental PII leakage
Setting up and maintaining a list of fields that would be considered PII
Discussion on ADR for backwards compatibility
Discussion on error handling if a field is not found in an event
Way forward for ADR of enterprise_uuid related modifications in edx enterprise and edx platform
Notes
1 and 2. PII leakage
Problem statements
A - A new event type is created. The integration developer forgets to call
extract_subdict_by_keys
for that new event type. Hence, all fields are forwarded on.B - Even though an event type calls
extract_subdict_by_keys
, developer doesn’t realize that field X is a PII field.
Possible solutions
For Problem B
1 - Blocklist of field names - the code in
extract_subdict_by_keys
then does fuzzy comparison to ensure the extracted field is not in the blocklist.Note: need to handle “fuzzy” name matching
For example: “ip_address” is listed in Blocklist versus “ip_addr” used in edX event
For Problem A
2 - Create a test harness - that is a base test for all events - leveraging Base Transformer
Double check that
extract_subdict_by_keys
is called.
3 - Have an ADR
publicized on the README
explains our approach to this PII concern
says new event types MUST call
extract_subdict_by_keys
links to the test
4 - Linter (LATER)
[Linter - automated parsing of code that checks for new event types and warns the developer]
5 - Standardization Process (LATER)
3. Backwards compatibility
What do the specs say?
Add our own versions
Field Types
Spec-required
Spec-optional
OpenedX-optional
Problem statements
A - We add a new field to our transformed xAPI/Caliper event.
Solution idea: bump our own minor version (separate from spec).
Good communication tool to inform community and consumers.
B - We remove an existing field from our transformed xAPI/Caliper event.
B1 - We remove an existing field from our transformed xAPI/Caliper event, which is core to the spec.
Solution - Never do this! Caught by our tests.
B2 - We remove an existing field from our transformed xAPI/Caliper event, which is optional in the spec or absent from the absent.
C - The edX event removes fields.
C1 - Removed fields are required for an xAPI/Caliper.
C2 - Removed fields are optional for an xAPI/Caliper.
(NON-Problem) D - The edX event adds new fields to an existing transformed xAPI/Caliper event.
4. Error handling
Acked. No longer creating silent failures.
5. Enterprise
check-in with George
enterprise_uuid - ADR review; PR review - Engineer
project plan for rollout - PM
2021-06-10
Agenda
Review the ADR for addressing PII leakage issue
What: We are NOT letting fields pass through, unless they are explicitly extracted.
How: Abstraction layers
accessor method to extract the field.
extract_subdict_by_keys
Future Ideas (maybe future iterations):
Automation tooling and linting
Fuzzy comparison
Review the ADR for version control of transformers
This decision is per-event type. Each event transformer will have its own versioning value.
This is NOT for package versions. Use https://open-edx-proposals.readthedocs.io/en/latest/oep-0047-bp-semantic-versioning.html instead.
Will use Major and Minor version numbers.
Mapping document for events xAPI and Caliper
Connecting with Enterprise
sent an email to George
Async & Resiliency
problem statements: not blocking on web workers
handling failure scenarios when the recipient is offline.
Action Items
Project Plan
2021-06-25
Agenda:
PR for find_nested method
Update on discussion with George
Review project plan
Notes:
find_nested
- be mindful of potential performance costs with deep searches in python JSON structuresADRs are currently in
pending
state.ACTION Nim: review the ADRs
Project plan
ACTION Edly: ADR for access control
ACTION Edly: Update Read-the-docs to explain each config setting
ACTION Edly: ADR for resiliency
2021-07-08
Agenda:
Discuss comments on access control ADR PR
Decision: Update ADR to be explicit about enterprise access control and university-partner access control.
The logic has been agreed upon and we can start the development.
Discuss PR for problem_check events
Requested Nimisha’s feedback. Not a high priority task.
Discuss project plan
Discuss problem_completed and video_completed event
Take a look at the completion framework and its usage: https://github.com/edx/completion
We need to use completion framework for video.completed as it is for now.
Problem completed event may be skipped for initial release because it’s logic is no different from problem.submitted
Notes:
Allowlist instead of Whitelist
Add a separate epic for resiliency
Persistence is part of the to-do list but MAYBE we can do without it in the initial release. Need to ask George what their current strategy is.
Clarify treatment (# of retry attempts, persistence, etc) of different exception types
Ensure debug logs are self-evident and clear - so we know for what reason an event was skipped
Look at grades/tasks.py as a model example - of usage of persistence and differential treatment of exception types.
2021-07-15
Agenda:
Discuss ADR for resiliency
Delete 3
Elaborate on saving celeray tasks instead of events
Every time a persisted event is retried, a method checks the number of entries for xapi/caliper and generate an alert (relic, log etc.)
Discussion on having filter processors as synchronous or asynchronous
Move event name filter to synchronous layer and have string comparisions instead of regex.
Adding Unique event IDs - in the synchronous processor
Add in ADR: have a robust method to avoid duplicate events in case one of the LRS is down
Discussion on problem_check ADR
Changing a type to a list - could be an issue for LRS parsers.
Add question in the statement and mention in ADR that it can be incomplete (incase of multiplechoice etc.)
Explore sending multiple events in the case of multiple questions in a single problem
Maybe divide the grade among events and add a “parent_problem_grade” in the events
Maybe send an additional event having the overall grade of the problem.
2021-07-28
Final review of ADR for resiliency
#1 - add detail of nested celery task
#4 - Discussion with Zia whether changes will be generic or specific to app
#3 - remove the link is down thing
Filters based on event name reconfigured as to synchronous processors
This is now done.
Unique event IDs added to xAPI events (already exist in Caliper events)
This is now added.
LRS can rely on this - if needed for extraneous circumstances.
Discussion on problem_check ADR (list response)
Inputs we can get
Roy Shillo - long-term consumer and producer of xAPI statements
IMS representative
Canvas and Moodle’s implementation
edX team that owns CAPA problem
2021-08-03
Agenda:
Discussion on PRs related to course completion event (old and new PR)
Discussion on PR related to change in ADR of xAPI
Decision:
edx.course.completed will be emitted based on timestamp of persisted grade in the database.
Need to emit course.passed and course.failed events in addition to course.completed, when learner achieves a passing or failing grade respectively.
Implementation notes: code is probably better placed within the grades django app, rather than a common utility folder or the certificates app.
2021-08-23 (meeting with Roi and Merav, Campus.il)
As next steps (see #4 from 2021-07-28):
ii. IMS - can connect after researching iii.
iii. In addition to Canvas and Moodle, check h5p.org.
v. Can also connect with Learning Locker, as the developer of LRS.
Vocabulary
2021-08-25
PR Reviews
Compounded Problem Type Events
Moodle generates multiple answered events for each item in a Survey
Moodle decouples scoring event from the problem-answered event for the Drag-n-Drop problem
2021-09-10
Final PRs are being merged.
Final end-to-end testing in progress for xAPI.
Using Scorm Cloud, same developers as TinCan API.
Caliper’s LRS is an open-source one without any validations.
Caliper certification - seeking an edX squad to own this.
Remainder Effort
Enterprise usage
Caliper certification
Multi-problem CAPA
Community Evangelization
Video demo
Configuration documentation
Opinions on event transformations
Better to have a consistent standard model, than multiple variations.