Follow up on slack message related to Caliper
Clarification request on Processor: Validator in OEP-26 proposal doc
Setting up periodic meeting/status sharing
Discussion on previous Caliper certification attempt
Discussion regarding scalability of solution (meaning, concerns, challenges etc.)
Discussion on ADR procedure
Preventing inadvertent PII leakage
Requires all new transformers to use extract_subdict_by_keys properly
How to ensure this happens for all new event types?
Create a test harness - that is a base test for all events
Have an ADR
publicized on the README
explains our approach to this PII concern
links to the test
[Linter - automated parsing of code that checks for new event types and warns the developer]
Decision: Move forward with #1 and #2. Aamir will verify #1 with Zia.
Membership is required for Test site access and slack access.
Will revisit once the rest of the framework is further along and we’ve added the course-completion event.
#1 Performance retained with growth of users and events or #2 Adding more event types?
1. Performance concerns
Granular code optimizations - discovered those through Python profiling
Doing heavy-lifting operation within the web request of a user/learner
Not block on responding to web requests
→ emits event → happens in the web process
→ transformation of event → ? is this happening in a separate celery process or in the original web process?
→ sending events to consumers → asynchronous job
Performance Testing: https://openedx.atlassian.net/wiki/spaces/AC/pages/29688477
Heavy-weight - instead, prefer gradual rollout and monitoring in production
Python Profile testing tool
2. Reliability and Resiliency concerns
sending events to consumers → asynchronous job
Handling errors when consumers are offline
Handling errors when our own backend system breaks temporarily
3. Latency concerns
Latency - meaning - how long does it take between a learner performs an action and the consumer receives the corresponding event?
We’ll want to optimize this at some point - but it doesn’t have to be part of v1.
For adaptive learning (in the future), we’ll want latency to be as minimal as possible.
long-term compatibility and API support
To ensure have a strategy for not breaking our consumers when fields are added/removed/etc
Notice silent failure if field doesn't exist in the edX event:
IMS certification testing → 1-2x a year
A nice-to-have - can revisit at another time
Discussion on solution for addressing accidental PII leakage
Setting up and maintaining a list of fields that would be considered PII
Discussion on ADR for backwards compatibility
Discussion on error handling if a field is not found in an event
Way forward for ADR of enterprise_uuid related modifications in edx enterprise and edx platform
A - A new event type is created. The integration developer forgets to callextract_subdict_by_keys for that new event type. Hence, all fields are forwarded on.
B - Even though an event type calls extract_subdict_by_keys, developer doesn’t realize that field X is a PII field.
For Problem B
1 - Blocklist of field names - the code in extract_subdict_by_keys then does fuzzy comparison to ensure the extracted field is not in the blocklist.
Note: need to handle “fuzzy” name matching
For example: “ip_address” is listed in Blocklist versus “ip_addr” used in edX event
For Problem A
2 - Create a test harness - that is a base test for all events - leveraging Base Transformer
Double check that extract_subdict_by_keys is called.
3 - Have an ADR
says new event types MUST call extract_subdict_by_keys
4 - Linter (LATER)
5 - Standardization Process (LATER)
What do the specs say?
Add our own versions
A - We add a new field to our transformed xAPI/Caliper event.
Solution idea: bump our own minor version (separate from spec).
Good communication tool to inform community and consumers.
B - We remove an existing field from our transformed xAPI/Caliper event.
B1 - We remove an existing field from our transformed xAPI/Caliper event, which is core to the spec.
Solution - Never do this! Caught by our tests.
B2 - We remove an existing field from our transformed xAPI/Caliper event, which is optional in the spec or absent from the absent.
C - The edX event removes fields.
C1 - Removed fields are required for an xAPI/Caliper.
C2 - Removed fields are optional for an xAPI/Caliper.
(NON-Problem) D - The edX event adds new fields to an existing transformed xAPI/Caliper event.
4. Error handling
Acked. No longer creating silent failures.
check-in with George
enterprise_uuid - ADR review; PR review - Engineer
project plan for rollout - PM
Review the ADR for addressing PII leakage issue
What: We are NOT letting fields pass through, unless they are explicitly extracted.
How: Abstraction layers
accessor method to extract the field.
Future Ideas (maybe future iterations):
Automation tooling and linting
Review the ADR for version control of transformers
This decision is per-event type. Each event transformer will have its own versioning value.
This is NOT for package versions. Use https://open-edx-proposals.readthedocs.io/en/latest/oep-0047-bp-semantic-versioning.html instead.
Will use Major and Minor version numbers.
Mapping document for events xAPI and Caliper
Connecting with Enterprise
sent an email to George
Async & Resiliency
problem statements: not blocking on web workers
handling failure scenarios when the recipient is offline.
Review of PR and ADR related to PII leakage
PR for find_nested method
Review of PR and ADR related to transformer versions
Update on discussion with George
Review project plan
find_nested - be mindful of potential performance costs with deep searches in python JSON structures
ADRs are currently in pending state.
ACTION Nim: review the ADRs
ACTION Edly: ADR for access control
See organization tpes in edX system
ACTION Edly: Update Read-the-docs to explain each config setting
ACTION Edly: ADR for resiliency
Discuss comments on access control ADR PR
Decision: Update ADR to be explicit about enterprise access control and university-partner access control.
The logic has been agreed upon and we can start the development.
Discuss PR for problem_check events
Requested Nimisha’s feedback. Not a high priority task.
Discuss project plan
Discuss problem_completed and video_completed event
Take a look at the completion framework and its usage: https://github.com/edx/completion
We need to use completion framework for video.completed as it is for now.
Problem completed event may be skipped for initial release because it’s logic is no different from problem.submitted
Allowlist instead of Whitelist
Add a separate epic for resiliency
Persistence is part of the to-do list but MAYBE we can do without it in the initial release. Need to ask George what their current strategy is.
Use persist on failure
Clarify treatment (# of retry attempts, persistence, etc) of different exception types
Ensure debug logs are self-evident and clear - so we know for what reason an event was skipped
Look at grades/tasks.py as a model example - of usage of persistence and differential treatment of exception types.
Discuss ADR for resiliency
Elaborate on saving celeray tasks instead of events
Every time a persisted event is retried, a method checks the number of entries for xapi/caliper and generate an alert (relic, log etc.)
Discussion on having filter processors as synchronous or asynchronous
Move event name filter to synchronous layer and have string comparisions instead of regex.
Adding Unique event IDs - in the synchronous processor
Add in ADR: have a robust method to avoid duplicate events in case one of the LRS is down
Discussion on problem_check ADR
Changing a type to a list - could be an issue for LRS parsers.
Add question in the statement and mention in ADR that it can be incomplete (incase of multiplechoice etc.)
Explore sending multiple events in the case of multiple questions in a single problem
Maybe divide the grade among events and add a “parent_problem_grade” in the events
Maybe send an additional event having the overall grade of the problem.
Final review of ADR for resiliency
#1 - add detail of nested celery task
#4 - Discussion with Zia whether changes will be generic or specific to app
#3 - remove the link is down thing
Filters based on event name reconfigured as to synchronous processors
This is now done.
Unique event IDs added to xAPI events (already exist in Caliper events)
This is now added.
LRS can rely on this - if needed for extraneous circumstances.
Discussion on problem_check ADR (list response)
Inputs we can get
Roy Shillo - long-term consumer and producer of xAPI statements
Canvas and Moodle’s implementation
edX team that owns CAPA problem
Discussion on PRs related to course completion event (old and new PR)
Discussion on PR related to change in ADR of xAPI
edx.course.completed will be emitted based on timestamp of persisted grade in the database.
Need to emit course.passed and course.failed events in addition to course.completed, when learner achieves a passing or failing grade respectively.
Implementation notes: code is probably better placed within the grades django app, rather than a common utility folder or the certificates app.
As next steps (see #4 from 2021-07-28):
ii. IMS - can connect after researching iii.
iii. In addition to Canvas and Moodle, check h5p.org.
v. Can also connect with Learning Locker, as the developer of LRS.
Compounded Problem Type Events
Moodle generates multiple answered events for each item in a Survey
Moodle decouples scoring event from the problem-answered event for the Drag-n-Drop problem
Final PRs are being merged.
Final end-to-end testing in progress for xAPI.
Using Scorm Cloud, same developers as TinCan API.
Caliper’s LRS is an open-source one without any validations.
Caliper certification - seeking an edX squad to own this.
Opinions on event transformations
Better to have a consistent standard model, than multiple variations.
remaining: course completion and course enrollments with enterprise
then: schedule a demo with edX folks
Eliminating dependency with edx-platform