Data WG 2022-03-10 Meeting Notes

5M

Brainstorm Lightening talks

@Edward Zarecor in absentia – can push to next time.

• We are encouraging all Working Groups to propose lightening talks.

  • The talk could simply be about the group to encourage joining

• We missed the window, but could still propose some topics in case there are drop outs

• Ideas:

  • Explain group charter, discuss areas of focus

  • Thoughts about the future of Insights

    • What’s the community option here

    • How do we as a community collaborate

    • What’s the batteries included option for the community

  •  

5M

Work Group Summit on Friday, April 29th

@Edward Zarecor in absentia – can push to next time.

• There will be an opportunity meet in person at the conference

• Who will be attending?

  • @Edward Zarecor

  • @Simon Chen

  • @Maria Fernanda Magallanes Z – hoping to go

  • @Andrés González

  • @Diego Millan

• We will also have Working Group colloquium for all groups to get together to discuss process and practices

  • 
    
    

10M

Vector, Clickhouse and Superset

@Maria Fernanda Magallanes Z

• These are the tools Cairn uses

• Have you used them?

• Have you considered using these tools for an open source proposal?

• What other tools could be used?

• What small steps could be done to start contributing?

• Notes

  • Andy has been looking at next gen insights but cannot guarantee that 2U would prioritize this work.

    • They are looking at buy/build for the front end

      • Andy has been doing an evaluation/exploration of Superset.

      • @Tobias Macey can connect Andy or others with the Superset team if we have any issues or blockers

    • They are likely to use their proprietary ELT/ETL tools

    • The big sticking point for them is course level security – Superset usage will depend on this

      • Tobias mentions OSS data governance tools like Apache ranger, and proprietary like Immuta.

    • Row level security conversation

      • Is appropriate course/org/admin security currently in our existing security model

    • Is there anyone who thinks we should continue to maintain the Insights frontend

      • Pierre Maillot – his opinion is being shared second hand for the record – would like it to be supported unless it’s current functionality is available and there’s a sensible migration path.

  • Independently @Edward Zarecor has been looking into MySQL replication into Clickhouse

  • Tracking log synchronizing tools

    • Currently in Cairn vector

      • small batches

      • no delivery guarantees

    • Kafka

    • Pulsar

  • Events, what’s next

    • Should we formalize our events at the level of standard protocols xAPI/Caliper?

    • Can/should we still support flexible event in the tracking even stream?

    • Could we start to enforce conformance when an event graduates from tracking.log to supported xAPI/Caliper event?

    • Any recommends that the community avoid pulling all of 2U’s problems along if they don’t need to.

    • [ ] Could we have a task to deal with truncated events?

    • [ ] Adding a version specification to event?

    • [ ] Explore the delta between what we need today and existing supported events on the xAPI/Caliper side

    • Has 2U considered replacing the data backend with a purpose built LRS.

  • Build/buy data visualization tool

    • [ ] Superset discovery

  • BI events and pedagogy events feel distinct, could we isolate BI events to the event stream

10M

ElasticSearch upgrade

@Edward Zarecor

• There’s currently work on-going at 2U to determine how they will deal with the ES/OS fork

• Jeremy Bowman discovered this library that might open up the possibility of moving away from ES.

• There’s this DEPR ticket

• Some of us were involved in this conversation, but there was no convergence/resolution on the thread.

10m

Insights for tutor users

@Sofiane Bebert

• Thread - Users looking to transition from native to tutor but who still need to run Insights.