Data WG 2022-05-18 Meeting notes

10 min

Andrés González

• Announcing the first beta releases of panorama-elt and tutor-contrib-panorama, the basic tools to integrate Open edX and other systems into a datalake. Contributions are welcome!

• Notes

  • Python based ELT toolkit that attempts to be modular and support diverse data sources and data lakes

  • Currently focused on AWS and only supports Athena today

  • Tutor plugin allows running the ELT tools alongside tutor, but expects an AWS destination for the data

  • Full support is currently only available in the Kubernetes version, locally only the ELT part for RDBMS tables

  • Athena TLDR;

    • put files in an s3 bucktet

    • Athena allows SQL over CSV, JSON and other formats

    • Athena is based on Hive, so there is something available that is open source, but there are no plans to work on this

  • The plugin is usable for local installations and dev installations

    • Docs are currently empty

    • Tracking logs are not supported locally

5 min

Insights 2.0?

Edward Zarecor

• At the conference the 2U product team spoke about “Insights 2.0.”

• Can someone describe the scope of that effort?

• Will it be valuable outside of 2U?

10 min

Bite-sized work

Edward Zarecor

Let’s think about what bite-sized work would be valuable to start on now and commit to doing some of it. I have a few idea.

• https://github.com/openedx/data-wg/issues/14

• https://github.com/openedx/data-wg/issues/15

• https://github.com/openedx/data-wg/issues/16

2 min

Starting a conversation about Embargoing the Open edX event system

Edward Zarecor

• https://github.com/openedx/data-wg/issues/6

• I’ll start with a thread in discourse to discuss

• Hope to have that published later today

5 min

Completion Aggregator

Sofiane Bebert

• openedx-completion-aggregator