Approach Memo & Technical Discovery: React-based Video Player

Owned by Jenna Makowski
Last updated: Apr 26, 2023 by Sarina Canelake
11 min read

Project Leads (Product): @Brad Brown @Jenna Makowski @Jeremy Ristau @Sarina Canelake

Date Proposed: 14-02-2023

Last Updated: 19-04-2023

Status: Community Review

 

Improving the Video Editor

Many edX partners feel that the Video Editor experience is lacking and needs improvement. We have outlined a three step process to improve the video editor. These steps are:

  • Update video editor settings (this work is complete). This has moved the Video Editor experience (settings) away from the older frontend and to the new(er) course authoring MFE. The feature parity replacement of settings has allowed us to move the existing experience from edx-platform/studio-frontend into the Course Authoring domain/MFE. This advances our architectural vision, as well as allows us to isolate Video Editor into its own deployable subsystem. New features like Video Uploads are now easier and faster to implement, and are done on a new tech stack so we do not incur more tech debt/increase the monolith. 

  • Build a React based video player (the focus of this document), which is a requirement for video previews as well as the new video uploads. The video uploads path is planned to be incorporated into the Video Editor. The new Video Editor is react based, and the current player does not support that.  We want to have an embedded video preview in the editor after upload, and cannot deliver that until we have a react-based player. The new video player will improve upon the user experience for additions and uploads to edX courses and reduces maintenance costs.

  • Construct the new video uploads (a future project), which is a faster and easier process and addresses one of the most common course author complaints (video management tools in general and uploads in particular are inefficient).

 

Additionally, Phase 2 of the Modular Content Initiative is reliant on the completion of BD-14 work. The video upload enhancement is the final piece of BD-14 (along with the currently ongoing visual problem editor work). Therefore, a new React based video player enables the video uploads enhancement, and the completion of BD-14 enables Phase 2 of modular content. 

Problems

Current Video Player

  • Currently, many members of the Open edX community have embarked on creating their own video player implementations to overcome deficiencies with the player in the standard Open edX installation. The work represented in this document will result in an editor that the community can coalesce around. The benefits of this are a community-supported video editor that everyone can agree should be used. Our goal is to come to community consensus on the problems of the current video player and agree upon an implementation direction.

  • The current video player cannot be rendered directly inside of an MFE

  • The current video player does not work as a "video preview" player inside of the Video Settings Editor

  • It is not possible to have video preview on the video uploads page with the current player, as described above.

Platform Map

 The “Platform Map” is a list of all features throughout the platform that 2U maintains. Periodically, 2U surveys their partners to get feedback on how the platform is working, and maps that feedback to Platform Map areas.

  • Video Block Platform Map Rating: The data in the chart below are from interviews with 17 partners of the edx.org website. They were asked to rate, for a variety of features, how critical a given feature is (“Criticality”) and how far behind the Open edX software is on delivering this feature at a satisfactory level (“Sentiment Behind”). The “gap rating” indicates features that are the most critical and the most behind, and therefore need the most attention.  

  • The survey showed that the video block is rated as the fourth most important “feature gap” from all the features in courseware + Studio For TNL (behind import/export, text block, and problem block). [Note: TNL stands for Teaching and Learning, an engineering team and focus area at 2U] The map shows that 63% of edx.org partners feel that we are behind on video and 100% feel that the video block is a critical feature. After our work on video settings, React-based player upgrade, and video upload improvements, the team at 2U/edX we will conduct user interviews to test whether these ratings have improved.

Product Requirements

Basic Requirements

  • Unified video player experience across the platform (Studio, Content Libraries, LMS)

  • Will work in units with multiple types of components (text, problems, etc)

  • Player is FOSS and compatible with the Open edX platform’s software license

Parity with Current Video Player

MFE Compatibility

  • Must work inside of an MFE directly

    • Specifically, the course-authoring MFE, Library Authoring MFE

    • Rework the Video component so that it directly renders in the Learning MFE with no iframe 

  • Contingency: The legacy LMS frontend uses a way out-of-date version of React, so getting a new video player working within the iframe may present its own challenges. However, an upgrade of React in the legacy LMS frontend is on the Roadmap.

Enhancements from Current Player Capabilities

  • Ability for authors to upload multiple video files, and for video viewers to select from different video options

    • Use case 1: Internationalization of videos by providing different language options

    • Use case 2: Accessibility option for Blind individuals by providing an Audio Description track

  • Support for DASH video stream

  • Support for Audio Descriptions - more detail in page comment Re: Approach Memo & Technical Discovery: React-based Video Player | Comment

  • Need a player that will work as a "video preview" player inside of the Video Settings Editor (see Appendix for Video Preview Mock). A new video player is required to include a video preview on the Video Settings page. The existing video player is (according to previous discovery) a pre-React, pre-webpack player which is not viable for our use case of having a Video Preview capability in the Video Settings Editor.

  • Player supports the ability to preview video slices (e.g. a chapter) to an external source (e.g. having videos previewed inline for a search engine’s results)

  • Have the player be a standalone, embeddable player for other platforms outside of Open edX/edx.org.

  • Perform downscaling/play downscaled video to handle both high and low-bandwidth clients

    • Open questions to be solved in technical discovery: Should this be automatic, manual, or both? Should this be dynamic, or rely on multiple hosted file options?

    • We’re open to any set of options depending on outcome of discovery + feasibility.

  • Ability to track buffering issues that occur during video streaming

    • Currently, the player lacks reporting on buffering issues, making them impossible to diagnose

  • Support for additional in-video features, such as comments

  • Display VTT captions

Data Compatibility

We have a lot of analytic events that get triggered from the current video player. We MUST keep these requirements as-is, or supply an ADR regarding what would change.

  • The events we currently support are here:

event-routing-backends/docs/event-mapping/Supported_events.rst at master · openedx/event-routing-backends

  • And new ones are being added for enabling / disabling captions, and changing video speed in this PR currently:

feat: Added support for video interaction events by ziafazal · Pull Request #245 · openedx/event-routing-backends

Nice to Haves

Technical Discovery

Background

The current video player cannot be rendered directly inside of an MFE. A new video player is required to include a video preview on the Video Settings page. The current build of the video player has the video player with a custom control bar. The control bar is generated based on the values stored in the Video XBlock. All the features in the custom control bar are part of the regular HTML5 video component except transcripts. It is important to note that transcripts and captions are two different functions. The captions are played in the video view while the transcripts are played next to the video. These two functions do rely on the same file to generate content.

Note: Transcripts are saved as .srt files but in the backend are converted to .sjson

Initial Technical Discovery (edX)

Initial edX Technical Discovery (React video player)

Technical Requirements

  • It needs to be a Javascript-based component

  • Almost certainly, it needs an HTML5 video component base

  • There are a slew of legacy video player compatibility requirements, including (but possibly not limited to):

    • Discovery will need to be done on what fields currently exist in the XBlock, and as much as possible should go through the Deprecation process before beginning the process of migrating any over

      • Such as the multi-speed YouTube upload option, which we used in 2012 before YT started generating multiple speeds for you. Open questions: Is anyone using video bumpers? What features weren’t ported to the MFE, or are no longer exposed in the UI? etc

    • Discovery will also address the details of how we manage transcript files today

      • Transcripts aren’t standardized and may be in various locations - YouTube, local storage, or in edx-val - a system that edx.org uses, but unclear if anyone in the community does

    • What do we need to do with legacy Transcript utilities?

  • Ideally, this is a drop-and-replace player within the current XBlock, such that instances upgrading the version of the block would get the player across all courses with no intervention

    • If this proves to be difficult, what would be the plan for a new XBlock + migration plan to the new block? How much more/less difficult is this?

Proposed Options

Option 1: https://www.npmjs.com/package/React-player

Option 2: https://www.npmjs.com/package/video-React (based on npm: video.js )

Option 3: npm: video.js   (This comes highly recommended by edX’s accessibility specialist. Additionally, it is being used successfully by Prospectus, edX’s package for displaying the course catalog on edx.org )

Option 4: npm: plyr

Opinions and Further Questions

Frontend Input 

  • For subtitles, moving to vtt is probably the right path forward. It's a well documented format.

  • For transcripts, vtt will also serve us well. There are plenty of ways to parse vtt and make it pretty.

  • The video player should support theme / skin customization. Ideally, we’d be able to associate CSS variables from Paragon to the styles for the video player so that things like colors are consistent between the video player and the rest of the application.

  • Consider the right home for the new video player (e.g., is it a new UI component in the Paragon design system?). If one of the goals of this project is to ensure consistent video player experience across the platform, it may make sense to live in Paragon.

Multiple Components

  • How will this play with units that have multiple types of components? For example: text followed by a video followed by a multi-choice problem.

  • At some point, I expect folks will want other types of components to run outside the IFrame as well. Can the Video component be reimplemented in a way that leaves the door open to a more general solution for React-based learning components?

Is there a reason that we shouldn’t migrate to using .vtt files? 

  • At the very least, we need to get rid of .sjson files in favor of using SRT-only. The advantage of using SRT only is because they exist in the filesystem (and edx-val) already, so the migration path is simple. We’d be able to dump a lot of hacky sjson code. However, VTT comes with better accessibility options. So…

  • It mostly comes down to hammering out the migration path. We do a lot of hacky SRT/SRT-JSON conversion, and we store these files in a few different ways. Also some of the metadata we have that points to these files will be in OLX, meaning that changes to it will trigger course publishing. We can do things to work around this, but they all add to the complexity of the XBlock itself.

  • A possible way to migrate to VTT: Scan filesystem, make VTT equivalents of all sjson files - we can do this via file upload because that won’t trigger a course publish. Once we have that, we then have the player code preferentially look for the VTT files. 

    • On edx.org, some of the subtitle stuff is managed via SRTs in edx-val. When that happens, the SRT files are loaded into S3 and that overrides everything else. So in that case, migration would be 2 steps; one migration of edx-val (making new VTT files) then updating db refs to srt & sjson files. 

  • It would also be possible to convert SRT to VTT on-the-fly.  Apparently, it is not rocket science.

Compatibility

  • The video player is layer upon layer upon layer of crufty compatibility code. We could probably make something a lot simpler overall if we created a new XBlock entirely and migrated video data over to it, though that would make gradual rollout/rollback more difficult.

  • It is believed that the way to do this while keeping some sense of sanity is to have an explicit transformation step in the code (even if it's not persisted).

    • Make it so that the new video player has a simple, sane data model to follow, and write a terrible shim layer to convert (up front and in one place) the layers of old terrible into the format the new system expects. 

    • Some pieces of data can just be dropped entirely, like multi-version-YT nonsense used ages ago to provide different video speeds (before YT provided this functionality natively).

    • New videos can just persist that new, sane data model, while old ones get converted as needed.

  • Most backwards compatibility is focused on not breaking content, even as we discover and implement better ways to do things.

Recommendation

We recommend video.js be used for this project.

edX engineering has concluded that the four options have many commonalities. edX advocates for option number three, video.js, in that this option is best suited to handle edX’s accessibility requirements (WCAG 2.1). However, we do not feel that those requirements trump all other requirements, or that the customizability of other players could not get us to full WCAG coverage. This means that while edX currently favors option 3, we are interested in hearing other opinions as to which option best suits the needs of the Open edX community. 

Axim’s recommendation is also for option three, video.js.  It is not only the more mature project, but also the more consistently maintained one.  In addition, the fact it is not written in React makes it the more flexible option: it will be possible to write an in-house React component based on video.js, but tailored specifically to the (many) project specifications without second-guessing the underlying project's React assumptions.  There is also ample documentation on how to achieve this.  Finally, video.js' plugin architecture makes it possible to implement virtually any needed feature that is not already exposed by video.js itself or one of the plethora of existing plugins.  As an added bonus, DefinitelyTyped supports video.js as of a few months ago.

Appendix

Video Uploads

The project to improve video uploads is contingent on a React-based player. Video uploads is a separate workstream that can and will be prioritized after the new video player is built. With the newly enhanced video settings, video uploads are easier and faster to implement, and are also done on a new tech stack so we do not incur more tech debt. 

User Problem: The process of uploading videos needs to be improved as can be seen from three comments from edX user interviews in regard to the video block

  • “Really struggling with this one right now. Process to upload a video in one place, copy the ID and figure out where you want it to be and then separately upload the captions, has been a real challenge for us. And then if we need to make a correction or update to the video, we have to re-upload the video again.” 

  • “The video management tools in general are probably one of the most inefficient systems in edX. Makes her crazy that she has to:

  1. upload videos via one tab

  2. copy a code

  3. go back

  4. edit that component

  5. paste that bit

  6. delete the YouTube link

  7. Put in a name

  8. And then save it” 

  • “Why can’t I say ‘I want to insert a video’ and let me choose which of my videos I want?”

 

Video Preview Mockup