New video player architecture (React-based Video Player)

Document owners: @Glib Glugovskiy @Mariia Moskalenko

The goal of the document

Clarify and specify requirements and use cases needed to be covered by the new video player. Define approaches to the use cases and make high-level decisions on the solution for them (it can be a high-level solution approach or ADR for some).

Context

Non-functional requirements and use cases were discovered through the document analysis of the Approach Memo & Technical Discovery: React-based Video Player, exploration of the latest Open edX release (Palm) UI and back-end with feature flags for the video player and new studio on the MFE, new studio design, community feedback, project stakeholders' feedback.

Glossary

Open edX

is a service-oriented platform for authoring and delivering online learning at any scale

edX platform

LMS and CMS services of Open edX

Video component

Current vanila JS video player xblock that is used in edx-platform

New video player

ReactJS implementation of the video block based on video.js library

New Video xblock

New xblock with ReactJS frontend and Python backend (that is refactored version of Video Block), it should be backward compatible with Video Component

New video editor

the new design with the new user experience of the live video preview

Quality attributes and non-functional requirements

The structure can be changed during the discovery.

Priorities:

high - requirements are needed for the beta-testing (for 10 courses on edx.org)

medium - requirements are needed for the launch, but not crucial for beta-testing

low / out of scope - requirements can be added later as a plug-in or developed by community members - for now, considered out-of-scope items

Quality attribute/Rules

Non-functional requirement

Priority

Implementation approach

Quality attribute/Rules

Non-functional requirement

Priority

Implementation approach

The player is FOSS and compatible with the Open edX platform’s software license

Compatibility

Video.js video player has to be used as a new player in Open edX as the best option due to the discovery among several options.

Video.js Library was selected as the implementation for video player, it alligned with Open edX platform’s software license.

high

ADR#1 - https://github.com/openedx/edx-platform/pull/33671

The video player is layer upon layer upon layer of crufty compatibility code. We could probably make something a lot simpler overall if we created a new XBlock entirely and migrated video data over to it, though that would make gradual rollout/rollback more difficult

Compatibility

The new video player has to be integrated into Open edX infrastructure to make possible all necessary changes and improvements.

High

ADR#1 - https://github.com/openedx/edx-platform/pull/33671

New videos can just persist that new, sane data model, while old ones get converted as needed.

Compatibility

New video player has to coexist with the Video component.

Some videos can be processed through the Video component and some of them will be processed through the new video player.

High

ADR#1 - https://github.com/openedx/edx-platform/pull/33671

Gradual rollout/rollback

Administrability

The new video player experience has to be implemented in a way that during the upgrade to the new version of the Open edX Platform admin will be able to test the new video player experience with existing videos within the particular organizations and on the particular courses and switch back to the video component in case of unexpected difficulties.

High

ADR#1 -

The video player is layer upon layer upon layer of crufty compatibility code.

Compatibility

The video component has to be refactored and some features have to be deprecated to make further development easy.

High

ADR#1 or #2?

Features of the old video player have to be revised and deprecated as needed

Some pieces of data can just be dropped entirely, like multi-version-YT nonsense used ages ago to provide different video speeds (before YT provided this functionality natively).

Multi-speed YouTube upload option

We used in 2012 before YT started generating multiple speeds for you. Open questions: Is anyone using video bumpers? What features weren’t ported to the MFE, or are no longer exposed in the UI? etc

Maintainability

Features of the old video player have to be revised and deprecated as needed.

high

ADR#1 or #2?

Make it so that the new video player has a simple, sane data model to follow, and write a terrible shim layer to convert (up front and in one place) the layers of old terrible into the format the new system expects.

User experience

The goal is to introduce the new feature without additional effort from the end-users side.

  • Instructors and edX partners will not recreate the course content for the new video player.

  • Instructors and edX partners will not do any clicks to get the new video player experience.

High

ADR#1 -

User experience

The new video player has to work with the old video player data model so that users of the new player will not do any additional migration steps.

High

ADR#2

Most backwards compatibility is focused on not breaking content, even as we discover and implement better ways to do things.

Compatibility

If the user used the content (uploaded it) on the old version of the video player then the user should be able to continue working with the old version of the video player. The new video player will be available for the new content.

All uploaded to the system video files and links have to work in the Video component and in the new video player (not simultaneously).

Course creators and student who are still using the Video component will see their video files and links in video settings and students will be able to watch videos in LMS.

Course creators and student who are using a new video player will be able to see their video files and links in a new video editor and students will be able to watch video in LMS.

High

ADR#1 -

Unified video player experience across the platform (Studio, Content Libraries, LMS)

Consistency

The video player has to be implemented and tested in the:

high

ADR#1 -

Consider the right home for the new video player (e.g., is it a new UI component in the Paragon design system?). If one of the goals of this project is to ensure consistent video player experience across the platform, it may make sense to live in Paragon.

Consistency

A video player component has to be added to the Paragon design system.

Suggest a different approach if it is impossible.

Different approach: The paragon design system will use the video player, video player will not be a Paragon component. Due to requirement for player skin customizations (lower in the list).

high

Should be mentioned in the ADR#1, described in more details in ADR#2

The legacy LMS frontend uses a way out-of-date version of React, so getting a new video player working within the iframe may present its own challenges. However, an upgrade of React in the legacy LMS frontend is on the Roadmap.

 

Compatibility

Video player has to be rendered in legacy LMS frontend or there has to be an option to use the old version of the video player for legacy LMS frontend.

high

ADR#1 -

Ideally, this is a drop-and-replace player within the current XBlock, such that instances upgrading the version of the block would get the player across all courses with no intervention

If this proves to be difficult, what would be the plan for a new XBlock + migration plan to the new block? How much more/less difficult is this?

Seamlessness

Seamless migration to the new version of the video player from the old version.

If there is no possibility for seamless migration - a migration plan has to be created.

High

ADR#2

It needs to be a Javascript-based component

Almost certainly, it needs an HTML5 video component base

Extensibility

The video.js player has a lot of features OOB and can be easely extended with plugins.

We have to introduce extensibility strategy to the community in order to get more useful features from the community.

high

Described in the ADR#2

We have a lot of analytic events that get triggered from the current video player. We MUST keep these requirements as-is, or supply an ADR regarding what would change.

Compatibility

Current events are supported in a new video player.

Add new events to the new video player (enabling / disabling captions, and changing video speed)

high

ADR#2

Features from the current video player have to be recreated with a new video player

Compatibility

Define all front-end, backend features that need to be recreated and tested for the new video player.

high

ADR#2

The player supports different video sources

Flexibility

The list of sources that have to be supported:

  1. edX video ID (uploaded video files using edx-val library)

  2. mp4

  3. webm

  4. mpeg

  5. ogg

  6. HLS

  7. DASH

  8. DASH video stream

  9. Youtube

high

ADR#2

Extensibility

  1. Vimeo

Out of scope

ADR#2

It needs to be a Javascript-based component

Almost certainly, it needs an HTML5 video component base

Extensibility

The video.js player has a lot of features OOB and can be easely extended with plugins.

We have to introduce extensibility strategy to the community in order to get more useful features to the basic Open edX installation.

high

ADR#2

The player supports accessibility features

Accessibility

Existing features:

  • Transcripts for downloading for reading the text instead of listening to the video

  • Transcripts for reading the text while watching the video

  • Captions for reading the text while watching the video

high

ADR#2

Improvements:

  • Keyboard-based controls for operation the video and features of the video

  • Ability to reformat the transcript into full-sentence chunks

  • Change the position and the look of transcripts for reading

  • Audio Description track for giving an overview of what’s happening on the video while watching it

  • Change captions appearance for making them readable (invert colors option for ex.)

  • Positioning of captions on the screen for separating dialog when there are multiple speakers

  • Positioning of captions on the screen to avoid any underlying text

  • Support ASL and other signed languages by use of one of the encodings for signed languages and a remapping of those onto the Languages/Tracks menu

  • Support ASL and other signed languages by addition of ASL Interpretation or Cued Speech transliteration as options in Studio

  • Check transcripts and captions to report possible interpretations, transcription, or translation error

  • Cued Speech Transliteration

  • PPT / DOC / PDF A11y Remediation

  • HTML content review for a11y

out of scope

 

Ensure that the link has an expiration period and connot be used forever (or used on the other platform)

Securability

The player can support the key for the link to the video file created in AWS Cloudfront or Cloudflare

medium

ADR#2

Check existing plugins in video.js marketplace, if not custom development is required.

Ability to track buffering issues that occur during video streaming

Currently, the player lacks reporting on buffering issues, making them impossible to diagnose

Manageability

New video player has to provide analytics and monitoring of buffering issues

out of scope

ADR#2

Buffering issues analytics feature has to be added to the backlog.

Buffering issue - what do we call the issue?

Who and where will see these analytics or/and monitoring reports?

  • Buffering issue is when a loader is displayed in the video

  • Buffering issues analytics
    Is not possible out of the box, the custom development for monitoring can be done.
    ref:

  • Buffering should be handled

  • Buffering analytics feature require

Perform downscaling/play downscaled video to handle both high and low-bandwidth clients

Accessibility

Different video resolutions support

  • for files uploaded to the Open edX instance? or any other source?

  • when the video is uploaded by a content creator it has to be transcoded into different-quality videos?

  • the video has to be downloaded in different resolutions by a content creator?

  • learner can choose the available resolution of the video

Dynamic video adjustment - dynamically adjusts the video quality based on the client's available bandwidth.

  • the player should automatically switch between different video bitrates or resolutions in real-time to ensure smooth playback and reduce buffering

  • when the learner’s network fails or a speed reduces she will see the video resolution suitable for the connection

  • the transition has to be smooth and not noticeable

  • learner should be able to see that the video resolution has changed

  • video has to have downloaded buffers (chapters) for seamless transitions

out of scope

ADR#2

  • Open questions to be solved in technical discovery: Should this be automatic, manual, or both? Should this be dynamic, or rely on multiple hosted file options?

  • We’re open to any set of options depending on outcome of discovery + feasibility.

The player enables video slice previews (e.g., chapters) to external sources, such as displaying video snippets directly in search engine results for enhancing user experience and facilitate content discovery

User experience

The video player have to support video chapters (example) for all video sources:

  1. edX video ID (uploaded video files using edx-val library)

  2. mp4

  3. webm

  4. mpeg

  5. ogg

  6. HLS

  7. DASH

  8. DASH video stream

  9. Youtube

  10. Vimeo

Chapters should be visible on the external sources:

Google

?

 

low

Feature is added to the feature list.

A new feature, the development is required for configuration options in video edit mode CMS.

We may also automatically populate chapters using API.

Demo

Have the player be a standalone, embeddable player for other platforms outside of Open edX/edx.org

Interoperability

The video player has to allow users to embed or share the video to other platforms:

  • create an embedded code

  • share to SM

  • share via email

  • share to other platforms (ex. blogger, skyrock)

low

Feature has to be added to the list.

Course creators should be able to share the video

Course creators should be able to create a sharable video so that learners will be able to share it

Support for additional in-video features, such as comments for better learners' engagement

Control the video player via javascript called elsewhere on the page. It’s how we implement pop-up questions, pop-up links, and measurements of how much of a video someone watched (for loose definitions of “watch”).

Interactivity

The video player can support in-video comments

The video player can support pop-up questions, pop-up links

The video player can support measurements of how much of a video someone watched

Example of inline video comments

Other features or just comments to test the capability?

low

Features added to the feature list.

The video player should support theme / skin customization. Ideally, we’d be able to associate CSS variables from Paragon to the styles for the video player so that things like colors are consistent between the video player and the rest of the application.

Consistency

Automatic customisation for the video player as a part of Open edX installation (use the same colors for the player as for the whole platform).

Elements can be customised:

  • Player controls color, icons and buttons (play, pause, volume, mute, seek bar, fullscreen, and playback speed buttons)

  • Progress bar color (the progress bar that shows the current playback position and buffering status)

  • Subtitle/Caption Display (the style and layout of subtitles or captions that appear on the video)

  • Video quality selection (menu and buttons)

  • Buffering Indicator (icon/anamation)

  • Title and description

  • Control Bar (the design and position)

  • Error messages

  • Watermarks or logos

  • Background and overlay (colors or images that appear behind or on top of the video, interactive overlays)

  • Caption styling (font, size, and color of captions or subtitles)

  • Loading spinner (color)

low

Features added to the list.

Internationalization of videos by providing different language options

Localizability

--

low

Different language options can be added to the video block so that students will be able to choose what language they want to watch.

Support audio player

Extensibility

 

low

Add audio without the video

As a new xblock? Or change the type of the player?

  • support audio formats MP3, WAV, AAC, OGG

  • change the view of the player

  • support audio-specific settings (audio balance, audio quality settings, or support for stereo/mono output)

  • support audio descriptions, captions, and transcript features for audio

Example of the audio xblock https://wordpress.com/support/wordpress-editor/blocks/audio-block/

We will need a way to manage re-runs efficiently. Either preservation of existing video id and caption association or documentation as to how we can pragmatically update a course tarball prior to a re-run.

User experience

Re-runs of the course have to stay as is with the new video player

Out of scope

 

Features and use cases from the current Video component (in progress)

Feature

UC/UCes name

Description

Feature approach (link to ADR)

Feature

UC/UCes name

Description

Feature approach (link to ADR)

FE Video x-block in Studio_add a video component (out of scope, because its a part of Studio MFE)

 

 

 

Other videoin a new tab??

 

 

 

FE Studio_video editor_settings

Video sourse

Superuser/Admin/Course Admin/Course Staff can:

  1. Add a video by a video ID in CMS.

  2. Add a video by a link to youtube.

  3. Add a video by a link ends in .mp4, .mpeg, .webm, or .ogg.

  4. Set video downloads settings.

Vimeo support

.m3u8 playlist support? (from a comment on the memo)

Thumbnail

Thumbnail upload and delete

Transcripts

Authors can upload transcripts

Authors can download transcripts to edit and reupload

Transcripts can be added in other languages, and can be selected by video viewers for display

Duration

Authors can set start and end times for videos

Handout

License

Transcripts aren’t standardized and may be in various locations - YouTube, local storage, or in edx-val - a system that edx.org uses, but unclear if anyone in the community does

What do we need to do with legacy Transcript utilities?

For transcripts, vtt will also serve us well. There are plenty of ways to parse vtt and make it pretty.

Migration to VTT format for transcripts (initial thoughts)

 

 

FE Studio_video editor_video preview

 

A new video player is required to include a video preview on the Video Settings page. The existing video player is (according to previous discovery) a pre-React, pre-webpack player which is not viable for our use case of having a Video Preview capability in the Video Settings Editor.

Rework the Video component so that it directly renders in the Learning MFE with no iframe

The existing video player is (according to previous discovery) a pre-React, pre-webpack player which is not viable for our use case of having a Video Preview capability in the Video Settings Editor.

FE Studio_video editor_captioning?

Closed captioning is visible in the video player's view

Display VTT captions

For subtitles, moving to vtt is probably the right path forward. It's a well documented format.

 

FE Video x-block in Studio_video preview

 

 

Rework the Video component so that it directly renders in the Learning MFE with no iframe 

How will this play with units that have multiple types of components? For example: text followed by a video followed by a multi-choice problem.

At some point, I expect folks will want other types of components to run outside the IFrame as well. Can the Video component be reimplemented in a way that leaves the door open to a more general solution for React-based learning components?

FE Video x-block in Studio_access/duplicate/move/delete (out of scope, because its a part of Studio MFE)

 

 

 

FE LMS_see a Thumbnail

 

 

 

FE LMS_view the video from different sources

 

 

Rework the Video component so that it directly renders in the Learning MFE with no iframe 

How will this play with units that have multiple types of components? For example: text followed by a video followed by a multi-choice problem.

At some point, I expect folks will want other types of components to run outside the IFrame as well. Can the Video component be reimplemented in a way that leaves the door open to a more general solution for React-based learning components?

FE LMS_view the video with a specific duration

 

 

 

FE LMS_download the video

 

 

 

FE LMS_download transcripts

 

 

 

FE LMS_see transcripts near the video

Transcripts can be selected by video viewers for display

Transcript shifts to left side for RTL languages

 

 

FE LMS_download a handout

 

 

 

FE LMS_see the video license

 

 

 

FE Content Library_add a video component (out of scope, because its a part of Studio MFE)

 

 

 

FE Content Library_preview the video

 

 

Rework the Video component so that it directly renders in the Learning MFE with no iframe 

How will this play with units that have multiple types of components? For example: text followed by a video followed by a multi-choice problem.

FE Content Library_video editor_settings

The same use cases as in studio

 

 

FE Content Library_video editor_preview

 

 

Rework the Video component so that it directly renders in the Learning MFE with no iframe

FE_Video controls

replay speeds

?SRT captions

 

 

In_video quizzes

Helper XBlock to locate CAPA problems within videos

 

 

In_video quizzes

Javascript-controlled video

(example from Harvard)

 

 

Architecture notes

 https://openedx.atlassian.net/wiki/spaces/OEPM/pages/3835494415/DRAFT+Architecture+Doc+for+Technical+Discovery+React+Video+Player

WIP Whiteboard with Arch diagrams: https://excalidraw.com/#room=bb5ec8f1a9402a2ef3b5,QXIMmn8cmiQV9WhcTkQc-w