[Draft] Product Quality Definition

This is a draft document that is under construction

We’ve changed the name of this document from “Product Health” to “Product Quality”, which is something to strive for and always improve, as opposed to health, which denotes something that should be good by default, and need only be maintained.

1 Purpose Statement
2 Product Quality Checklist
- 2.1 Potential Checklist Attributes
3 Product Quality exercise

Purpose Statement

This is a follow-up to the discussion in [Brainstorming] Product Health . During this brainstorming discussion (on 22 November’s Product WG meeting), the implicit consensus was that “health” is a checklist of attributes that a product should have. We’ve changed the name of this concept from “Product Health” to “Product Quality”, which is something to strive for and always improve, as opposed to health, which denotes something that should be good by default, and need only be maintained. This document proposes a full definition for product quality. It seeks to answer the question:

What is the general checklist of attributes a product should have?
- This does not include domain-specific attributes, such as items that all MFEs need that no other types of features need.
To be a “supported” feature in the Open edX release, to what extent of quality standards does the feature need to meet? That is, to what degree should a product have each attribute?

Currently the document does not answer the following questions, which will need to be answered once we agree on the checklist and its usage:

What happens when a previously high-quality “supported feature” has a drop in its quality rankings?
- Is the list of “supported” features guaranteed release over release, or can a future release drop a previously supported feature?
- What leverage does the PWG have to force a feature’s maintainers into implementing/improving high-quality characteristics?
What happens when an “unsupported feature” is consistently meeting high-quality standards and under continuous active development?

Product Quality Checklist

Potential Checklist Attributes

We need to define what each attribute means, and the degree to which we would expect a given feature to achieve it.

Attribute	Audience	Definition	Metrics

Attribute	Audience	Definition	Metrics
Accessibility (a11y)	learners	See the project’s accessibility guidelines.
Internationalization (i18n)	translators	All learner-facing features have strings that have been properly internationalized, enabling translators to provide a localized experience	QA/user testing with 100% of scraped strings translated (or using dummy translations)
Internationalization (i18n)	translators	All author-facing features have strings that have been properly internationalized, enabling translators to provide a localized experience	QA/user testing with 100% of scraped strings translated (or using dummy translations)
Localization (l10n)	learners	The learner-facing parts of the feature has been adapted to any of the 12 (fact check) formally supported Open edX languages.
Localization (l10n)	authors	The author-facing parts of the feature has been adapted to any of the 12 (fact check) formally supported Open edX languages.
Localization (l10n)	operators	Ease by which a site operator can localize their instance to a language that’s not one of the formally supported Open edX languages.
Performance	learners	Learner-facing features operate smoothly: the site does not hang when loading the feature; clicking buttons and other interactive elements react quickly.
Performance	authors	Author-facing features operate smoothly: the site does not hang when loading the feature; clicking buttons and other interactive elements react quickly. Publishing units and courses is fast, regardless of course size.
Performance	operators	The project provides efficient, cost-saving backends and the ability to scale (to some degree as yet undefined)
Performance	software developers	The Open edX development environment is fast and enables quick iterative development
Usability	learners	The feature and its associated workflows are intuitively designed. Learners can quickly ascertain what the feature does, when and why they would use it, and how to use it.	wireframe and/or beta tests
Usability	authors	The feature and its associated workflows are intuitively designed. Authors can quickly ascertain what the feature does, when and why they would use it, and how to use it. The process to configure a new feature in Studio Advanced Settings is easy, seamless and clearly documented.	wireframe and/or beta tests Few to no questions in forums Documentation exists for new feature addition User tests/interviews prove ease of use
Usability	software developers	Developing on the Open edX platform is done via a stable interface that does not require huge amounts of head-bashing maintenance, esoteric knowledge, or vast amounts of 1-1 assistance.	Usages are well documented Few to no questions in forums
Usability	operators	The process to install and configure a new feature is easy, seamless and clearly documented.	Few to no questions in forums
Scalability	learners	Interacting with learner features, other learners, and course team members works the same whether the course has a large or small number of units, enrollees, and/or concurrent number of learners.	Baseline metrics stay in place (such as performance metrics, usability metrics) at scale
Scalability	authors	Interacting with author features, other authors, learners, and course team members works the same whether the course has a large or small number of units, enrollees, and/or concurrent number of learners.	Baseline metrics stay in place (such as performance metrics, usability metrics) at scale
Scalability	operators	The feature can be scaled up or down to support the requisite number of concurrent users as required by the Instance’s use case.	Baseline metrics stay in place (such as performance metrics, usability metrics) at scale
Content Modularity	authors	If applicable, the new feature supports the ability to easily move content around a course outline, to reuse content, share, export.	beta tests user tests documentation exists for the feature’s use in both environments
Site Modularity	operators	Replacing or altering site components, such as a page’s header, an MFE, an object storage provider, or new XBlocks, is easy, seamless and well-documented.	If applicable, is themeable Quality, comprehensive documentation exists around installing, enabling/disabling, and modifying the feature
Maintainability	Open edX product team	This feature is easy to maintain due to many qualities, such as quality documentation and low feature complexity. This encourages feature longevity and addition of future developers working on the feature as it will be easy to onboard them. Addition of the feature does not negatively impact performance or velocity of Named Releases.	quality of documentation codebase POLA (needs further definition) feature complexity amount of tech debt
Maintained	Open edX product team	Does this feature have a community of developers surrounding it? If not, it runs the risk of becoming out of date, broken, or with major security holes.	Does this feature have one or more “maintainers” in the Open edX maintainer program? Is there more than one contributor/maintainer? Are PRs reviewed and merged in a reasonable timeframe? What is the likelihood that this feature/project will be around in 5 years?

Product Quality exercise

For this exercise, we’ll pick three features and run them through this Product Quality list. We’ll see how they stack up versus where our gut feeling says the feature is. Our hope is that going through this exercise will help us understand how to weigh the various aspects determined above - what falls out that ends up being the most important aspects? What things are we more willing to live with at less than 100%?

For the exercise, see this document: https://docs.google.com/spreadsheets/d/1jcSMeBD5Zl7muJaEXrxOYKhb72zYcR2z4WTJpHbMlt0/edit?usp=sharing