[Draft] Product Quality Definition
This is a draft document that is under construction
We’ve changed the name of this document from “Product Health” to “Product Quality”, which is something to strive for and always improve, as opposed to health, which denotes something that should be good by default, and need only be maintained.
Purpose Statement
This is a follow-up to the discussion in [Brainstorming] Product Health . During this brainstorming discussion (on 22 November’s Product WG meeting), the implicit consensus was that “health” is a checklist of attributes that a product should have. We’ve changed the name of this concept from “Product Health” to “Product Quality”, which is something to strive for and always improve, as opposed to health, which denotes something that should be good by default, and need only be maintained. This document proposes a full definition for product quality. It seeks to answer the question:
What is the general checklist of attributes a product should have?
This does not include domain-specific attributes, such as items that all MFEs need that no other types of features need.
To be a “supported” feature in the Open edX release, to what extent of quality standards does the feature need to meet? That is, to what degree should a product have each attribute?
Currently the document does not answer the following questions, which will need to be answered once we agree on the checklist and its usage:
What happens when a previously high-quality “supported feature” has a drop in its quality rankings?
Is the list of “supported” features guaranteed release over release, or can a future release drop a previously supported feature?
What leverage does the PWG have to force a feature’s maintainers into implementing/improving high-quality characteristics?
What happens when an “unsupported feature” is consistently meeting high-quality standards and under continuous active development?
Product Quality Checklist
Potential Checklist Attributes
We need to define what each attribute means, and the degree to which we would expect a given feature to achieve it.
Attribute | Audience | Definition | Metrics |
|---|---|---|---|
Accessibility (a11y) | learners | See the project’s accessibility guidelines. |
|
Internationalization (i18n) | translators | All learner-facing features have strings that have been properly internationalized, enabling translators to provide a localized experience |
|
Internationalization (i18n) | translators | All author-facing features have strings that have been properly internationalized, enabling translators to provide a localized experience |
|
Localization (l10n) | learners | The learner-facing parts of the feature has been adapted to any of the 12 (fact check) formally supported Open edX languages. |
|
Localization (l10n) | authors | The author-facing parts of the feature has been adapted to any of the 12 (fact check) formally supported Open edX languages. |
|
Localization (l10n) | operators | Ease by which a site operator can localize their instance to a language that’s not one of the formally supported Open edX languages. |
|
Performance | learners | Learner-facing features operate smoothly: the site does not hang when loading the feature; clicking buttons and other interactive elements react quickly. |
|
Performance | authors | Author-facing features operate smoothly: the site does not hang when loading the feature; clicking buttons and other interactive elements react quickly. Publishing units and courses is fast, regardless of course size. |
|
Performance | operators | The project provides efficient, cost-saving backends and the ability to scale (to some degree as yet undefined) |
|
Performance | software developers | The Open edX development environment is fast and enables quick iterative development |
|
Usability | learners | The feature and its associated workflows are intuitively designed. Learners can quickly ascertain what the feature does, when and why they would use it, and how to use it. |
|
Usability | authors | The feature and its associated workflows are intuitively designed. Authors can quickly ascertain what the feature does, when and why they would use it, and how to use it. The process to configure a new feature in Studio Advanced Settings is easy, seamless and clearly documented. |
|
Usability | software developers | Developing on the Open edX platform is done via a stable interface that does not require huge amounts of head-bashing maintenance, esoteric knowledge, or vast amounts of 1-1 assistance. |
|
Usability | operators | The process to install and configure a new feature is easy, seamless and clearly documented. |
|
Scalability | learners | Interacting with learner features, other learners, and course team members works the same whether the course has a large or small number of units, enrollees, and/or concurrent number of learners. |
|
Scalability | authors | Interacting with author features, other authors, learners, and course team members works the same whether the course has a large or small number of units, enrollees, and/or concurrent number of learners. |
|
Scalability | operators | The feature can be scaled up or down to support the requisite number of concurrent users as required by the Instance’s use case. |
|
Content Modularity | authors | If applicable, the new feature supports the ability to easily move content around a course outline, to reuse content, share, export. |
|
Site Modularity | operators | Replacing or altering site components, such as a page’s header, an MFE, an object storage provider, or new XBlocks, is easy, seamless and well-documented. |
|
Maintainability | Open edX product team | This feature is easy to maintain due to many qualities, such as quality documentation and low feature complexity. This encourages feature longevity and addition of future developers working on the feature as it will be easy to onboard them. Addition of the feature does not negatively impact performance or velocity of Named Releases. |
|
Maintained | Open edX product team | Does this feature have a community of developers surrounding it? If not, it runs the risk of becoming out of date, broken, or with major security holes. |
|
Product Quality exercise
For this exercise, we’ll pick three features and run them through this Product Quality list. We’ll see how they stack up versus where our gut feeling says the feature is. Our hope is that going through this exercise will help us understand how to weigh the various aspects determined above - what falls out that ends up being the most important aspects? What things are we more willing to live with at less than 100%?
For the exercise, see this document: https://docs.google.com/spreadsheets/d/1jcSMeBD5Zl7muJaEXrxOYKhb72zYcR2z4WTJpHbMlt0/edit?usp=sharing