[Brainstorming] Product Health

Inspired by Ghassan's thoughts on product definition , we be spent time during the 22 November PWG meeting brainstorming on the topic of “Product Health”.

Prompt

“Product Health” could include coming up with a concise definition of product health, what might constitute “health”, how “health” relates to “supported/unsupported components”, and whatever else we think of. We’ll do group brainstorming in a shared wiki page, discussing over the call but also everyone will get a chance to get their thoughts out in text.

Ghassan: important to think systematically not just about what’s in the product, but how the product is actually behaving. I want us to have a space to think about what we offer and how it acts.

Brainstorming Notes

• Like the idea of knowing product characteristics and what’s there, that something might be implemented

• (Kelly) Is one goal: we want to evaluate before adding new features to Open edX? ex: we want to add a new MFE - do we use these characteristics to evaluate health?

  • (Felipe) Think it could help with product review, to have a shared definition of “healthy” - as a developer, I know what to strive for before adding something new

  • (Ed) How do we decide if something new is in the core of the project? I think health would be a prerequisite. Health is judged by people using Open edX, what they decide on

    • (Sarina) Yes, Jenna and I are really looking for the Product working group to work out a definition for “Health” and describe how it will be used.

• (Ali & Kelly) Accessibility & usability are two characteristics that stand out as important (meet a specific level of a11y compliance)

• (Ed) non-functional requirements are important, eg performance, scalability.

  • (Ed) Overtime we should have agreed upon standards in terms of how we meet project-wide non-functional requirements.

• (Santiago) Have metrics, data–must be measurable, have tools to collect these metrics. Number of users, initiatives, whatever we want to measure.

• (Shelly) a healthy product should have up to date docs

• (Sarina) How do we align on what things are more or less important to health?

  • What if a feature is super important but not that healthy?

  • What is the long term health?

  • What happens when something is healthy in one release but falls behind in a later release–what do we do?

• (Dave) What is the intersection of “supported” and “healthy”?

  • (Ed) ex: We have a maintained component. What are the standards? Where is there flexibility? Something can be maintained and not healthy

  • (Sarina) Separate concepts in my mind - “supported” defined in OEP-57, is what we’re calling the set of all components on by default in a release that the Open edX project backs (ie through release testing). “unsupported” not guaranteed to be release tested, etc. Question is, what do we do when supported things are unhealthy? (what about unsupported things being consistently healthy?)

    • (Xavier) Rephrasing the definition heard and liked: “supported” would be the formal commitment by people or organizations to keep a component “healthy” – while being “healthy” would be the state of that component at a given time, like a CI run.

    • (Adolfo) Could be a way to measure before a release. Look at supported things, determine health, decide whether it needs remediation, or possibly drop it from being supported in that release?

• (Sarina) Implicit consensus seems to be that health is a checklist of attributes that a product should have.

  • (Ed) One core checklist plus probably other ones depending on type of feature (eg, MFEs may have an additional checklist)

    • Example: There is an MFE checklist already, that we’ve used successfully for Olive fixes

  • (Felipe) Also a function of the health over time–i.e. something that goes back and forth from release to release might be considered “unhealthy” even if it checks off all the boxes this month.

  • (Sarina) What’s the next steps to defining this? Smaller group of people that make a strawman and share?

  • (Ghassan) Keep in mind course author and operator type features may have different definitions of “health”.

  • (Ghassan) is analytics a feature or a characteristic?

    • (Ed) What do you mean by analytics being feature v characteristic? (Sarina) there is a definite feature called “Analytics” which is the ability for instructors to see analytics about how learners are performing in the course. When you say analytics is also a “characteristic”, are you defining “analytics” in a different way (such as usage analytics for operators to use)?

    • (Ghassan follow up), So I think of Analytics as

      • As a feature just like the above paragraph explanation

      • As a characteristic: when it can help us gather data about how a certain feature(s) is being used, adopted…etc. Put it on other way, a feature has analytic characteristic if it can report back how, if its being used, by what percentage…etc. Now the question how this relate to analytics as a feature, how both might overlap, is probably big question. And what I wanted to say in the call, that we might have a challenge here, because getting data usage won’t be easy or straightforward, given we have different providers, and each with their clients, not to mention to compliance with data privacy laws…

• (Ed) Thought experiment. As of pretty recently, healthy for an MFE would include “remote configuration.” Would we have known that far enough in advance to make a difference for Olive? Having that checked, will obviously be beneficial for the future, so it would reduce rework and panic, but would it have helped now?

  • (Adolfo) Example: There is an MFE checklist already, that we’ve used successfully for Olive fixes