Developer Documentation and Training Survey Results - 2021-Jan

The following results come from two similar surveys on Developer Documentation and Training sent in January 2021. The survey was sent to all community members. One survey was addressed to engineers employed by edX and Arbisoft, and the other was addressed to remaining community engineers listed here as “Pas edX”. (Pas = “not” in French)

 

Community Respondents

We received 53 responses in total, with more than half from Pas edX.

  • edX (23): 43.4%

  • Arbisoft (2): 3.8%

  • Pas edX (28): 52.8%

Development Focus

The majority of respondents are full-stack engineers.

edX: How would you describe your main development focus with respect to the edX codebase?

Pas edX: How would you describe your main development focus with respect to Open edX?

Development Focus

edX/Arbisoft

Pas edX

Development Focus

edX/Arbisoft

Pas edX

 

Full stack

68% (17)

85.7% (24)

Front end

4% (1)

--

Back end

16% (4)

10.7% (3)

Dev ops

4% (1)

3.6% (1)

Other: Data Engineering

4% (1)

--

Other: Data Modeling

4% (1)

--

edX/Arbisoft: Open Responses

The edX and Arbisoft respondents seem to specialize in certain areas of the platform, though not everyone.

Question: In which areas or on which major features of the edX codebase have you developed? (If this does not apply to you, enter "N/A".)

edx-mktg,course-discovery,edx-platform,ecommerce,frontend-app-payment,optimizely

Video Encode Manager, Discussions(in terms of accessibility), Proctoring

Adding the most recent ones: video encode manager and related code, logistration MFE (including backend changes to edx-platform)

Warehouse-transforms (data warehouse)

Almost all IDAs and many of the MFEs.

Platform, some IDAs

cybersource SDN fallback code, and the upsell experience

edx-proctoring, edx-platform, registrar, various MFEs

registrar, gradebook, ora

edx-platform, ecommerce, some MFEs

edX enterprise repo - integrated_channels mostly

I really don't know what you're expecting to get that would be actionable from this question.

edx-enterprise

MFEs and other frontend libraries.

Open Response Assessment (ORA), Gradebook, edx-platform

grades, masters, enterprise, ORA

edx-platform, ci, configuration, data engineering repos (insights, warehousetransforms)

edx-platform, edx-enterprise, ecommerce

Infrastructure

edx-enterprise, frontends for enterprise, edx-platform

Enterprise, Paragon

courseware, studio, xblock, devstack, master's, micro-frontends, tools

Pas edX: Open Responses

Many Pas edX respondents have experience with XBlocks, inner courseware, and integrations.

Question: Which features have you developed in your work with Open edX code? If this does not apply to you, enter "N/A".

Custom theme, filesystem-based ORA2 uploads, various bugfixes, Waffle documentation, API documentation...

i18n, marketing site, multi-tenancy, edx-ace

new API call (stopped developing a year ago)

Learner reviews and rating, additional search filters, multilanguage support, etc.

https://github.com/edx/edx-platform/pulls?q=is%3Apr+author%3A0x29a+is%3Aclosed

https://github.com/edx/edx-ora2/pulls?q=is%3Apr+is%3Aclosed+author%3A0x29a

https://github.com/edx/frontend-app-learning/pull/95

SGA xBlock, CCX, custom capa problems (sandbox), feature toggles

rapid-response-xblock, some pieces of edx-sga

XBlocks, Blockstore, Bookmarks

I have developed some custom reports

OpenBadges implementation, Libraries V2 authoring frontend, XBlock-poll, several contributions to XBlocks and libraries.

I've worked on patches to edx/configuration, written an XBlock and supported quite a few, worked on bug fixes to the LMS and CMS code, and more recently, I've led the development of the library-authoring MFE and helped optimized the performance of blockstore.

LabXchange

Course Review & Rating, Chat with Instructor, Surveys, Zoom webinars integration with LMS

added reports to analytics, translations to XBlocks, publicly-accessible course content, enhancements to course updates, enhancements to ORA2, configuration/ansible additions, named-release dependency and various bug fixes.

  • Overwriting video transcripts in the edx video abstraction layer

  • Adding customizable tinymce plugins to the edx platform

I have added a way to directly add ORA from Xblock panel

Libraries, Blockstore

ORA reports, simple_theme role, video auto-advance, new feature flags, course metadata, bug fixes, others. All of them as part of a team (OpenCraft)

XBlocks, content libraries, blockstore, SAML SSO

Several bug fixes, Expose banner image url

Experience Level

Codebase Experience

There is an approximately 50-50 split between respondents who have less than 2 years of experience with the codebase (27 total) and respondents with 2+ years of experience with the codebase (26 total).

25% of the Pas edX respondents have more than 5 years of experience with the codebase.

edX: How long have you been working with the edX codebase?
Pas edX: How long have you been working with Open edX code?

Codebase Experience

edX/Arbisoft

Pas edX

Codebase Experience

edX/Arbisoft

Pas edX

 

0-2 years

56% (14)

46.4% (13)

2-5 years

40% (10)

28.6% (8)

5+ years

4% (1)

25% (7)

Career Level

The Pas edX respondents have a significantly greater percentage of seniority (60% vs 40%).

Question: What is your overall career level in your current field?

Career Level

edX/Arbisoft

Pas edX

Career Level

edX/Arbisoft

Pas edX

 

Junior (0-3 years)

16% (4)

21.4% (6)

Mid-level (3-8 years)

44% (11)

17.9% (5)

Senior (8+ years)

40% (10)

60.7% (17)

Most Wanted Documentation

The most desired types of documentation are big-picture architecture (13 total) and specific how-tos (15 total).

Of the 6 open-ended responses, half of them are related to wanting feature-specific how-tos and setup information.

  • edX: If you could choose one type of documentation to add to the edX codebase in the next six months, what would it be?

  • Pas edX: If you could choose one type of documentation to add to the Open edX suite in the next six months, what would it be?

Requested Documentation

edX/Arbisoft

Pas edX

Requested Documentation

edX/Arbisoft

Pas edX

 

Architecture: Big picture overview of the platform architecture and high-level components

40.0% (6)

25% (7)

Dev Setup: Development environment setup documentation for developer role groups (front end, back end, and so on)

12% (3)

17.9% (5)

How-Tos: "How to" documentation on specific development topics

26.7% (4)

39.3% (11)

Briefs: "Quick starts" that allow you to perform a simple development task in a couple of hours

12% (3)

14.3% (4)

Feature Setup: Feature-specific setup

8% (2)

3.6% (1)

Specific wordings under “Other”

Org

Assumed Category

Actual Wording

edX

Feature-specific setup

Documentation on what key features are, what their business rules are, and how they can be easily set up locally.

edX

Feature-specific setup

Feature set-up! The most time is unnecessarily lost when it's time to fix a bug related to a specific feature (timed exams, a particular kind of discount, etc) and you spend the first couple days being able to set it up in your local environment. I'd love documents that give all the steps (including waffle flags that need to be set!) to see a particular feature appear in your local devstack

Pas edX

Feature-specific setup

A technical reference manual of all Open edX operations and settings

Pas edX

How-Tos

Integration with Mobile App

Pas edX

Architecture

I'd like to see this course having more content (ideally everything needed to learn about Open edX architecture): https://courses.edx.org/courses/course-v1:edX+101+arch2018/course/

Pas edX

How-Tos

One specific how to: how to change the theme to customize email templates. This touches on custom dev settings as well as a common -- maybe universal -- use case.

Developer Onboarding Friction

Reading the open-ended responses in this section will give the Open edX Platform maintainers and leaders a humbling experience.

Stumbling Blocks

46 respondents shared the stumbling blocks they ran into.

Surprisingly, there are many entries related to devstack, although this survey is primarily catered for documentation. While some reported devstack issues are related to docs, some are not.

Other topics include configuration complexity, missing and outdated documentation, understanding complex components, and understanding overall architecture.

edX: Which stumbling blocks did you encounter when you started working with the edX codebase? (If this does not apply to you, enter "N/A".)
Pas edX: Which stumbling blocks did you encounter when you started working with Open edX code? (If this does not apply to you, enter "N/A".)

Configuration Complexity

edX

Understanding how configuration worked, and what needed to change in which repo. That's been simplified a little bit, but having a good doc with some examples and pictures could be really helpful.

Data Access

edX

initially, lack of access to production database contents, something I'd come to take for granted having on day 1 from my last job

Devstack

edX

Devstack took me 4 days to set up.

edX

Misunderstanding of devstack setup ("where should I run each command"?), understanding how the components of platform fit together and how external components plug in,

edX

devstack has always been tricky to set up, though it is better lately.

edX

mainly getting devstack up and running, migration state issues, redis dying and not being apparent etc. incrementally updating devstack has been challenging.

Pas edX

devstack breaks occasionally. the code is very big with many systems that has some tight coupling.

Pas edX

  1. How long it takes to download and provision. 2. How confusing it is to get configure the development environment (i.e. which files to change settings in)

Pas edX

Devstack is still hard to get up and running.

Pas edX

Docker is not well supported for Mac (disk handling is slow-- the NFS setup solution in Devstack is brittle)-- ended up switching to Linux eventually.

The workflow for editing frontend assets takes a while to find.

Didn't find out there was a full ReadTheDocs book on the Devstack until months later.

Pas edX

The devstack needs to set good expectations regarding the stack - how heavy it is, how long does setting it up take, how long do tests take and good ways to test only your changes - which require some digging around right now

Pas edX

The devstack required too much RAM (in vagrant. And now in docker, with too many containers). Some features in the documentation apply only to edx.org. Designing themes was slow and had obscure complexities. Some features (Insights, ecommerce, discovery) and 2nd-category andmuch harder to set up and hack

Pas edX

Setting up the dev environment

Devstack / Devstack Feature Setup

edX

devstack and its various CLI commands/layers of commands are very opaque and not well-documented. It's really hard to know what to do at any given time and what the most efficient ways of keeping your environment functioning and up to date are.

Alternately, trying to enable different features and optional plugins/packages in devstack is very difficult and error prone. Setting up exams, XBlocks, courses in particular states, etc., all difficult and finding documentation is challenging.

Devstack / Missing Docs

Pas edX

Getting the development environment setup is a pain, then you don't understand on the surface of what IDA is and in development do you really need to start all of them. There is heavy lack of documentation when a new feature is introduced which should be curbed ASAP.

Devstack Feature Setup

Arbisoft

when working for sustaining and escalations team (edx-spartans), it was fairly difficult to setup devstack for proper workflows to reproduce the bug. Like how to set entitlements locally, how to setup credentials, how to create and enable programs etc

edX

not having the type of documentation I described in the last question (Feature set-up!)

edX

Issues with devstack, setting up the environment in a way that can replicate whatever bug I am trying to solve (for example, with a bundle course). Documentation is not always up to date so you go by trial and error.

edX

Configuration during development is rarely the same as production. Build out working test data sets locally is painful.

Pas edX

Figuring out how to enable a feature or to run a specific operation required (and still does) extensive grepping ("git grep ...") of the source code.

Pas edX

Figuring out how to modify theme and add/modify existing components.

IDA Differences

edX

I stumbled when learning that each service is a little different in it's own way.

Missing Docs

Pas edX

Lack of documentation

Pas edX

No LMS autogenerated documentation easily accessible. Code is massive, so it's easy to lose yourself without real good explanation on several components

Pas edX

could not find any documentation for common errors faced while the development

Pas edX

Lack of context for how pieces of code are used, and what the future plans were. Lack of API documentation.

Missing Docs / Outdated Docs

Pas edX

Incomplete/missing documentation of feature flags and configuration options, out-of-date information on how to set up features like certificates, course search, course modes.

Not Understanding Complex Features

edX

Usually the hardest thing for me is understanding how some complex feature is supposed to work - this is more product documentation than developer documentation. For example, knowing how Feature Based Enrollments is supposed to be behave, and how to set up Feature Based Enrollments locally.

Pas edX

It was not so easy for me to understand how modulestore works.

Pas edX

There is no clear documentation for insight, ecommerce and what little there is is not clear.

Not to mention that the part of forums for notifications, which should be easy is not.

Not Understanding Overall Architecture

Arbisoft

Understanding edX's architecture, keeping notes of its different components(internal tools included), and how they fit in with the development stack.

edX

Lack of knowledge about upstream systems, data integration & data flows

edX

Difficulty of finding an organized documentation ecosystem. Didn't know if there was no documentation OR if I was looking in the wrong places

edX

No clear documentation on the overall design and how the IDAs all connect.

edX

lack of understanding what all the repos are for and how they relate to each other; what xblocks are

edX

Structure of edx-platform. More generally, how our systems "fit together"

Pas edX

In the beginning, the hardest part was figuring out how the several components came together in a scalable fashion.

Not Understanding Overall Architecture / Devstack Feature Setup

edX

It felt like there was always some service I didn't know about that I needed to spin up, or there was a waffle flag that I didn't know about that I needed to turn on, etc, etc, etc. I did not have the necessary knowledge of the system to complete my development tasks.

Not Understanding Overall Architecture / Poor Doc Search

edX

Which repos were responsible for what, how do they/are they supposed to interact with each other, no authoritative set of expectations around code quality/processes ("do what you want", "we prioritize speed over everything else")

Open edX Hosting

Pas edX

Open edX is too big to host locally on a modest computer.
Open edX is tied to third party non-free services for which, IMHO, some free/self hosted alternatives should exist either fully implemented or as stubs.

Pas edX

Install OpenEDX on Cloud server.

Outdated Docs

edX

Lots of out of date doc that should be nuked

Outdated Docs / Poor Doc Search

edX

a lot of documentation is out of date and written by people who don't work here anymore, and a lot of tribal knowledge is kinda just shared within owning teams and not published anywhere (or hard to find), so if you need to jump into a new area it's often hard to figure out what is happening and why

edX

Getting to a point where you can develop is sometimes challenging (i.e. docs out of date, broken commands, have gaps/assumptions about developer knowledge). Sometimes it led to some rabbit holes

Poor Doc Search

edX

Documentation is scattered everywhere. One search is not good enough and I need to search on multiple platforms like Confluence, github, google docs and so on.

Poor Doc Search / Missing Docs

Pas edX

lack of examples, not begin told/not finding development conventions

Resolving Stumbling Blocks

49 respondents shared how they found solutions to their stumbling blocks.

The vast majority of respondents rely on tribal knowledge by asking others, exploring the codebase, trial and error, and after their gaining own experience.

One respondent shared they ultimately gave up on enabling a service.

Question: If you encountered stumbling blocks, were you able to find a solution, and how? (If this does not apply to you, enter "N/A".)

Ask Around

Arbisoft

I would generally ask people in my team if they knew of any documentation regarding the problem, the last resort and mostly the only resort was to look at the code and see what database tables need to be populated and what feature flags need to be updated to make the code running (a very tedious task)

edX

Went dev by dev until I found an answer

edX

Fortunately everyone at edX is friendly and ready to help, so the workaround is playing 20 questions with whatever volunteer you can find

edX

Usually, I can only find solutions by finding the right people to discuss.

edX

Most times I just end up having to ask in slack, or ask a more-senior developer

edX

Only incomplete solutions, and by asking everyone/anyone willing to answer questions (aka tribal knowledge)

edX

Asking for a lot of help in Slack

edX

Usually by bugging the developers responsible for the original features.

edX

I think I bothered devops and got practice until it clicked.

edX

Often times, team members were able to help.

edX

Yes, by conferring with teammates, posting questions in Slack.

edX

I generally required some amount of tribal knowledge and relied on our principal and staff engineers when getting stuck, however I felt like this was a burden to them.

edX

Most of the time? I got some solutions in the #devstack-questions channel, but a lot of knowledge seems like it's just in the heads of engineers who have been here for a while, so I got a lot of help from Alex Dusenbury, for example.

Pas edX

Found answers through Google Groups, looking at code in Github, asking questions on Slack or Discourse. Regular discussions with devops team at edX / Open edX.

Pas edX

Slack or open edx community

Pas edX

Slack

Pas edX

I had to struggle a lot, slack is kind of not responsive when it comes to finding solution. The only place is forum and with forum you have to dig deep to get the answers.

Ask Around / Docs

edX

For devstack: mostly solutions were found after rounds of slack conversations, and doc reading. Docs could potentially add more troubleshooting cases.

Ask Around / Docs

Pas edX

Yes and no. Mostly through discuss.openedx.org and Lawerence's Blog.

Ask Around / Experience

edX

Devstack setup helped by senior dev walkthrough, for understanding broader architecture, have mostly had to go it alone but with specific questions and starting points asked for in slack channels

Pas edX

Slowly learning more and discussing with people at edX

Ask Around / Explore Code

edX

Ask around, try and figure it out based on existing SQL code & models

Pas edX

I debug the code. I search on GitHub, discuss and Slack. Ask for help, but it's hard to find quality timely answers.

Pas edX

yes. Asking people, experimenting

Pas edX

In very few cases I have solved them, but if I have done them it is by modifying one or two lines of the code.

I almost always search the forum or slack, asking if anyone has been through a similar problem.

Pas edX

joined the slack and tried to figure out

Ask Around / Search

Arbisoft

Yes Mostly, by going through Confluence or asking the relevant team. Many a times, there would be more than one document or oudated documents which made getting rid of deterrent somewhat painful. But having a chat with the knowledgeable person helped in removing the blocker.

Ask Around / Trial and Error

edX

slack channels, stoping and starting devstack again, trial and error

edX

Usually by asking questions in Slack and praying someone responds with something useful. Failing that, progressively nuking more and more of the environment until the problem is fixed.

Pas edX

Trial and error, consult with peers, read the code

Asked but not Answered

Pas edX

I cann't find result in 3 days.

Experience

edX

Usually I have the solution because I happen to know it from tribal knowledge, not because it was documented somewhere.

edX

The solution only came a few years later when I had the opportunity to work with other teams and our escalation team. This seems late to me.

Experience / Architecture Demo

edX

architecture demo by Nimisha; working on enough tickets to familiarize with the codebases.

Explore Code

edX

In some cases, it just didn't exist, and I was left to guess. eg. Data model digrams for key application databases.

Pas edX

I still do a lot of grepping, but we are moving towards better docs with toggle and setting documentation, as part of BD-21.

Pas edX

Investigating the code

Pas edX

I read the code. It's the only true source of information!

Pas edX

By identifying an objective, and diving through the code to reach that objective, you can eventually find the solution. It isn't the best method, but it helped me learn about the platform and make necessary changes.

Pas edX

Yes. Using simplified company-set up VMs instead of an always-broken local devstack. Taking notes to complement the missing documentation about obscure variables/settings. Experience. Staying away from theming tasks.

Explore Code / Docs

Pas edX

Yes, mostly by going through old GH issues/forum posts, and reading the code

Found Doc

edX

Yes, I eventually found https://openedx.atlassian.net/wiki/spaces/SRE/pages/26182437

Give Up

Pas edX

I gave up. I got LMS up and running but not studio.

Recorded Workshop / Trial and Error

Pas edX

The only hints on scalability I had to go on were a short talk on the subject by Feanil Patel during Open edX Con 2015, and the sandbox playbook in edx/configuration. I managed to figure things out based on that, but the the way the information is presented can definitely be improved.

Search / Ask Around

edX

Searching confluence, following git-blame and reading related tickets/pages/PRs, asking more senior developers

Still Struggling

Pas edX

Not yet. There are not many tutorials and the community is very small

Trial and Error

Pas edX

We had to use a remote vps with a native install. It takes too much time to push changes, update assets, but it worked for us.

Pas edX

Yes. Switched to Linux to fix disk performance issues. For the rest just leaned on my own debugging capabilities and memory from when I worked on Open edX code the first time years ago and fiddled til I got it right.

Pas edX

I was able to test stuff on my local devstack, which gave me a good idea on what could be failing when I check the logs

Source Data (limited access)

There is limited access to the following source data.