Technology Radar Vol. 23 (November 2020)

Owned by Jeremy Bowman (Deactivated)
Last updated: Nov 10, 2020
4 min read

As of this writing, this is the current edition available at https://www.thoughtworks.com/radar . The PDF version should remain available even after the next edition is released.

This was discussed at the 2020-11-10 Arch Study Group (slides here). Relevant content from the meeting notes have been copied here for further discussion and future reference.

Techniques

  • Dependency Drift Fitness Function (Adopt) - We’re really close to this with all the package upgrade management we have in place. +2

  • Tailored service templates (Adopt)

  • Security Policy as Code (Adopt): We have it for edx-platform for XSS, but not for much else (we’ve looked at using it w/ tfsec) +1

  • Data mesh (Trial)

  • Diagrams as Code (Trial) - would be nice to see more of this in our docs, potentially easier to keep up to date +3

    • edX has been in the experimental phase

      • We’ve used SequenceDiagrams website and PlantUML

    • Benefits

      • Version control

      • Would it encourage people to create and update diagrams?

      • How might we incorporate into our process?

    • Drawbacks

      • Not as much control on customizing the diagram.

    • Next steps

  • Zero Trust Architecture (Trial) +

  • Parallel run with reconciliation (Trial) - we’ve talked about this before, but a Python tool for it seems to have matured since then: https://github.com/joealcorn/laboratory (inspired by GitHub Scientist, which is for Ruby) +3

  • Distroless Docker Images (Trial): We have a lot of stuff in our docker containers, even more than just the distro. Maybe multistage builds would let us compress our existing (or our newly built) containers to have smaller layer footprints?

  • Kube-managed cloud services (Assess) - I think prefect is one of these

  • Log Aggregation for business analytics (Hold): Are we still building any business analytics off the raw event logs? Can/should we prioritize moving those into dedicated events and/or using DBT to extract consistent data from the tracking logs? +

  • Microfrontend Anarchy (Hold) -  where is edX with this? Seems relevant to us. +1

Platforms

  • JupyterLab (Trial) - heard good things about this but fuzzy on how it differs from Jupyter Notebooks  +

  • Backstage (Assess) - seems potentially useful, this also came up when we were looking at how other places do “devstack” +2

    • Is edX using something like this today?

      • No - we currently have multiple sites (NR, confluence, etc)

    • Benefits

      • Developer onboarding to understanding how we do services

      • Central place to manage ownership of services

      • This would address a weak spot of edX

    • Drawbacks

      • May be hidden from the Open edX community

    • Next steps

  • Pulumi (Assess) - they seemed to like this as a Terraform alternative that addresses some of its drawbacks, is it something worth considering at this point?

  • Tekton (Assess) - Kubernetes-based CI/CD platform, is this useful to us?  Travis and Jenkins aren’t a great fit in this space. (Argo is a pretty good fit though) +1

  • Node overload (Hold) - Node has come up several times recently, and it’s worth noting that it shouldn’t be blindly adopted for the wrong reasons. +1

Tools

  • Airflow (Adopt) - Though we are using Prefect and Argo currently

  • Dependabot (Adopt) - Though we don’t use dependabot, should we be considering different dependency management  (I techniques to better use off-the-shelf tools here? (I thought we did use dependabot…?) +

    • There’s a brief note on Dependabot in https://openedx.atlassian.net/wiki/spaces/AT/pages/1529741317/Handling+Automated+Pull+Requests#Dependabot-PRs . Last we looked, it didn’t do JS upgrade PRs as well as Renovate and was missing some key features for Python upgrade PRs that led us to write our own tooling for that instead.

  • Helm (Adopt) - Specifically Helm 3 +

  • Trivy (Adopt) - this came up recently, seemed pretty easy to turn on (it’s on now) +

  • Kustomize (Trial)

  • Concourse (Trial) - this came up previously as a compelling alternative to Travis and Jenkins for CI/CD, they rate it even stronger now +2

    • edX uses Travis and Jenkins today

    • Next Steps

      • Hold off on doing anything on this for edX

        • Yet another tool for SRE to support to manage - WIP with k8s rollout right now that we probably don’t want to also take on CI/CD changes

        • The CI/CD industry is also in flux right now - may be better to wait

  • ShellCheck (Trial) - for shell script linting, this routinely trips us up for new scripts +1

  • Yarn (Trial)

  • Sentry (Trial) - We don’t use this yet (some engineers have used it at previous organizations) +

  • LGTM (Assess) - looks for coding patterns prone to security problems

Languages & Frameworks

  • single-spa (Trial) - +1

  • Rust (Trial) - we don’t use this yet, but it keeps moving up the scale and has some nice properties +1

  • Redux (Trial) - Our findings seem to mirror Thoughtworks’. +1

  • Recoil (Assess) - We don’t use this yet, but may want to access it too.

  • LitElement (Assess) & web components in general

  • Testing Library (Assess)