Understanding frontend app deployments

This document breaks down deployment of our frontend apps into these main topics:

1 Infrastructure
- 1.1 What do we use at edX?
  - 1.1.1 Source https://www.cloudflare.com/multi-cloud/aws/
- 1.2 How does all this good stuff get set up?
  - 1.2.1 Terraform
  - 1.2.2 Atlantis
2 Deployment Pipelines
- 2.1 Key concepts for GoCD deployments
  - 2.1.1 Pipelines
    - 2.1.1.1 Source: https://www.gocd.org/getting-started/part-1/#concept2
- 2.2 How do we set all this up?
  - 2.2.1 edX-created python scripts using Gomatic
- 2.3 Our pipelines in summary
3 Deployments in action
- 3.1 Tubular: scripts executed in the pipe[line]
  - 3.1.1 frontend_build.py is responsible for:
  - 3.1.2 frontend_deploy.py is responsible for:

Infrastructure

The systems and machines we use to deliver our micro-frontend code to users.

What do we use at edX?

AWS: S3, Route53
Cloudflare

Source https://www.cloudflare.com/multi-cloud/aws/

How does all this good stuff get set up?

Terraform

We use Terraform “Infrastructure as Code” to provision and manage our infrastructure on AWS and Cloudflare. By writing our infrastructure as declarative configuration files we are better able to collaborate, version, and automate infrastructure management.

We write our declarative configuration files (*.tf files known as Terraform configuration) in our https://github.com/edx/terraform/ repository. Files relevant to frontend infrastructure:

modules/frontend/main.tf is a module that defines a common collection of infrastructure resources and variables for frontends. This module is included in each specific frontend’s terraform configuration. (resources include S3, Route53, CloudFlare, Cloudflare Worker, and CloudFront).
plans/edx/prod-frontends/*
- https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/aws.tf defines default information related to AWS as a provider and how we’ll interact with it: roles, regions, etc. A couple acronyms you’ll see:
  - arn = AWS Resource Name
  - iam = Identity and Access Management
- https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/cloudflare.tf defines a Cloudflare zone_id. A zone is a domain name along with its subdomains and other identities.
- https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/frontend.tf defines a user (frontend_deployer_user) with which infrastructure will be manipulated. ??? is this true ???
- are common variable definitions. They are automatically included in the environment when a frontend-app-*** is planned. (you can see how this is done in ).
- (vpc = Virtual Private Cloud) defines the environment “prod” or “stage”
- or any frontend-app-*.tf config file provides the specific config values for a given microfrontend and leverages the modules/frontend/main.tf module above to define all the resources needed to serve that mfe properly.
plans/edx/stage-frontends/* contains the same set of files and definitions as the corresponding prod-frontends directory.

Atlantis

We use Atlantis to automate the generation of Terraform execution plans from our configuration files in pull requests.

Configuration for Atlantis automation itself lives here:

When pull requests are made in the the edx/terraform repository, Atlantis will automatically run terraform plan and comment on the PR with the plan output. The plan is applied by commenting atlantis apply on the PR after a reviewer has approved. If all plans apply successfully Atlantis will automatically merge the PR and Github will delete the branch automatically.

Deployment Pipelines

We run continuous deployments of our frontends to the infrastructure we set up using Terraform in the previous section. These continuous deployments are performed using GoCD.

Key concepts for GoCD deployments

Much of the content below is copied from: https://www.gocd.org/getting-started/part-1/

The Site Reliability Engineering Team maintains a detailed document about GoCD and related technologies here

GoCD Servers and Agents

The Go Agents do the work of building apps or deploying code as configured by the users or administrators of the system.
The Go Server provides work for the agents to do and serves the admin user interface.

Pipelines

A pipeline is a representation of a workflow or a part of a workflow. In our case, we use pipelines to represent how to build and deploy our microfrontend application. A pipeline in GoCD is comprised of stages which are in turn comprised of jobs and tasks.

Material

A material is an object that GoCD will “watch” for changes, triggering a pipeline to run. Typically, a material is a source control repository (like Git, Subversion, etc) and any new commit to the repository is a cause for the pipeline to trigger. A pipeline needs to have at least one material and can have as many materials of different kinds as you want.

Source: https://www.gocd.org/getting-started/part-1/#concept2

How do we set all this up?

edX-created python scripts using Gomatic

Gomatic is a Python API for configuring GoCD. Under the hood “Gomatic uses the same mechanism as editing the config XML through the GoCD web based UI. It gets the current config XML from the GoCD server, re-writes it as a result of the methods called, and then posts the re-written config XML back to the GoCD server.

We leverage Gomatic to configure GoCD in scripts in the repository.

Key files related to frontend deployments:

A script to install pipelines that can deploy frontend apps. I
List of frontend apps for which we should create deployment pipelines.
Constants used for building GoCD pipelines.
A set of standard overridable material definitions.
Code for generating pipelines for frontend apps.
- Common gomatic Jobs patterns for frontend pipelines.

When is executed, it reads the list of frontend apps in . For each app it runs generate_frontend_deployment_pipelines contained in .

...
    app_material = GitMaterial(
        app_github_url,
        # Material name is required to label pipelines with a commit SHA. GitMaterials
        # return their SHA when referenced by name.
        material_name=app_name,
        polling=True,
        destination_directory=app_name,
        branch=app_git_branch,
    )
    extra_materials = [
        materials.EDX_INTERNAL(),
        materials.TUBULAR(),
    ]

    stage_edp = EDP('stage', 'edx', app_name)
    _generate_pipeline(
        stage_edp,
        app_material,
        extra_materials,
        pipeline_group,
        config[stage_edp],
        purge_cache,
        manual_deployment=False,
        multideployment=multideployment
    )
...

The content of jobs in a frontend pipeline are found in generate_build_frontend and generate_deploy_frontend within .

    ...
    tasks.generate_package_install(build_job, 'tubular')
    build_script = 'frontend_multi_build.py' if multideployment else 'frontend_build.py'
        build_job.add_task(tasks.tubular_task(
        build_script,
        [
            '--common-config-file',
            '{}/frontends/common/{}_config.yml'.format(
                constants.INTERNAL_CONFIGURATION_LOCAL_DIR,
                edp.environment,
            ),
            '--env-config-file',
            '{}/frontends/{}/{}_config.yml'.format(
                constants.INTERNAL_CONFIGURATION_LOCAL_DIR,
                edp.play,
                edp.environment,
            ),
            '--app-name',
            edp.play,
            '--version-file',
            '{}/dist/version.json'.format(edp.play)
        ],
        working_dir=None,
    ))
    ...

Note here, we are adding a task to run scripts inside “tubular.” An example command produced by this code may look like:

frontend_build.py
  --common-config-file edx-internal/frontends/common/prod_config.yml
  --env-config-file edx-internal/frontends/frontend-app-learning/prod_config.yml
  --app-name frontend-app-learning
  --version-file frontend-app-learning/dist/version.json

Our pipelines in summary

For each pipeline we create (currently ‘edx stage’ and ‘edx prod’) we have two stages: build_frontend and deploy_frontend. These pipelines have the following materials:

Frontend application source code
edx-internal (configuration)
tubular (build and deploy scripts)

With these materials present, our build and deploy tasks run frontend_build.py and frontend_deploy.py in . With the pipelines set up, what happens when they run?

Deployments in action

With deployment pipelines set up and ready in GoCD, let’s take a look at what happens within a deployment pipeline. Broadly the goal for frontend applications is to build the app with the proper environment variables set and then take the dist/ output directory and upload it to the appropriate S3 bucket.

Tubular: scripts executed in the pipe[line]

Tubular is an edX name. Do you get the surfing reference?

contains edX-written scripts that perform work to enable continuous delivery (CD) for https://edx.org. These scripts are called from various tasks/jobs/stages in GoCD pipelines.

Relevant files in tubular for frontend deployments:

Command-line script to build a frontend application.
Command-line script to deploy a frontend app to s3.
Utility file with helper classes for building and deploying frontends. Keeps single and multi deployment scripts DRY.

frontend_build.py is responsible for:

Reading and combining common and application-level configuration from edx-internal .yml files
Adding the combined configuration variables to the environment (via command line, e.g. MY_VAR=value npm run build)
Running npm run build to build the frontend application

frontend_deploy.py is responsible for:

Reading application-level configuration from the appropriate edx-internal .yml file
Syncing dist files to S3: aws s3 sync' $app_path, $bucket_uri --delete