This document breaks down deployment of our frontend apps into these main topics:
The systems and machines we use to deliver our micro-frontend code to users.
AWS: S3, Route53
Cloudflare
We use Terraform “Infrastructure as Code” to provision and manage our infrastructure on AWS and Cloudflare. By writing our infrastructure as declarative configuration files we are better able to collaborate, version, and automate infrastructure management.
We write our declarative configuration files (*.tf
files known as Terraform configuration) in our https://github.com/edx/terraform/ repository. Files relevant to frontend infrastructure:
modules/frontend/main.tf is a module that defines a common collection of infrastructure resources and variables for frontends. This module is included in each specific frontend’s terraform configuration. (resources include S3, Route53, CloudFlare, Cloudflare Worker, and CloudFront).
https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/aws.tf defines default information related to AWS as a provider and how we’ll interact with it: roles, regions, etc. A couple acronyms you’ll see:
arn
= AWS Resource Name
https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/cloudflare.tf defines a Cloudflare zone_id
. A zone
is a domain name along with its subdomains and other identities.
https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/frontend.tf defines a user (frontend_deployer_user
) with which infrastructure will be manipulated. ??? is this true ???
https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/terraform.tfvars are common variable definitions. They are automatically included in the environment when a frontend-app-*** is planned. (you can see how this is done in https://github.com/edx/terraform/blob/master/utils/env.sh).
https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/vpcvars.tf (vpc
= Virtual Private Cloud) defines the environment “prod” or “stage”
https://github.com/edx/terraform/blob/0fcc8ab346db9b9c38e66343d638b03e822e7e6f/plans/edx/prod-frontends/frontend-app-learning.tf or any frontend-app-*.tf
config file provides the specific config values for a given microfrontend and leverages the modules/frontend/main.tf module above to define all the resources needed to serve that mfe properly.
plans/edx/stage-frontends/* contains the same set of files and definitions as the corresponding prod-frontends directory.
We use Atlantis to automate the generation of Terraform execution plans from our configuration files in pull requests.
Configuration for Atlantis automation itself lives here: https://github.com/edx/terraform/blob/5d02d5085a32338ed0b63a7ea3e4cc9b62366d48/atlantis.yaml
When pull requests are made in the the edx/terraform repository, Atlantis will automatically run terraform plan
and comment on the PR with the plan output. The plan is applied by commenting atlantis apply
on the PR after a reviewer has approved. If all plans apply successfully Atlantis will automatically merge the PR and Github will delete the branch automatically.
We run continuous deployments of our frontends to the infrastructure we set up using Terraform in the previous section. These continuous deployments are performed using GoCD.
Much of the content below is copied from: https://www.gocd.org/getting-started/part-1/
The Site Reliability Engineering Team maintains a detailed document about GoCD and related technologies here GoCD Deployment Pipeline Tooling |
GoCD Servers and Agents
The Go Agents do the work of building apps or deploying code as configured by the users or administrators of the system.
The Go Server provides work for the agents to do and serves the admin user interface.
A pipeline is a representation of a workflow or a part of a workflow. In our case, we use pipelines to represent how to build and deploy our microfrontend application. A pipeline in GoCD is comprised of stages which are in turn comprised of jobs and tasks.
Material
A material is an object that GoCD will “watch” for changes, triggering a pipeline to run. Typically, a material is a source control repository (like Git, Subversion, etc) and any new commit to the repository is a cause for the pipeline to trigger. A pipeline needs to have at least one material and can have as many materials of different kinds as you want.
Gomatic is a Python API for configuring GoCD. Under the hood “Gomatic uses the same mechanism as editing the config XML through the GoCD web based UI. It gets the current config XML from the GoCD server, re-writes it as a result of the methods called, and then posts the re-written config XML back to the GoCD server.
We leverage Gomatic to configure GoCD in scripts in the https://github.com/edx/edx-gomatic repository.
Key files related to frontend deployments:
https://github.com/edx/edx-gomatic/blob/master/edxpipelines/pipelines/cd_frontends.py A script to install pipelines that can deploy frontend apps. I
https://github.com/edx/edx-gomatic/blob/master/edxpipelines/pipelines/config/frontend_apps.yml List of frontend apps for which we should create deployment pipelines.
https://github.com/edx/edx-gomatic/blob/master/edxpipelines/constants.py Constants used for building GoCD pipelines.
https://github.com/edx/edx-gomatic/blob/master/edxpipelines/materials.py A set of standard overridable material definitions.
https://github.com/edx/edx-gomatic/blob/master/edxpipelines/patterns/frontend_pipelines.py Code for generating pipelines for frontend apps.
https://github.com/edx/edx-gomatic/blob/master/edxpipelines/patterns/frontend_jobs.py Common gomatic Jobs patterns for frontend pipelines.
When https://github.com/edx/edx-gomatic/blob/master/edxpipelines/pipelines/cd_frontends.py is executed, it reads the list of frontend apps in https://github.com/edx/edx-gomatic/blob/master/edxpipelines/pipelines/config/frontend_apps.yml. For each app it runs generate_frontend_deployment_pipelines
contained in https://github.com/edx/edx-gomatic/blob/master/edxpipelines/patterns/frontend_pipelines.py#L15.
... app_material = GitMaterial( app_github_url, # Material name is required to label pipelines with a commit SHA. GitMaterials # return their SHA when referenced by name. material_name=app_name, polling=True, destination_directory=app_name, branch=app_git_branch, ) extra_materials = [ materials.EDX_INTERNAL(), materials.TUBULAR(), ] stage_edp = EDP('stage', 'edx', app_name) _generate_pipeline( stage_edp, app_material, extra_materials, pipeline_group, config[stage_edp], purge_cache, manual_deployment=False, multideployment=multideployment ) ... |
The content of jobs in a frontend pipeline are found in generate_build_frontend
and generate_deploy_frontend
within https://github.com/edx/edx-gomatic/blob/master/edxpipelines/patterns/frontend_jobs.py.
... tasks.generate_package_install(build_job, 'tubular') build_script = 'frontend_multi_build.py' if multideployment else 'frontend_build.py' build_job.add_task(tasks.tubular_task( build_script, [ '--common-config-file', '{}/frontends/common/{}_config.yml'.format( constants.INTERNAL_CONFIGURATION_LOCAL_DIR, edp.environment, ), '--env-config-file', '{}/frontends/{}/{}_config.yml'.format( constants.INTERNAL_CONFIGURATION_LOCAL_DIR, edp.play, edp.environment, ), '--app-name', edp.play, '--version-file', '{}/dist/version.json'.format(edp.play) ], working_dir=None, )) ... |
Note here, we are adding a task to run scripts inside “tubular.” An example command produced by this code may look like:
frontend_build.py --common-config-file edx-internal/frontends/common/prod_config.yml --env-config-file edx-internal/frontends/frontend-app-learning/prod_config.yml --app-name frontend-app-learning --version-file frontend-app-learning/dist/version.json |
For each pipeline we create (currently ‘edx stage’ and ‘edx prod’) we have two stages: build_frontend and deploy_frontend. These pipelines have the following materials:
Frontend application source code
edx-internal (configuration)
tubular (build and deploy scripts)
With these materials present, our build and deploy tasks run frontend_build.py
and frontend_deploy.py
in https://github.com/edx/tubular. With the pipelines set up, what happens when they run?
With deployment pipelines set up and ready in GoCD, let’s take a look at what happens within a deployment pipeline. Broadly the goal for frontend applications is to build the app with the proper environment variables set and then take the dist/
output directory and upload it to the appropriate S3 bucket.
Tubular is an edX name. Do you get the surfing reference? |
https://github.com/edx/tubular contains edX-written scripts that perform work to enable continuous delivery (CD) for https://edx.org. These scripts are called from various tasks/jobs/stages in GoCD pipelines.
Relevant files in tubular for frontend deployments:
https://github.com/edx/tubular/blob/master/tubular/scripts/frontend_build.py Command-line script to build a frontend application.
https://github.com/edx/tubular/blob/master/tubular/scripts/frontend_deploy.py Command-line script to deploy a frontend app to s3.
https://github.com/edx/tubular/blob/master/tubular/scripts/frontend_utils.py Utility file with helper classes for building and deploying frontends. Keeps single and multi deployment scripts DRY.
Reading and combining common and application-level configuration from edx-internal .yml
files
Adding the combined configuration variables to the environment (via command line, e.g. MY_VAR=value npm run build
)
Running npm run build
to build the frontend application
Reading application-level configuration from the appropriate edx-internal .yml
file
Syncing dist files to S3: aws s3 sync' $app_path, $bucket_uri --delete