Versions Compared

Old Version 141

changes.mady.by.user Joe Mulloy (Deactivated)

Saved on

compared with

New Version Current

changes.mady.by.user Brian Mesick

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  •  Manually pause all of the following pipelines to prevent automatically running migrations under Django 1.11
    • STAGE_edxapp_M-D
    • PROD_edx_edxapp_M-D
    • PROD_edge_edxapp_M-D
  •  

    Branch off the Django 1.11 branch, and name it "pwnage101/django-1.10-from-rc".  Add a new commit containing a django 1.10 version override:

    Code Block
    git checkout pwnage101/bump-django-to-1.11
git checkout -b pwnage101/django-1.10-from-rc
echo 'Django==1.10.8' >requirements/edx/django.txt
git add requirements/edx/django.txt
git commit -m 'change django to 1.10 to mitigate 1.11 migration bug' -m 'see https://code.djangoproject.com/ticket/29193'
git push --set-upstream origin pwnage101/django-1.10-from-rc


  •  Use alton to construct 1.10 AMIs based on each new 1.11 AMI:
    •  
      @alton cut ami for stage-edx-edxapp from stage-edx-edxapp with edx_platform_version=pwnage101/django-1.10-from-rc configuration=72b759b73a6e76b723f7dc403c1d0223bd91c1f0 configuration_secure=92a03ff886760c9f11ac3211397eee1463353eff configuration_internal_version=3b91306e02a45df855b2783313d7a21a971d3a49 using ami-43a15f3e
    •  
      @alton cut ami for prod-edx-edxapp from prod-edx-edxapp with edx_platform_version=pwnage101/django-1.10-from-rc configuration=72b759b73a6e76b723f7dc403c1d0223bd91c1f0 configuration_secure=92a03ff886760c9f11ac3211397eee1463353eff configuration_internal_version=3b91306e02a45df855b2783313d7a21a971d3a49  using ami-43a15f3e
    •  
      @alton cut ami for prod-edge-edxapp from prod-edge-edxapp with edx_platform_version=pwnage101/django-1.10-from-rc configuration=72b759b73a6e76b723f7dc403c1d0223bd91c1f0 configuration_secure=fad8d710d6b9bd5c3f8e1d99c59999110d665689 configuration_internal_version=e22d3c5d08a571030923f49cfd0531a67cfda92b using ami-43a15f3e
  •  
    Merge Django 1.11 version bump and migrations to edx-platform release-candidate
    • use existing branch pwnage101/bump-django-to-1.11
  •  Wait for the Django 1.11 AMIs to be built by the pipeline:
  •  For stage-edx-edxapp:
    •  From the EC2 console, launch a new instance "like" an existing Django 1.8 edxapp worker, but with a few tweaks:
      •  make sure to use the same instance type (c4.2xlarge)
      •  change the AMI to the AMI for stage-edx-edxapp Django 1.10.  Refer to the table above
      •  delete the tag key "services" so that the worker app won't actually start
      •  append "-django-1.10" to the the instance identifier
      •  change the security groups to:
        • stage-edx-WorkerServerSecurityGroup-93NZAY5JJ3DM
      •  launch the instance
    •  Login to the new instance lauched in the previous step
      •  create a tmux session
      •  

        obtain the DB migrate username and password from https://github.com/edx-ops/edx-secure/blob/master/ansible/vars/db/stage-edx-edxapp.yml#L19-L21

      •  

        confirm django_migrations state:

        Code Block
        languagetext
        read DB_MIGRATION_PASS
export DB_MIGRATION_PASS
DB_MIGRATION_USER=migrate001 time /edx/bin/edxapp-migrate-cms --list auth


      •  

        Confirm that only 2 migrations have not been run(0007 and 0008)

        Code Block
         [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [ ] 0007_alter_validators_add_error_messages
 [ ] 0008_alter_user_username_max_length


    •  

      Activate this waffle switch (account creations and password changes will be disabled on edx.org) from https://courses.stage.edx.org/admin/waffle/switch/

      Code Block
      languagetext
      user_api.prevent_auth_user_writes


    •  

      Migrate the auth app:

      Code Block
      languagetext
      DB_MIGRATION_USER=migrate001 time /edx/bin/edxapp-migrate-cms auth


    •  Monitor New Relic for slow response times
      •  

        When you notice slower response times stop supervisor on all stage-edx-edxapp machines.

        Code Block
        ansible -i stage-edx-inventory.ini all -m shell -a "/edx/bin/supervisorctl stop edxapp:*"


      •  Wait for the migration to finish in the tmux session.
      •  Re-enable supervisor


        Code Block
        ansible -i stage-edx-inventory.ini all -m shell -a "/edx/bin/supervisorctl start edxapp:*"


    •  

      Deactivate this waffle switch (re-enabling account creations and password changes) from https://courses.stage.edx.org/admin/waffle/switch/

      Code Block
      languagetext
      user_api.prevent_auth_user_writes


    •  

      On the read replica (tools-gp.edx.org), confirm that the auth_user table was successfully altered:

      Code Block
      languagetext
      $ /edx/bin/stage-edx-edxapp-mysql.sh
mysql> SELECT COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS
           WHERE TABLE_SCHEMA = 'wwc' AND TABLE_NAME = 'auth_user' AND COLUMN_NAME = 'username';
+--------------+
| COLUMN_TYPE  |
+--------------+
| varchar(150) |
+--------------+


    •  Unpause the stage_edx_deploy_M-D pipeline.  This will trigger the remaining Django 1.11 migrations and run the deployment on stage.
  •  For prod-edx-edxapp:
    •  From the EC2 console, launch a new instance "like" an existing Django 1.8 edxapp worker, but with a few tweaks:
      •  make sure to use the same instance type (c4.2xlarge)
      •  change the AMI to the AMI for prod-edx-edxapp Django 1.10.  Refer to the table above
      •  delete the tag key "services" so that the worker app won't actually start
      •  append "-django-1.10" to the the instance identifier
      •  change the security groups to:
        • prod-edx-WorkerServerSecurityGroup-1BXCIEREYRREX
      •  launch the instance
    •  Login to the new instance lauched in the previous step
      •  create a tmux session
      •  

        obtain the DB migrate username and password from https://github.com/edx-ops/edx-secure/blob/master/ansible/vars/db/prod-edx-edxapp.yml#L20-L22

      •  

        confirm django_migrations state:

        Code Block
        languagetext
        read DB_MIGRATION_PASS
export DB_MIGRATION_PASS
DB_MIGRATION_USER=migrate001 time /edx/bin/edxapp-migrate-cms --list auth


      •  

        Confirm that only 2 migrations have not been run(0007 and 0008)

        Code Block
         [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [ ] 0007_alter_validators_add_error_messages
 [ ] 0008_alter_user_username_max_length


    •  

      Activate this waffle switch (account creations and password changes will be disabled on edx.org) from https://courses.edx.org/admin/waffle/switch/

      Code Block
      languagetext
      user_api.prevent_auth_user_writes


    •  

      Migrate the auth app:

      Code Block
      languagetext
      DB_MIGRATION_USER=migrate001 time /edx/bin/edxapp-migrate-cms auth


    •  Monitor New Relic for slow response times
      •  

        When you notice slower response times stop supervisor on all prod-edx-edxapp machines to bring up the maintenance page.

        Code Block
        ansible -i prod-edx-inventory.ini all -m shell -a "/edx/bin/supervisorctl stop edxapp:*"


      •  Wait for the migration to finish in the tmux session.
      •  

        Re-enable supervisor

        Code Block
        ansible -i prod-edx-inventory.ini all -m shell -a "/edx/bin/supervisorctl start edxapp:*"


    •  

      On the read replica (tools-gp.edx.org), confirm that the auth_user table was successfully altered:

      Code Block
      languagetext
      $ /edx/bin/prod-edx-edxapp-mysql.sh
mysql> SELECT COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS
           WHERE TABLE_SCHEMA = 'wwc' AND TABLE_NAME = 'auth_user' AND COLUMN_NAME = 'username';
+--------------+
| COLUMN_TYPE  |
+--------------+
| varchar(150) |
+--------------+


    •  

      Deactivate this waffle switch (re-enabling account creations and password changes) from https://courses.edx.org/admin/waffle/switch/

      Code Block
      languagetext
      user_api.prevent_auth_user_writes


    •  

      On the read replica (tools-gp.edx.org), confirm that the auth_user table was successfully altered:

      Code Block
      languagetext
      $ /edx/bin/prod-edx-edxapp-mysql.sh
mysql> SELECT COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS
           WHERE TABLE_SCHEMA = 'wwc' AND TABLE_NAME = 'auth_user' AND COLUMN_NAME = 'username';
+--------------+
| COLUMN_TYPE  |
+--------------+
| varchar(150) |
+--------------+


    •  Unpause the prod_edx_edxapp_deploy_M_D pipeline.  This will trigger the remaining Django 1.11 migrations and run the deployment to prod.
    •  at 12:00 EDT take down the system maintenance banner using the Global Status Message configuration: http://courses.edx.org/admin
  •  For prod-edge-edxapp:
    •  Switch to the edge AWS account
    •  From the EC2 console, launch a new instance "like" an existing Django 1.8 edxapp worker, but with a few tweaks:
      •  make sure to use the same instance type (c4.2xlarge)
      •  change the AMI to the AMI for prod-edx-edxapp Django 1.10.  Refer to the table above
      •  delete the tag key "services" so that the worker app won't actually start
      •  append "-django-1.10" to the the instance identifier
      •  change the security groups to:
        • prod-edge-WorkerServerSecurityGroup-CIXQOM99GRRI
      •  launch the instance
    •  Login to the new instance lauched in the previous step
      •  create a tmux session
      •  

        obtain the DB migrate username and password from https://github.com/edx-ops/edge-secure/blob/master/ansible/vars/db/prod-edge-edxapp.yml#L19-L21

      •  

        confirm django_migrations state:

        Code Block
        languagetext
        read DB_MIGRATION_PASS
export DB_MIGRATION_PASS
DB_MIGRATION_USER=migrate001 time /edx/bin/edxapp-migrate-cms --list auth


      •  

        Confirm that only 2 migrations have not been run(0007 and 0008)

        Code Block
         [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [ ] 0007_alter_validators_add_error_messages
 [ ] 0008_alter_user_username_max_length


    •  

      Activate this waffle switch (account creations and password changes will be disabled on edx.org) from https://edge.edx.org/admin/waffle/switch/

      Code Block
      languagetext
      user_api.prevent_auth_user_writes


    •  

      Migrate the auth app:

      Code Block
      languagetext
      DB_MIGRATION_USER=migrate001 time /edx/bin/edxapp-migrate-cms auth


    •  Monitor New Relic for slow response times
      •  

        When you notice slower response times stop supervisor on all prod-edge-edxapp machines to bring up the maintenance page.

        Code Block
        ansible -i prod-edge-inventory.ini all -m shell -a "/edx/bin/supervisorctl stop edxapp:*"


      •  Wait for the migration to finish in the tmux session.
      •  

        Re-enable supervisor

        Code Block
        ansible -i prod-edge-inventory.ini all -m shell -a "/edx/bin/supervisorctl start edxapp:*"


    •  

      Deactivate this waffle switch (re-enabling account creations and password changes) from https://edge.edx.org/admin/waffle/switch/

      Code Block
      languagetext
      user_api.prevent_auth_user_writes


    •  

      On the read replica (tools-gp.edx.org), confirm that the auth_user table was successfully altered:

      Code Block
      languagetext
      $ /edx/bin/prod-edge-edxapp-mysql.sh
mysql> SELECT COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS
           WHERE TABLE_SCHEMA = 'wwc' AND TABLE_NAME = 'auth_user' AND COLUMN_NAME = 'username';
+--------------+
| COLUMN_TYPE  |
+--------------+
| varchar(150) |
+--------------+


    •  Unpause the prod_edge_edxapp_deploy_M_D pipeline.  This will trigger the remaining Django 1.11 migrations and run the deployment on edge.

...