Versions Compared

Old Version 2

changes.mady.by.user Edward Zarecor (Do Not Use)

Saved on

compared with

New Version Current

changes.mady.by.user Feanil Patel

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info

The Hangout agenda is open to any topics related to deploying edX and running it in production.  Add your agenda topics directly to ths document.

Joining the hangout does require an invitation.  If you need one, please send an email to openedx-operations-hangout@edx.org from the account that you would like us to invite.

Agenda:

  • Team update
  • Forums ES update
    • Upgrading from 0.90 to 1.5
    • Will also be providing a run book for migrating the data with minimal impact
      • Includes
        • forums
        • notes
        • team search
        • course search
    • Expected to land in the next few sprints
    • We are using logstash to migrate the data
    • We are actively testing the process documented in the runbook
    • The cut over processes will involve a backward incompatible change.  We are managing this using our red/black deploys
    • The first named release that have the upgraded code is Ginko
    • If you are fast-following master you will need to be prepared for the change.
    • Questions
      • Will you support 5.2?
        No we will only be support 1.5.  There's an open questinon about whether we will force services to use the same version of ES in the future.
  • Potential MySQL performance degradation in Ficus
  • General plans for support Mongo DB 3.4
    • We would love to run it, but it's not the top of our priority as 3.0 is working well
    • The process will require going from 3.0 → 3.2 → 3.4
    • We can do this without doing a pymongo upgrade based on some testing done by the platform team
    • We are currently running our clusters on 14.04
    • 3.0 has not announced an end of life.
    • They are interested in a lot of availability and stability features
  • S3 Block Structure Caching
    • A transformed version of the transformed course structure document
    • Expense parsing and realizing is cached
    • Using django storages so you can cache in s3, swift, filesystem, etc.
    • Again this is a Ginko feature
    • Design goal is optimization for the read case
    • Turning this on will be a waffle switch
  • How can you make OPENEdx recoverable from a disaster
    • Are you in a position to rebuild your servers?
      • Yes could do it in 1 - 2 hours
    • How are you backing up your Mongo data?
      • Currently they are not backing up the data from MongoDB.
      • Options
        • Dump the data and add it to something like S3.  However there are some possible consistency issues.
        • Disk level snapshot, could be EBS, could be LVM.  MongoDB recommend quiescing writes.
        • Use CloudManager and back it up
        • We hope to avoid disaster by crossing AZ, but are not resilient to using an entire region.
    • How are you backing up your MySQL data?
      • Currently they are using RDS and have backup enabled
      • Options
        • Using built in S3 snapshots
        • replication across regions
        • AWS DMS could also be used.
    • How are you backing up your S3 data?
      • Options
        • Cross region data replication in AWS
    • How are you backing up your configuration?
    • How are you baking up your SSL certs?

Notes: