Notifier Context

Intro

The edX notifier is a python django application that interacts with the LMS and the comments service to send digest emails to forum users who have opted in and are following a discussion. Currently the notifier runs on a single server. It is not idempotent and does not currently have a method for distributing jobs that allows redundancy, although this is planned. Building a new server from scratch takes about 15 minutes with ansible.

Provisioning

Since the notifier runs on a single machine and can't be turned into an AMI, the easiest way to provision and update the notifier is by running ansible directly against the machine. A sample of how to do that can be found below.

ansible-playbook -i ec2.py -l tag_Name_loadtest-edx-notifier notifier.yml -e@/home/feanil/src/edx-internal/ansible/vars/edx.yml -e@/home/feanil/src/edx-internal/ansible/vars/loadtest-edx.yml -e@/home/feanil/src/edx-secure/ansible/vars/edx.yml -e@/home/feanil/src/edx-secure/ansible/vars/loadtest-edx.yml -D

Testing

Make sure the user you are testing with has opted into receiving digests: Instructions

To generate a test digest for yourself you can do something like this (Make sure to update the date to be in the future.):

$ ssh notifier-box.edx.org
$ source /edx/app/notifier/notifier_env
$ sudo -E /edx/app/notifier/virtualenvs/notifier/bin/python /edx/app/notifier/src/manage.py forums_digest --users=5182021 --to_datetime=2015-09-03 --show-html

To manually test on Stage:

Follow instructions above to receive digests
Make a change in Discussions that would be sent through the digest
Verify the following:

- make sure you retrieve an email (i.e. checks scheduler is finding users and finding updates for digests)
- make sure the html output looks good
- make sure the text output looks good

Processes

There are two key components currently.

The Digest Creator

The digest creator which runs from the notifier user's crontab. Any admin on the box can su to the notifier role. As that user, your environment will contain all the environment specific variable required to configure the application. This job runs once a day at 00:00UTC and produces 24hrs, 1440 minuts, worth of digests.

The Celery Workers

These grab messages from a rabbit queue and send email. This process is controlled by supervisord. Run supervisorctl as root to check on the process and restart it. Note: The current prod version of the supervisor job has a but which causes it to not stop cleanly. This is fixed on stage and will be release on Monday. In the meantime you will need to run "stop <job-name>" from the supervisorctl shell and then exit and run killall /opt/wwc/notifier/virtualenvs/notifier/bin/python as root to shutdown the subprocesses.

Known issues

Logging to rsyslog seems to have stopped working. I'm still debugging the root cause. However, useful log information for the celerly workers is being redirect to /tmp by supervisor. Logging from the cron job is going to /var/spool/mail/notifier.