Page Comparison

As part of the Django 1.11 upgrade we need to address that they dropped support for the zh-cn language code in Django 1.9 in favor of zh-hans. These are the results of my efforts in trying to figure out where and how we store language codes to make a plan for the update. I was kind of starting from scratch with edx-platform loc so a lot of this is beginner stuff, but may be helpful to other people trying to figure out the whole system end-to-end.

...

English strings are created in a bunch of different places
1. Django templates
2. Mako templates
3. Python code
4. Javascript
5. underscore.js templates
6. Other places?
English strings are gathered
1. paver i18n_extract
  1. shells out "i18ntool_extract"
2. reads config info from conf/locale/config.yaml
3. Excecute platform child extractions
  1. mako
  2. underscore
  3. django-admin makemessages django
  4. django-admin makemessages djangojs
4. Execute 3rd party extractions
  1. wiki
5. All of this generates different .po files for different "segments" to be translated at Transifex by priority
  1. django-partial.po
  2. django-studio.po
  3. mako.po
  4. mako-studio.po
  5. wiki.po
  6. djangojs-partial.po
  7. djangojs-studio.po
  8. underscore.po
  9. underscore-studio.po
Upload English strings to Transifex
1. paver i18n_transifex_push
  1. shells out "i18n_tool transifex push"
    1. shells out "tx push -s"
Strings are translated at Transifex
Update strings
1. Ned manually does the following two steps every two weeks
2. "paver i18n_robot_pull" does a bunch of things
  1. git clean local files
  2. download translated .po files from Transifex
    1. paver i18n_transifex_pull
      1. By default pulls all translated segments of all languages listed in conf/locale/config.yaml with at least 10% reviewed translations
      2. shells out "tx pull -f -mode=reviewed -l {lang}" for each language
      3. Can be used to only pull some
      4. config.yaml stores language codes in standard format (zh_CN)
  3. There are also paver commands to pull and generate ltr and rtl languages separately, it's not immediately clear which is generally used
  4. extract new po files
  (not sure why?)
  1. 1. paver i18n_extract
  2. generate dummy strings
    1. Dummy strings are 2 translations we have for testing. One left-to-right, one right-to-left. More info can be found here.
    2. paver i18n_dummy
    3. runs the static javascript generation command "compilejsi18n"
    4. This generates the compiled dummy languages for testing, with the side effect of compiling all other language files too
  3. compile translated strings
    1. paver i18n_generate_strict
    2. Shells out "i18n_tool generate"
    3. Merge segmented files back together and perform some cleanup
      1. config.yaml tells Transifex how to merge the segments back into two .po files per language with a result like:
        conf/locale/zh_CN/LC_MESSAGES/django.po
        conf/locale/zh_CN/LC_MESSAGES/djangojs.po
    4. django-admin.py compilemessages
      1. Takes the human-readable .po files and compiles them to the tighter .mo files that Django reads in for gettext to use
3. "paver i18n_robot_push"
  1. This is the command that actually performs steps 2 and 3 above
Django reads .mo files for non-js files
1. https://docs.djangoproject.com/en/1.11/topics/i18n/translation/#how-django-discovers-translations
  1. tl;dr: settings.LOCALE_PATHS prioritized in the order they're listed
2. Most importantly it still uses the standard locale names to find files here
We do not use Django's javascript i18n functionality
1. We serve up static pre-localized javascript files generated in "generate dummy strings" above
2. See: https://github.com/zyegfryed/django-statici18n
Use translated strings
1. In a web page
  1. Language selection for a page seems to be handled by the LanguagePreferenceMiddleware (openedx/core/djangoapps/lang_pref/middleware.py)
  2. This uses a combination of cookies, request headers, and user preferences to manipulate Django's accept header parsing into displaying the correct thing
  3. Django uses the re-touched accept header to find the correct localization to load up for gettext to translate
    1. It will spin through the list of accepted languages and try to find one that we support (from settings.LANGUAGES)
    2. If it cannot find an exact match (zh-tw) it will fall back on "sister" languages (zh-hans) or the macrolanguage code (zh)
      1. See https://github.com/django/django/blob/1.11.3/django/utils/translation/trans_real.py#L449
      2. This means we may actually be ok, provided the checking is not strict?
  4. The path for javascript translations is included in the top level page templates for scripts to use
    1. Studio - base.tpl
    2. LMS - main.html
2. As part of a celery task (some celery tasks use translations, if only for error messaging)
  1. bulk_email
    1. Uses the course_language, if set at current there does not see to be any courses using zh_CN
  2. import_olx / export_olx
    1. Store the language from the request that started the task and use that

...

Dark / Dark Database (do we need to do both? what do they each do?)
Settings files (update settings.LANGUAGES)
Transifex (conf/locale)
- Need to update config.yaml to have a lang_map entry to make sure zh_CN gets correctly downloaded as zh_HANS
  - In the [main] section of .tx/config we need this line:
    - lang_map: zh_CN : zh_HANS
  - This is a 1:1 mapping, we can't download the Chinese translations to both zh_CN and zh_HANS. It's unclear if the download process will delete the old CN translation files or if we can just leave them during the transition. If not we may need to add some shim code to copy the files after they are pulled.
What else?

...

Versions Compared

Old Version 13

New Version 14

Key