edx-platform/docs/static_assets.rst

#######################################
edx-platform Static Asset Pipeline Plan
#######################################

Static asset handling in edx-platform has evolved in a messy way over the years.
This has led to a lot of complexity and inconsistencies. This is a proposal for
how we can move forward to a simpler system and more modern toolchain. Note that
this is not a detailed guide for how to write React or Bootstrap code. This is
instead going to talk about conventions for how we arrange, extract, and compile
static assets.

Big Open Questions (TODO)
*************************

This document is a work in progress, as the design for some of this is still in
flux, particularly around extensibility.

* Pluggable third party apps and Webpack packaging.
* Keep the Django i18n mechanism?
* Stance on HTTP/2 and bundling granularity.
* Optimizing theme assets.
* Tests

Requirements
************

Any proposed solution must support:

* Externally developed and installed Django apps.
* Theming.
* XBlock assets.
* Existing tests.
* Fast builds.
* An incremental implementation path.
* Other kinds of pluggability???

Assumptions
***********

Some assumptions/opinions that this proposal is based on:

* We want to shift as much as possible to Webpack and the JavaScript stack of
  technologies, leaving the Python layer as thin as possible.
* While we will try to make theming upgrades straightforward, we will be moving
  around where files are located and where they're compiled out to.
* We will be pushing towards a world that is more Django app-centric than LMS
  vs. Studio centric, to reduce duplication.
* At the same time, we want to consolidate assets far more efficiently than we
  are doing today.
* Leaning towards more static front ends + API calls.
* However we still need to be compatible with Django's asset system for things
  like third party apps (e.g. Django Rest Framework browsing assets, Swagger,
  etc.)
* It should be possible to pre-build static assets and deploy them onto S3 or
  similar.

Where We Are Today
******************

We have a static asset pipeline that is mostly driven by Django's built-in
staticfiles finders and the collectstatic process. We use the popular
``django-pipeline`` library, with UglifyJS as the JavaScript compressor (the
binary is installed via node into node_modules). We also use the less well known
``django-pipeline-forgiving`` extension to ``django-pipeline`` so we don't error
out when files are missing (added when we started dynamically scanning XBlocks
for assets).

The ``django-pipeline`` config is aware of CSS files for the purposes of
concatenation, but it does *not* know about the source Sass files (or handful of
remaining CoffeeScript files). Those are processed with paver tasks before
``django-pipeline`` ever sees them.

We also have the following custom extensions to Django's builtin ``STATICFILES``
mechanism:

``openedx.core.djangoapps.theming.finders.ThemeFilesFinder``
  Custom finder that overrides any static asset with a version from the themes
  directory (``COMPREHENSIVE_THEME_DIRS`` defined in ``lms.envs.json`` and
  ``cms.envs.json``).

``openedx.core.lib.xblock_pipeline.finder.XBlockPipelineFinder``
  Custom finder that accesses and extracts assets from pip-installed XBlocks via
  ``pkg_resources``.

``openedx.core.storage.DevelopmentStorage/ProductionStorage``
  Custom ``FileStorage`` classes that mostly exist for theme-awareness.

LMS and Studio/CMS Separation
-----------------------------

LMS and Studio have their own directories for source assets (``lms/static`` and
``cms/static``), and have symlinks to shared assets in ``common/static``. We
treat the static asset compilation and collection phase for LMS and Studio as
separate projects that happen to share a lot of pieces. They output to different
places (typically ``/edx/var/edxapp/staticfiles`` for LMS and
``/edx/var/edxapp/staticfiles/studio`` for Studio) and can be collected
separately. However in practice they're always run together because we deploy
them from the same commits and to the same servers.
 
Django vs. Webpack Conventions
******************************

The Django convention for having an app with bundled assets is to namespace them
locally with the app name so that they get their own directories when they are
gathered together into a common static directory by collectstatic. For example,
the edx-enterprise app has a ``static/enterprise`` folder, so its assets are
compiled to ``/edx/var/edxapp/staticfiles/enterprise`` by edx-platform and will
not conflict with assets from any other Django app.

Webpack conventions would have us create a single set of configuration files at
the root of edx-platform, which would specify all bundles in the project.

TODO: The big, "pluggable Webpack components" question.

Proposed Repo Structure
***********************

All assets that are in common spaces like ``common/static``, ``lms/static``,
and ``cms/static`` would be moved to be under the Django apps that they are a
part of and follow the Django naming convention (e.g.
``openedx/features/course_bookmarks/static/course_bookmarks``). An app's
``templates/{appname}`` directory will only be for server side templates, and
any client-side templates will be put in ``static/{appname}/templates``.

Proposed Compiled Structure
***************************

This is meant to be a sample of the different types of things we'd have, not a
full list:

::

  # Webpack bundles/post-processed assets
  /webpack/css
          /fonts
          /js
          /vendor ?

  # Django apps that are in the edx-platform repo
  /course_bookmarks
  /course_experience

  # edX authored, installed via separate repo
  /enterprise

  # Entirely third party apps that we need to maintain compatiblity with.
  /admin
  /rest_framework

  # Themes are part of the "theming" app
  /theming/themes/open-edx
                 /red-theme
                 /edx.org
  
  # XBlocks still collect their assets into a common space (/xmodule goes away)
  # We consider this to be the XBlock Runtime's app, and it collects static
  # assets from installed XBlocks.
  /xblock

Django vs. Webpack Roles
************************

Rule of thumb: Django/Python still serves static assets, Webpack processes and
optimizes them.

Webpack would be responsible for all Sass compilation in edx-platform. It would
also be responsible for the optimization/minification of JavaScript assets, but
those optimized assets would only appear under the ``/webpack`` directory. Third
party assets that Webpack is not aware of may have hash suffixes applied to them
by the Django collectstatic layer, but will not otherwise be processed or
optimized in any way -- so no coffeescript/sass compilation, no uglifyjs
minification, etc.

The django-pipeline dependency should be removed altogether.

Themes
------

Theme handling is muddled. The fact that themes can override server-side
templates means that Python has to be aware of them. At the same time, we want
to shift over Sass compilation as a whole to Webpack, meaning that at least some
knowledge about where they are and how to compile them has to exist there. Also,
there are JS assets in some themes that provide additional functionality, and it
would be a performance degradation if those assets were no longer optimized.

What I do NOT want to happen:

* Significant end user performance degradation.
* Having an *additional* system in the asset pipeline (e.g. keeping
  django-pipeline around while having additional systems).

I think that means that conceptually, there exists a larger Static Asset system
that exists and that we think of both Webpack and Django being consumers of its
configuration. This is also very fuzzy at the moment.

Asset Groups
------------

There will be logical groupings of static assets. There should be uniformity and
no duplication within a group, but we would allow duplication between groups to
better facilitate independent deployment and isolation.

Example Groups:

* XBlock/XModule Assets
* LMS/Studio apps in edx-platform
* Third party app, such as edx-enterprise