WIP stab at the plan for static assets in edx-platform
This commit is contained in:
David Ormsbee
207
docs/static_assets.rst
Normal file
207
docs/static_assets.rst
Normal file
@@ -0,0 +1,207 @@
|
||||
#######################################
|
||||
edx-platform Static Asset Pipeline Plan
|
||||
#######################################
|
||||
|
||||
Static asset handling in edx-platform has evolved in a messy way over the years.
|
||||
This has led to a lot of complexity and inconsistencies. This is a proposal for
|
||||
how we can move forward to a simpler system and more modern toolchain. Note that
|
||||
this is not a detailed guide for how to write React or Bootstrap code. This is
|
||||
instead going to talk about conventions for how we arrange, extract, and compile
|
||||
static assets.
|
||||
|
||||
Big Open Questions (TODO)
|
||||
*************************
|
||||
|
||||
This document is a work in progress, as the design for some of this is still in
|
||||
flux, particularly around extensibility.
|
||||
|
||||
* Pluggable third party apps and Webpack packaging.
|
||||
* Keep the Django i18n mechanism?
|
||||
* Stance on HTTP/2 and bundling granularity.
|
||||
* Optimizing theme assets.
|
||||
* Tests
|
||||
|
||||
Requirements
|
||||
************
|
||||
|
||||
Any proposed solution must support:
|
||||
|
||||
* Externally developed and installed Django apps.
|
||||
* Theming.
|
||||
* XBlock assets.
|
||||
* Existing tests.
|
||||
* Fast builds.
|
||||
* An incremental implementation path.
|
||||
* Other kinds of pluggability???
|
||||
|
||||
Assumptions
|
||||
***********
|
||||
|
||||
Some assumptions/opinions that this proposal is based on:
|
||||
|
||||
* We want to shift as much as possible to Webpack and the JavaScript stack of
|
||||
technologies, leaving the Python layer as thin as possible.
|
||||
* While we will try to make theming upgrades straightforward, we will be moving
|
||||
around where files are located and where they're compiled out to.
|
||||
* We will be pushing towards a world that is more Django app-centric than LMS
|
||||
vs. Studio centric, to reduce duplication.
|
||||
* At the same time, we want to consolidate assets far more efficiently than we
|
||||
are doing today.
|
||||
* Leaning towards more static front ends + API calls.
|
||||
* However we still need to be compatible with Django's asset system for things
|
||||
like third party apps (e.g. Django Rest Framework browsing assets, Swagger,
|
||||
etc.)
|
||||
* It should be possible to pre-build static assets and deploy them onto S3 or
|
||||
similar.
|
||||
|
||||
Where We Are Today
|
||||
******************
|
||||
|
||||
We have a static asset pipeline that is mostly driven by Django's built-in
|
||||
staticfiles finders and the collectstatic process. We use the popular
|
||||
``django-pipeline`` library, with UglifyJS as the JavaScript compressor (the
|
||||
binary is installed via node into node_modules). We also use the less well known
|
||||
``django-pipeline-forgiving`` extension to ``django-pipeline`` so we don't error
|
||||
out when files are missing (added when we started dynamically scanning XBlocks
|
||||
for assets).
|
||||
|
||||
The ``django-pipeline`` config is aware of CSS files for the purposes of
|
||||
concatenation, but it does *not* know about the source Sass files (or handful of
|
||||
remaining CoffeeScript files). Those are processed with paver tasks before
|
||||
``django-pipeline`` ever sees them.
|
||||
|
||||
We also have the following custom extensions to Django's builtin ``STATICFILES``
|
||||
mechanism:
|
||||
|
||||
``openedx.core.djangoapps.theming.finders.ThemeFilesFinder``
|
||||
Custom finder that overrides any static asset with a version from the themes
|
||||
directory (``COMPREHENSIVE_THEME_DIRS`` defined in ``lms.envs.json`` and
|
||||
``cms.envs.json``).
|
||||
|
||||
``openedx.core.lib.xblock_pipeline.finder.XBlockPipelineFinder``
|
||||
Custom finder that accesses and extracts assets from pip-installed XBlocks via
|
||||
``pkg_resources``.
|
||||
|
||||
``openedx.core.storage.DevelopmentStorage/ProductionStorage``
|
||||
Custom ``FileStorage`` classes that mostly exist for theme-awareness.
|
||||
|
||||
LMS and Studio/CMS Separation
|
||||
-----------------------------
|
||||
|
||||
LMS and Studio have their own directories for source assets (``lms/static`` and
|
||||
``cms/static``), and have symlinks to shared assets in ``common/static``. We
|
||||
treat the static asset compilation and collection phase for LMS and Studio as
|
||||
separate projects that happen to share a lot of pieces. They output to different
|
||||
places (typically ``/edx/var/edxapp/staticfiles`` for LMS and
|
||||
``/edx/var/edxapp/staticfiles/studio`` for Studio) and can be collected
|
||||
separately. However in practice they're always run together because we deploy
|
||||
them from the same commits and to the same servers.
|
||||
|
||||
Django vs. Webpack Conventions
|
||||
******************************
|
||||
|
||||
The Django convention for having an app with bundled assets is to namespace them
|
||||
locally with the app name so that they get their own directories when they are
|
||||
gathered together into a common static directory by collectstatic. For example,
|
||||
the edx-enterprise app has a ``static/enterprise`` folder, so its assets are
|
||||
compiled to ``/edx/var/edxapp/staticfiles/enterprise`` by edx-platform and will
|
||||
not conflict with assets from any other Django app.
|
||||
|
||||
Webpack conventions would have us create a single set of configuration files at
|
||||
the root of edx-platform, which would specify all bundles in the project.
|
||||
|
||||
TODO: The big, "pluggable Webpack components" question.
|
||||
|
||||
Proposed Repo Structure
|
||||
***********************
|
||||
|
||||
All assets that are in common spaces like ``common/static``, ``lms/static``,
|
||||
and ``cms/static`` would be moved to be under the Django apps that they are a
|
||||
part of and follow the Django naming convention (e.g.
|
||||
``openedx/features/course_bookmarks/static/course_bookmarks``). An app's
|
||||
``templates/{appname}`` directory will only be for server side templates, and
|
||||
any client-side templates will be put in ``static/{appname}/templates``.
|
||||
|
||||
Proposed Compiled Structure
|
||||
***************************
|
||||
|
||||
This is meant to be a sample of the different types of things we'd have, not a
|
||||
full list:
|
||||
|
||||
::
|
||||
|
||||
# Webpack bundles/post-processed assets
|
||||
/webpack/css
|
||||
/fonts
|
||||
/js
|
||||
/vendor ?
|
||||
|
||||
# Django apps that are in the edx-platform repo
|
||||
/course_bookmarks
|
||||
/course_experience
|
||||
|
||||
# edX authored, installed via separate repo
|
||||
/enterprise
|
||||
|
||||
# Entirely third party apps that we need to maintain compatiblity with.
|
||||
/admin
|
||||
/rest_framework
|
||||
|
||||
# Themes are part of the "theming" app
|
||||
/theming/themes/open-edx
|
||||
/red-theme
|
||||
/edx.org
|
||||
|
||||
# XBlocks still collect their assets into a common space (/xmodule goes away)
|
||||
# We consider this to be the XBlock Runtime's app, and it collects static
|
||||
# assets from installed XBlocks.
|
||||
/xblock
|
||||
|
||||
Django vs. Webpack Roles
|
||||
************************
|
||||
|
||||
Rule of thumb: Django/Python still serves static assets, Webpack processes and
|
||||
optimizes them.
|
||||
|
||||
Webpack would be responsible for all Sass compilation in edx-platform. It would
|
||||
also be responsible for the optimization/minification of JavaScript assets, but
|
||||
those optimized assets would only appear under the ``/webpack`` directory. Third
|
||||
party assets that Webpack is not aware of may have hash suffixes applied to them
|
||||
by the Django collectstatic layer, but will not otherwise be processed or
|
||||
optimized in any way -- so no coffeescript/sass compilation, no uglifyjs
|
||||
minification, etc.
|
||||
|
||||
The django-pipeline dependency should be removed altogether.
|
||||
|
||||
Themes
|
||||
------
|
||||
|
||||
Theme handling is muddled. The fact that themes can override server-side
|
||||
templates means that Python has to be aware of them. At the same time, we want
|
||||
to shift over Sass compilation as a whole to Webpack, meaning that at least some
|
||||
knowledge about where they are and how to compile them has to exist there. Also,
|
||||
there are JS assets in some themes that provide additional functionality, and it
|
||||
would be a performance degradation if those assets were no longer optimized.
|
||||
|
||||
What I do NOT want to happen:
|
||||
|
||||
* Significant end user performance degradation.
|
||||
* Having an *additional* system in the asset pipeline (e.g. keeping
|
||||
django-pipeline around while having additional systems).
|
||||
|
||||
I think that means that conceptually, there exists a larger Static Asset system
|
||||
that exists and that we think of both Webpack and Django being consumers of its
|
||||
configuration. This is also very fuzzy at the moment.
|
||||
|
||||
Asset Groups
|
||||
------------
|
||||
|
||||
There will be logical groupings of static assets. There should be uniformity and
|
||||
no duplication within a group, but we would allow duplication between groups to
|
||||
better facilitate independent deployment and isolation.
|
||||
|
||||
Example Groups:
|
||||
|
||||
* XBlock/XModule Assets
|
||||
* LMS/Studio apps in edx-platform
|
||||
* Third party app, such as edx-enterprise
|
||||
Reference in New Issue
Block a user