diff --git a/docs/static_assets.rst b/docs/static_assets.rst new file mode 100644 index 0000000000..4224a0d015 --- /dev/null +++ b/docs/static_assets.rst @@ -0,0 +1,207 @@ +####################################### +edx-platform Static Asset Pipeline Plan +####################################### + +Static asset handling in edx-platform has evolved in a messy way over the years. +This has led to a lot of complexity and inconsistencies. This is a proposal for +how we can move forward to a simpler system and more modern toolchain. Note that +this is not a detailed guide for how to write React or Bootstrap code. This is +instead going to talk about conventions for how we arrange, extract, and compile +static assets. + +Big Open Questions (TODO) +************************* + +This document is a work in progress, as the design for some of this is still in +flux, particularly around extensibility. + +* Pluggable third party apps and Webpack packaging. +* Keep the Django i18n mechanism? +* Stance on HTTP/2 and bundling granularity. +* Optimizing theme assets. +* Tests + +Requirements +************ + +Any proposed solution must support: + +* Externally developed and installed Django apps. +* Theming. +* XBlock assets. +* Existing tests. +* Fast builds. +* An incremental implementation path. +* Other kinds of pluggability??? + +Assumptions +*********** + +Some assumptions/opinions that this proposal is based on: + +* We want to shift as much as possible to Webpack and the JavaScript stack of + technologies, leaving the Python layer as thin as possible. +* While we will try to make theming upgrades straightforward, we will be moving + around where files are located and where they're compiled out to. +* We will be pushing towards a world that is more Django app-centric than LMS + vs. Studio centric, to reduce duplication. +* At the same time, we want to consolidate assets far more efficiently than we + are doing today. +* Leaning towards more static front ends + API calls. +* However we still need to be compatible with Django's asset system for things + like third party apps (e.g. Django Rest Framework browsing assets, Swagger, + etc.) +* It should be possible to pre-build static assets and deploy them onto S3 or + similar. + +Where We Are Today +****************** + +We have a static asset pipeline that is mostly driven by Django's built-in +staticfiles finders and the collectstatic process. We use the popular +``django-pipeline`` library, with UglifyJS as the JavaScript compressor (the +binary is installed via node into node_modules). We also use the less well known +``django-pipeline-forgiving`` extension to ``django-pipeline`` so we don't error +out when files are missing (added when we started dynamically scanning XBlocks +for assets). + +The ``django-pipeline`` config is aware of CSS files for the purposes of +concatenation, but it does *not* know about the source Sass files (or handful of +remaining CoffeeScript files). Those are processed with paver tasks before +``django-pipeline`` ever sees them. + +We also have the following custom extensions to Django's builtin ``STATICFILES`` +mechanism: + +``openedx.core.djangoapps.theming.finders.ThemeFilesFinder`` + Custom finder that overrides any static asset with a version from the themes + directory (``COMPREHENSIVE_THEME_DIRS`` defined in ``lms.envs.json`` and + ``cms.envs.json``). + +``openedx.core.lib.xblock_pipeline.finder.XBlockPipelineFinder`` + Custom finder that accesses and extracts assets from pip-installed XBlocks via + ``pkg_resources``. + +``openedx.core.storage.DevelopmentStorage/ProductionStorage`` + Custom ``FileStorage`` classes that mostly exist for theme-awareness. + +LMS and Studio/CMS Separation +----------------------------- + +LMS and Studio have their own directories for source assets (``lms/static`` and +``cms/static``), and have symlinks to shared assets in ``common/static``. We +treat the static asset compilation and collection phase for LMS and Studio as +separate projects that happen to share a lot of pieces. They output to different +places (typically ``/edx/var/edxapp/staticfiles`` for LMS and +``/edx/var/edxapp/staticfiles/studio`` for Studio) and can be collected +separately. However in practice they're always run together because we deploy +them from the same commits and to the same servers. + +Django vs. Webpack Conventions +****************************** + +The Django convention for having an app with bundled assets is to namespace them +locally with the app name so that they get their own directories when they are +gathered together into a common static directory by collectstatic. For example, +the edx-enterprise app has a ``static/enterprise`` folder, so its assets are +compiled to ``/edx/var/edxapp/staticfiles/enterprise`` by edx-platform and will +not conflict with assets from any other Django app. + +Webpack conventions would have us create a single set of configuration files at +the root of edx-platform, which would specify all bundles in the project. + +TODO: The big, "pluggable Webpack components" question. + +Proposed Repo Structure +*********************** + +All assets that are in common spaces like ``common/static``, ``lms/static``, +and ``cms/static`` would be moved to be under the Django apps that they are a +part of and follow the Django naming convention (e.g. +``openedx/features/course_bookmarks/static/course_bookmarks``). An app's +``templates/{appname}`` directory will only be for server side templates, and +any client-side templates will be put in ``static/{appname}/templates``. + +Proposed Compiled Structure +*************************** + +This is meant to be a sample of the different types of things we'd have, not a +full list: + +:: + + # Webpack bundles/post-processed assets + /webpack/css + /fonts + /js + /vendor ? + + # Django apps that are in the edx-platform repo + /course_bookmarks + /course_experience + + # edX authored, installed via separate repo + /enterprise + + # Entirely third party apps that we need to maintain compatiblity with. + /admin + /rest_framework + + # Themes are part of the "theming" app + /theming/themes/open-edx + /red-theme + /edx.org + + # XBlocks still collect their assets into a common space (/xmodule goes away) + # We consider this to be the XBlock Runtime's app, and it collects static + # assets from installed XBlocks. + /xblock + +Django vs. Webpack Roles +************************ + +Rule of thumb: Django/Python still serves static assets, Webpack processes and +optimizes them. + +Webpack would be responsible for all Sass compilation in edx-platform. It would +also be responsible for the optimization/minification of JavaScript assets, but +those optimized assets would only appear under the ``/webpack`` directory. Third +party assets that Webpack is not aware of may have hash suffixes applied to them +by the Django collectstatic layer, but will not otherwise be processed or +optimized in any way -- so no coffeescript/sass compilation, no uglifyjs +minification, etc. + +The django-pipeline dependency should be removed altogether. + +Themes +------ + +Theme handling is muddled. The fact that themes can override server-side +templates means that Python has to be aware of them. At the same time, we want +to shift over Sass compilation as a whole to Webpack, meaning that at least some +knowledge about where they are and how to compile them has to exist there. Also, +there are JS assets in some themes that provide additional functionality, and it +would be a performance degradation if those assets were no longer optimized. + +What I do NOT want to happen: + +* Significant end user performance degradation. +* Having an *additional* system in the asset pipeline (e.g. keeping + django-pipeline around while having additional systems). + +I think that means that conceptually, there exists a larger Static Asset system +that exists and that we think of both Webpack and Django being consumers of its +configuration. This is also very fuzzy at the moment. + +Asset Groups +------------ + +There will be logical groupings of static assets. There should be uniformity and +no duplication within a group, but we would allow duplication between groups to +better facilitate independent deployment and isolation. + +Example Groups: + +* XBlock/XModule Assets +* LMS/Studio apps in edx-platform +* Third party app, such as edx-enterprise