Andal.LND/edx-platform
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move existing docs down a level to make room for more

Browse Source
This commit is contained in:
Ned Batchelder
2019-07-23 07:26:03 -04:00
parent f2be56c3fb
commit 3b69fe0b47
23 changed files with 14 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/guides/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
_build
cms
common
lms
openedx

22
docs/guides/Makefile Normal file
View File

@@ -0,0 +1,22 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help clean Makefile
clean:
rm -rf _build cms common lms openedx
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

0
docs/guides/__init__.py Normal file
View File

329
docs/guides/conf.py Normal file
View File

@@ -0,0 +1,329 @@
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
from __future__ import absolute_import, unicode_literals
import datetime
import os
import sys
from subprocess import check_call
import django
import edx_theme
import six
from path import Path
root = Path('../..').abspath()
# Hack the PYTHONPATH to match what LMS and Studio use so all the code
# can be successfully imported
sys.path.insert(0, root)
sys.path.append(root / "docs/guides")
sys.path.append(root / "cms/djangoapps")
sys.path.append(root / "common/djangoapps")
sys.path.append(root / "common/lib/capa")
sys.path.append(root / "common/lib/safe_lxml")
sys.path.append(root / "common/lib/symmath")
sys.path.append(root / "common/lib/xmodule")
sys.path.append(root / "lms/djangoapps")
sys.path.append(root / "lms/envs")
sys.path.append(root / "openedx/core/djangoapps")
sys.path.append(root / "openedx/features")
# Use a settings module that allows all LMS and Studio code to be imported
# without errors. If running sphinx-apidoc, we already set a different
# settings module to use in the on_init() hook of the parent process
if 'DJANGO_SETTINGS_MODULE' not in os.environ:
os.environ['DJANGO_SETTINGS_MODULE'] = 'docs_settings'
django.setup()
# -- Project information -----------------------------------------------------
project = u'edx-platform'
copyright = edx_theme.COPYRIGHT
author = edx_theme.AUTHOR
# The short X.Y version
version = u''
# The full version, including alpha/beta/rc tags
release = u''
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.coverage',
'sphinx.ext.doctest',
'sphinx.ext.ifconfig',
'sphinx.ext.intersphinx',
'sphinx.ext.mathjax',
'sphinx.ext.napoleon',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'edx_theme'
html_theme_path = [edx_theme.get_html_theme_path()]
html_theme_options = {'navigation_depth': 3}
html_favicon = os.path.join(edx_theme.get_html_theme_path(), 'edx_theme', 'static', 'css', 'favicon.ico')
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'edx-platformdoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'edx-platform.tex', u'edx-platform Documentation',
author, 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'edx-platform', u'edx-platform Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'edx-platform', u'edx-platform Documentation',
author, 'edx-platform', 'The Open edX platform, the software that powers edX!',
'Miscellaneous'),
]
# -- Options for Epub output -------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = project
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''
# A unique identification for the text.
#
# epub_uid = ''
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
# -- Extension configuration -------------------------------------------------
# -- Options for intersphinx extension ---------------------------------------
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {
'https://docs.python.org/2.7': None,
'django': ('https://docs.djangoproject.com/en/1.11/', 'https://docs.djangoproject.com/en/1.11/_objects/'),
}
# Mock out these external modules during code import to avoid errors
autodoc_mock_imports = [
'MySQLdb',
'contracts',
'django_mysql',
'pymongo',
]
# Start building a map of the directories relative to the repository root to
# run sphinx-apidoc against and the directories under "docs" in which to store
# the generated *.rst files
modules = {
'cms': 'cms',
'common/lib/capa/capa': 'common/lib/capa',
'common/lib/safe_lxml/safe_lxml': 'common/lib/safe_lxml',
'common/lib/symmath/symmath': 'common/lib/symmath',
'common/lib/xmodule/xmodule': 'common/lib/xmodule',
'lms': 'lms',
'openedx': 'openedx',
}
# These Django apps under cms don't import correctly with the "cms.djangapps" prefix
# Others don't import correctly without it...INSTALLED_APPS entries are inconsistent
cms_djangoapps = ['contentstore', 'course_creators', 'xblock_config']
for app in cms_djangoapps:
path = os.path.join('cms', 'djangoapps', app)
modules[path] = path
# The Django apps under common must be imported directly, not under their path
for app in os.listdir(six.text_type(root / 'common' / 'djangoapps')):
path = os.path.join('common', 'djangoapps', app)
if os.path.isdir(six.text_type(root / path)) and app != 'terrain':
modules[path] = path
# These Django apps under lms don't import correctly with the "lms.djangapps" prefix
# Others don't import correctly without it...INSTALLED_APPS entries are inconsistent
lms_djangoapps = ['badges', 'branding', 'bulk_email', 'courseware',
'coursewarehistoryextended', 'email_marketing', 'experiments', 'lti_provider',
'mobile_api', 'notes', 'rss_proxy', 'shoppingcart', 'survey']
for app in lms_djangoapps:
path = os.path.join('lms', 'djangoapps', app)
modules[path] = path
def update_settings_module(service='lms'):
"""
Set the "DJANGO_SETTINGS_MODULE" environment variable appropriately
for the module sphinx-apidoc is about to be run on.
"""
if os.environ['EDX_PLATFORM_SETTINGS'] == 'devstack_docker':
settings_module = '{}.envs.devstack_docker'.format(service)
else:
settings_module = '{}.envs.devstack'.format(service)
os.environ['DJANGO_SETTINGS_MODULE'] = settings_module
def on_init(app): # pylint: disable=unused-argument
"""
Run sphinx-apidoc after Sphinx initialization.
Read the Docs won't run tox or custom shell commands, so we need this to
avoid checking in the generated reStructuredText files.
"""
docs_path = root / 'docs' / 'guides'
apidoc_path = 'sphinx-apidoc'
if hasattr(sys, 'real_prefix'): # Check to see if we are in a virtualenv
# If we are, assemble the path manually
bin_path = os.path.abspath(os.path.join(sys.prefix, 'bin'))
apidoc_path = os.path.join(bin_path, apidoc_path)
exclude_dirs = ['envs', 'migrations', 'test', 'tests']
exclude_dirs.extend(cms_djangoapps)
exclude_dirs.extend(lms_djangoapps)
exclude_files = ['admin.py', 'test.py', 'testing.py', 'tests.py', 'testutils.py', 'wsgi.py']
for module in modules:
module_path = six.text_type(root / module)
output_path = six.text_type(docs_path / modules[module])
args = [apidoc_path, '--ext-intersphinx', '-o',
output_path, module_path]
exclude = []
if module == 'cms':
update_settings_module('cms')
else:
update_settings_module('lms')
for dirpath, dirnames, filenames in os.walk(module_path):
to_remove = []
for name in dirnames:
if name in exclude_dirs:
to_remove.append(name)
exclude.append(os.path.join(dirpath, name))
if 'features' in dirnames and 'openedx' not in dirpath:
to_remove.append('features')
exclude.append(os.path.join(dirpath, 'features'))
for name in to_remove:
dirnames.remove(name)
for name in filenames:
if name in exclude_files:
exclude.append(os.path.join(dirpath, name))
if exclude:
args.extend(exclude)
check_call(args)
def setup(app):
"""Sphinx extension: run sphinx-apidoc."""
event = b'builder-inited' if six.PY2 else 'builder-inited'
app.connect(event, on_init)

35
docs/guides/docs_settings.py Normal file
View File

@@ -0,0 +1,35 @@
"""
Django settings for use when generating API documentation.
Basically the LMS devstack settings plus a few items needed to successfully
import all the Studio code.
"""
from __future__ import absolute_import, unicode_literals
import os
if os.environ['EDX_PLATFORM_SETTINGS'] == 'devstack_docker':
from lms.envs.devstack_docker import *
from cms.envs.devstack_docker import (
ADVANCED_PROBLEM_TYPES,
COURSE_IMPORT_EXPORT_STORAGE,
SCRAPE_YOUTUBE_THUMBNAILS_JOB_QUEUE,
VIDEO_TRANSCRIPT_MIGRATIONS_JOB_QUEUE,
)
else:
from lms.envs.devstack import *
from cms.envs.devstack import (
ADVANCED_PROBLEM_TYPES,
COURSE_IMPORT_EXPORT_STORAGE,
SCRAPE_YOUTUBE_THUMBNAILS_JOB_QUEUE,
VIDEO_TRANSCRIPT_MIGRATIONS_JOB_QUEUE,
)
FEATURES['ENABLE_LTI_PROVIDER'] = True
INSTALLED_APPS.extend([
'contentstore.apps.ContentstoreConfig',
'course_creators',
'xblock_config.apps.XBlockConfig',
'user_tasks',
'lti_provider'
])

14
docs/guides/docstrings/cms_index.rst Normal file
View File

@@ -0,0 +1,14 @@
cms
***
The ``cms`` (course management system) directory in edx-platform is home to
the code needed for the Course Authoring Studio which is not also needed for
the LMS.
.. toctree::
:maxdepth: 2
cms/modules
cms/djangoapps/contentstore/modules
cms/djangoapps/course_creators/modules
cms/djangoapps/xblock_config/modules

24
docs/guides/docstrings/common_djangoapps.rst Normal file
View File

@@ -0,0 +1,24 @@
common/djangoapps
*****************
This directory contains Django applications intended to be used in both the
LMS and Studio.
.. toctree::
:maxdepth: 2
common/djangoapps/course_action_state/modules
common/djangoapps/course_modes/modules
common/djangoapps/database_fixups/modules
common/djangoapps/edxmako/modules
common/djangoapps/enrollment/modules
common/djangoapps/entitlements/modules
common/djangoapps/microsite_configuration/modules
common/djangoapps/pipeline_mako/modules
common/djangoapps/static_replace/modules
common/djangoapps/status/modules
common/djangoapps/student/modules
common/djangoapps/third_party_auth/modules
common/djangoapps/track/modules
common/djangoapps/util/modules
common/djangoapps/xblock_django/modules

14
docs/guides/docstrings/common_index.rst Normal file
View File

@@ -0,0 +1,14 @@
common
******
The ``common`` directory in edx-platform is home to an assortment of packages
used by the LMS and Studio. This is a legacy code organization decision, and
it is currently intended that most of the code here will eventually either be
moved into the ``openedx`` package or broken out into a separately installed
package.
.. toctree::
:maxdepth: 2
common_djangoapps
common_lib

14
docs/guides/docstrings/common_lib.rst Normal file
View File

@@ -0,0 +1,14 @@
common/lib
**********
This directory contains libraries which are installed locally so they can be
imported by name with no package hierarchy. They will most likely be split
out from edx-platform into separate packages at some point.
.. toctree::
:maxdepth: 2
common/lib/capa/modules
common/lib/safe_lxml/modules
common/lib/symmath/modules
common/lib/xmodule/modules

10
docs/guides/docstrings/docstrings.rst Normal file
View File

@@ -0,0 +1,10 @@
Python Docstrings
*****************
.. toctree::
:maxdepth: 2
cms_index
common_index
lms_index
openedx/modules

24
docs/guides/docstrings/lms_index.rst Normal file
View File

@@ -0,0 +1,24 @@
lms
***
The ``lms`` directory in edx-platform is home to the code needed for the LMS
(Learning Management System) which is not also needed for the Course Authoring
Studio.
.. toctree::
:maxdepth: 2
lms/modules
lms/djangoapps/badges/modules
lms/djangoapps/branding/modules
lms/djangoapps/bulk_email/modules
lms/djangoapps/courseware/modules
lms/djangoapps/coursewarehistoryextended/modules
lms/djangoapps/email_marketing/modules
lms/djangoapps/experiments/modules
lms/djangoapps/lti_provider/modules
lms/djangoapps/mobile_api/modules
lms/djangoapps/notes/modules
lms/djangoapps/rss_proxy/modules
lms/djangoapps/shoppingcart/modules
lms/djangoapps/survey/modules

88
docs/guides/frontend/bootstrap.rst Normal file
View File

@@ -0,0 +1,88 @@
.. _ui_bootstrap:
######################
Working with Bootstrap
######################
This topic describes the Bootstrap framework, and how it should be used to
build user interfaces within edX applications. Note that Bootstrap adoption
started after the Ginkgo release, and so this document applies to the edX
master branch and to the forthcoming Hawthorn release.
If you are interested in the rationale for edX choosing Bootstrap, you can
read about the decision in `OEP-16: Adopting Bootstrap
<https://open-edx-proposals.readthedocs.io/en/latest/oep-0016-bp-adopt-bootstrap.html>`_.
.. highlight:: none
***************
Getting Started
***************
Bootstrap is an open source front end component library that is used by many of
the world's most popular applications. It allows for rapid assembly of front end
components using a responsive grid system, a robust component library and easy
to configure theming capabilities to ensure that new components are rendered
consistently. EdX is using `Bootstrap 4`_ which is a reimplemented version using
Sass and that is currently in beta release.
All edX applications should use the `edx-bootstrap`_ package that can be
installed via `npm`_. This package provides two themes (a default Open edX
theme, as well as an edX branded version), and eventually will provide custom
Open edX styles for common patterns.
.. _ui_bootstrap_custom_designs:
*************************
Developing Custom Designs
*************************
Bootstrap provides a large number of components and layouts out-of-the-box, but
there will always be a need to implement custom designs. There are a number of
considerations to take into account when implementing your designs using Sass.
The most important rule is to avoid hard-coding values such as colors and fonts.
Using hard-coded values means that Bootstrap themes will not be able to affect
your styles, and so your new elements will look out of place. Whenever possible
you should instead use the functions and variables provided by Bootstrap to
access theme colors or fonts. See `Bootstrap customization options`_ for more
details.
For example, here is an example of a hard-coded style::
.my-element {
font-family: "Open Sans";
color: #0000ff;
}
The recommended alternative is as follows::
.my-element {
font-family: $font-family-sans-serif;
color: theme-color("primary");
}
If you do find the need for a custom color or font that isn't provided by
the edX Bootstrap library, consider first whether it makes sense to contribute
it back so that other applications can use this value too. If you decide to
add a custom value, define it as a variable that can be overridden by a theme by
using the ``!default`` flag. This allows themes to provide a different value
for this variable if they choose. See the Sass documentation for `default flag`_
for more details.
For example::
$my-custom-color: #0000ff !default;
.my-element {
font-family: $font-family-sans-serif;
color: $my-custom-color;
}
.. _Bootstrap 4: https://getbootstrap.com/docs/4.0/getting-started/introduction/
.. _Bootstrap customization options: https://getbootstrap.com/docs/4.0/getting-started/options/
.. _default flag: http://sass-lang.com/documentation/file.SASS_REFERENCE.html#Variable_Defaults___default
.. _edx-bootstrap: https://www.npmjs.com/package/@edx/edx-bootstrap
.. _npm: https://www.npmjs.com/

110
docs/guides/frontend/javascript.rst Normal file
View File

@@ -0,0 +1,110 @@
JavaScript in edx-platform
==========================
ES2015
------
All new JavaScript code in edx-platform should be written in ES2015.
ES2015 is not a framework or library -- rather, it is the latest and
greatest revision of the JavaScript language itself, natively supported
in all modern browsers and engines. Think of it as JavaScript's
equivalent to Python 3. ES2015 brings with it number of wonderful
syntactic features, such as classes, native imports, arrow functions,
and new data structures. To learn more about ES2015, check out `Luke
Hoban's comprehensive ES6 Features
repo <https://github.com/lukehoban/es6features>`__.
Although ES2015 is natively supported in modern browsers, older browsers
can't interpret it. Here at edX, we support the two latest versions of
every browser, plus IE11, so we need to do a little extra work to
support ES2015. This is where Webpack and Babel come in. Webpack is a
module bundler that transforms, minifies, and compiles frontend code
into pre-built "bundles" to include within pages. It works together with
Babel to transpile ES2015 code into ES5 code, which can safely be used
in all browsers.
Fortunately, you don't need to worry about the gritty details of Webpack
in order to write ES2015 code. You just need to make sure Webpack knows
where to find your files. It's also important to note that **Webpack is
not compatible with RequireJS**. Work is currently underway to move all
legacy RequireJS modules into Webpack, but until it is complete, you
will need to update legacy code yourself in order to use it with ES2015.
Adding a New ES2015 Module
~~~~~~~~~~~~~~~~~~~~~~~~~~
Don't mix ES2015 and ES5 modules within directories. If necessary,
create a new directory just for your new file. If you create a new
directory, run the following from edx-platform root to copy over an
appropriate eslint config:
::
cp cms/static/js/features_jsx/.eslintrc.js path/to/your/directory
Give your new file an UpperCamelCase filename, such as
``MyAwesomeModule.js``. If it is a React module, use the ``.jsx``
extension; otherwise, use the ``.js`` extension.
If you intend to include this module itself directly within a page, you
will need to tell Webpack about it. Add a line to the ``entry`` object
within ``webpack.common.config.js``.
::
'MyAwesomeModule': 'path/to/your/directory/MyAwesomeModule.js',
The new entry's key should be the name of your module (typically this is
the same as your filename), and its value should be the path to your
file relative to the edx-platform root.
Writing Your File
~~~~~~~~~~~~~~~~~
Structure your module using ``class``\ es. Typically, you'll want to
define and export one ``class`` per file. If you are going to be
including this module directly within a page and passing it through
Webpack, use a non-default export. ``MyAwesomeModule.js`` should look
something like this:
::
export class MyAwesomeModule {
// your awesome code here
}
Use two-space indentation. This is industry standard practice for
ES2015. If you need to pull in external dependencies, use ``import``
statements:
::
import moment from 'moment';
import 'jquery.cookie';
import { MyOtherModule } from './MyOtherModule';
Building Your File
~~~~~~~~~~~~~~~~~~
Devstack comes with two watcher containers specifically for building
assets. They compile frontend files very quickly, so you can see your
changes reflected in a browser almost immediately. You can run these
containers with:
::
make dev.up.watchers
and stop them with
::
make stop.watchers
If you make any changes to ``webpack.common.config.js`` while the
watchers are running, you will need to restart the watchers in order for
them to pick up your changes.
If your changes aren't being reflected in the browser, check the logs
with ``make logs`` to see if something went wrong. If you get stuck, ask
for help in the FedX hipchat room, or in #front-end on Slack.

205
docs/guides/frontend/static_assets.rst Normal file
View File

@@ -0,0 +1,205 @@
#######################################
edx-platform Static Asset Pipeline Plan
#######################################
Static asset handling in edx-platform has evolved in a messy way over the years.
This has led to a lot of complexity and inconsistencies. This is a proposal for
how we can move forward to a simpler system and more modern toolchain. Note that
this is not a detailed guide for how to write React or Bootstrap code. This is
instead going to talk about conventions for how we arrange, extract, and compile
static assets.
Big Open Questions (TODO)
*************************
This document is a work in progress, as the design for some of this is still in
flux, particularly around extensibility.
* Pluggable third party apps and Webpack packaging.
* Keep the Django i18n mechanism?
* Stance on HTTP/2 and bundling granularity.
* Optimizing theme assets.
* Tests
Requirements
************
Any proposed solution must support:
* Externally developed and installed Django apps.
* Theming.
* XBlock assets.
* Existing tests.
* Fast builds.
* An incremental implementation path.
* Other kinds of pluggability???
Assumptions
***********
Some assumptions/opinions that this proposal is based on:
* We want to shift as much as possible to Webpack and the JavaScript stack of
technologies, leaving the Python layer as thin as possible.
* While we will try to make theming upgrades straightforward, we will be moving
around where files are located and where they're compiled out to.
* We will be pushing towards a world that is more Django app-centric than LMS
vs. Studio centric, to reduce duplication.
* At the same time, we want to consolidate assets far more efficiently than we
are doing today.
* Leaning towards more static front ends + API calls.
* However we still need to be compatible with Django's asset system for things
like third party apps (e.g. Django Rest Framework browsing assets, Swagger,
etc.)
* It should be possible to pre-build static assets and deploy them onto S3 or
similar.
Where We Are Today
******************
We have a static asset pipeline that is mostly driven by Django's built-in
staticfiles finders and the collectstatic process. We use the popular
``django-pipeline`` library, with UglifyJS as the JavaScript compressor (the
binary is installed via node into node_modules). We also use the less well known
``django-pipeline-forgiving`` extension to ``django-pipeline`` so we don't error
out when files are missing (added when we started dynamically scanning XBlocks
for assets).
The ``django-pipeline`` config is aware of CSS files for the purposes of
concatenation, but it does *not* know about the source Sass files.
Those are processed with paver tasks before ``django-pipeline`` ever sees them.
We also have the following custom extensions to Django's builtin ``STATICFILES``
mechanism:
``openedx.core.djangoapps.theming.finders.ThemeFilesFinder``
Custom finder that overrides any static asset with a version from the themes
directory (``COMPREHENSIVE_THEME_DIRS`` defined in ``lms.envs.json`` and
``cms.envs.json``).
``openedx.core.lib.xblock_pipeline.finder.XBlockPipelineFinder``
Custom finder that accesses and extracts assets from pip-installed XBlocks via
``pkg_resources``.
``openedx.core.storage.DevelopmentStorage/ProductionStorage``
Custom ``FileStorage`` classes that mostly exist for theme-awareness.
LMS and Studio/CMS Separation
-----------------------------
LMS and Studio have their own directories for source assets (``lms/static`` and
``cms/static``), and have symlinks to shared assets in ``common/static``. We
treat the static asset compilation and collection phase for LMS and Studio as
separate projects that happen to share a lot of pieces. They output to different
places (typically ``/edx/var/edxapp/staticfiles`` for LMS and
``/edx/var/edxapp/staticfiles/studio`` for Studio) and can be collected
separately. However in practice they're always run together because we deploy
them from the same commits and to the same servers.
Django vs. Webpack Conventions
******************************
The Django convention for having an app with bundled assets is to namespace them
locally with the app name so that they get their own directories when they are
gathered together into a common static directory by collectstatic. For example,
the edx-enterprise app has a ``static/enterprise`` folder, so its assets are
compiled to ``/edx/var/edxapp/staticfiles/enterprise`` by edx-platform and will
not conflict with assets from any other Django app.
Webpack conventions would have us create a single set of configuration files at
the root of edx-platform, which would specify all bundles in the project.
TODO: The big, "pluggable Webpack components" question.
Proposed Repo Structure
***********************
All assets that are in common spaces like ``common/static``, ``lms/static``,
and ``cms/static`` would be moved to be under the Django apps that they are a
part of and follow the Django naming convention (e.g.
``openedx/features/course_bookmarks/static/course_bookmarks``). An app's
``templates/{appname}`` directory will only be for server side templates, and
any client-side templates will be put in ``static/{appname}/templates``.
Proposed Compiled Structure
***************************
This is meant to be a sample of the different types of things we'd have, not a
full list:
::
# Webpack bundles/post-processed assets
/webpack/css
/fonts
/js
/vendor ?
# Django apps that are in the edx-platform repo
/course_bookmarks
/course_experience
# edX authored, installed via separate repo
/enterprise
# Entirely third party apps that we need to maintain compatiblity with.
/admin
/rest_framework
# Themes are part of the "theming" app
/theming/themes/open-edx
/red-theme
/edx.org
# XBlocks still collect their assets into a common space (/xmodule goes away)
# We consider this to be the XBlock Runtime's app, and it collects static
# assets from installed XBlocks.
/xblock
Django vs. Webpack Roles
************************
Rule of thumb: Django/Python still serves static assets, Webpack processes and
optimizes them.
Webpack would be responsible for all Sass compilation in edx-platform. It would
also be responsible for the optimization/minification of JavaScript assets, but
those optimized assets would only appear under the ``/webpack`` directory. Third
party assets that Webpack is not aware of may have hash suffixes applied to them
by the Django collectstatic layer, but will not otherwise be processed or
optimized in any way -- so no sass compilation, no uglifyjs minification, etc.
The django-pipeline dependency should be removed altogether.
Themes
------
Theme handling is muddled. The fact that themes can override server-side
templates means that Python has to be aware of them. At the same time, we want
to shift over Sass compilation as a whole to Webpack, meaning that at least some
knowledge about where they are and how to compile them has to exist there. Also,
there are JS assets in some themes that provide additional functionality, and it
would be a performance degradation if those assets were no longer optimized.
What I do NOT want to happen:
* Significant end user performance degradation.
* Having an *additional* system in the asset pipeline (e.g. keeping
django-pipeline around while having additional systems).
I think that means that conceptually, there exists a larger Static Asset system
that exists and that we think of both Webpack and Django being consumers of its
configuration. This is also very fuzzy at the moment.
Asset Groups
------------
There will be logical groupings of static assets. There should be uniformity and
no duplication within a group, but we would allow duplication between groups to
better facilitate independent deployment and isolation.
Example Groups:
* XBlock/XModule Assets
* LMS/Studio apps in edx-platform
* Third party app, such as edx-enterprise

163
docs/guides/frontend/styling.rst Normal file
View File

@@ -0,0 +1,163 @@
#######################
Styling in edx-platform
#######################
Over time, our Sass styling has become a little convoluted, with three major
reworkings often leading to confusion when a developer needs to style in
multiple locations across edx-platform. The main endeavors were v1, v2 (or
pattern library) and Bootstrap. We are trying to move away from using v2, as the
pattern library is deprecated, but there may still be locations in the code that
reference those styles (please remove them as you see fit).
Our platform uses a system of Sass partials that combine to compile into a
single large css file to be rendered on the page. From the Sass docs:
You can create partial Sass files that contain little snippets of CSS
that you can include in other Sass files. This is a great way to
modularize your CSS and help keep things easier to maintain. A partial
is simply a Sass file named with a leading underscore. You might name it
something like _partial.scss. The underscore lets Sass know that the
file is only a partial file and that it should not be generated into a
CSS file. Sass partials are used with the ``@import`` directive.
This structure allows us to break up our styling into small pieces, making
readability and maintenance easier, while often at the expense of structural
complexity as a code base grows. Here is an example, directly from edx-platform,
for how the partials flow to a single scss file that compiles into CSS for the
page to use. This is a page that uses v1 styling.
.. figure:: v1_sass_pipeline.webp
:alt: Sass Compilation for v1 CSS
Sass Compilation for v1 CSS
Note that this only shows an example, there are far more partials that come
together to make the ``lms-main-v1.css`` final file.
As you can see, the ``lms-main-v1.scss`` file does not have a leading
underscore, telling the compilation to turn that scss file into an actual css
file to be rendered on the page. If you were to use the Chrome Inspector tool
and look at the styling, you will see that there is one unified ``lms-
main-v1.css`` file that contains all the element styles.
The ``lms-main-v1.scss`` file uses the ``@import`` statement to pull in the
``_build- base-v1.scss`` and ``build-lms-v1.scss`` files which in turn import
more partials down the road. It is critical to recognize that this tree works in
a depth-first, first-come-first-serve manner. That means that styles that are
imported early in the process cannot reference Sass variables imported later in
the process (ie: ``bootstrap/variables`` cannot reference variables from
``shared/header``).
This diagram describes the process for the **v1 styles**, and there is a similar
setup for ``lms-main-v2.scss`` as well as ``bootstrap/lms-main.scss``. Each
individual HTML page on the edx-platform specifies which of the three that page
wants to use for styling. Please note that as an organization, we are slowly
trying to 1) move everything over to the bootstrap/lms-main.scss file and 2)
deprecate and stop using any v2 files. ``lms-main-v1.scss`` can still be used,
but a migration over to ``bootstrap/lms-main.scss``, and a migration of any
relevant partials from the v1 structure, would be a valuable endeavor.
Please note that as you add partials, make sure that they are not already being
imported in another file (ie: you add a node to the tree above that already
exists on the tree).
What theming does, and how to do it
***********************************
By thinking about the styling as a tree, theming becomes a lot simpler. All
theming means is that you can override one of the partials above by matching
the exact path in the ``edx-platform/themes`` directory. So, for example, to
override the shared header file (located at ``lms/static/sass/shared/_header``),
you would simply go into the ``edx-platform/themes/[theme you want to
override]/lms/static/shared`` folder and add a _header.scss file. When django
compiles the assets, it will use this file as a replacement to the main
edx-platform implementation.
If you look at the actual code base, you will see that we have a standard of
using the partials directory for these overrides files. This keeps the specific
components isolated from the core styling to reduce errors when people override
files and forget to import other files that are needed elsewhere on the site.
This is a good practice that future development should adhere to.
Final Note: When dealing with front end changes, it is a good idea to also check
the edx-themes repo, that works exactly like our themes folder, but also
includes html templates that can further confuse things.
Bootstrap and edx-platform
**************************
In a month and a half endeavor in the Fall, Andy and I worked on integrating
Bootstrap into the platform for three main reasons.
1. **To unify our styling:** By specifying colors and variables that can be used
universally, reducing the '50 shades of grey' issue and inconsistencies in the
ways that we style components.
2. **To add a widely used component library:** Bootstrap has a robust community
of developers that contribute to their open source component library, allowing
for easier prototyping and create of front end experiences.
3. **To simplify theming:** Since we now have a single unified source of truth
for our variables, openedx instances and alternative themes can simply override
those files to customly style their sites.
Relating to the above styling conversation, we have pulled in the entire
bootstrap styling library into ``lms-main.scss``, but only partially pulled it
into the v1 and v2 implementations due to naming conflicts. For example, we
could not pull in the bootstrap modal, due to conflicts with our own modal
styling (both use the generic ``.modal`` class). We were, however, able **to
pull in the entire bootstrap javascript file**, since that file overrides jQuery
in ways that our current application does not (meaning no conflicts).
Therefore, any pages that use lms-main.scss can use any bootstrap component out
of the box. **To use bootstrap components in older, v1, v2 pages** we have to
manually specify the exact Sass partial from the ``edx-
platform/node_modules/bootstrap`` that we want. For example, to add an alert to
a v1 page, you would add to ``base/build`` the partial ``bootstrap/scss/alert``.
Steps outlined below.
1. Determine from the ``node_modules/bootstrap/scss`` directory which component
you want to import.
2. Add that import (i.e: ``bootstrap/scss/alert``) somewhere in the Sass tree,
most likely to ``base/build.scss``
3. NOTE: Run ``paver compile_sass`` to make sure it works, if not, there is
likely a bootstrap mixin that you are missing. If so, search for the mixin in
``node_modules/bootstrap/scss/mixins`` and import it (i.e:
``bootstrap/scss/mixins/alert``)
4. You are done. Add the component using HTML or Javascript. Use the Bootstrap
Component Docs for examples.
FAQ
***
I want to add a new scss file for a feature on a particular page, how do I add it?
##################################################################################
First, you want to check which root file the page uses, whether it is
lms-main-v1, lms-main-v2 or lms-main. Then, go to that file and trace down the
tree to find a good spot for the new scss file to live. So, for example, if you
are adding an LMS feature and the page uses lms-main-v1, you can trace down lms-
main-v1 > _build_lms_v1 and see that there are plenty of standalone imports that
look like features. Simply add your file to the lms/static/sass directory in a
similar manner and add it as an @import to that page.
Why isn't my bootstrap component styling like in the bootstrap docs?
####################################################################
In any pages that use the lms-main.css compiled file, the bootstrap component
will render. This case only arises when we are working with a legacy v1 or v2
page. As outlined earlier in this document, we cannot import the entire
bootstrap repository due to conflicts with old styling. For v1 and v2, we are
gradually pulling in styles for components, so this must mean that you are
building a component that has not yet been used in the LMS/Studio.
To add the styles, you first need to find them in the bootstrap package. To do
so, search at edx-platform/node_modules/bootstrap/scss for the file that you
need. Then add this to the v1 or v2 tree, most likely to the build/base.scss
file. Note that you may hit issues with mixins when you try to compile the SASS.
In this case, check the edx-platform/node_modules/bootstrap/scss/mixins folder
and import that into the lms or studio before trying to import the component
styling.

11
docs/guides/guides.rst Normal file
View File

@@ -0,0 +1,11 @@
Guides
******
.. toctree::
:maxdepth: 2
testing/testing
frontend/javascript
frontend/styling
frontend/bootstrap
frontend/static_assets

44
docs/guides/index.rst Normal file
View File

@@ -0,0 +1,44 @@
##########################
edx-platform Documentation
##########################
Developer documentation for `edx-platform` can be found in the following
locations.
* The `edx-platform docs directory`_ contains some local developer
documentation.
* The `Developer Documentation Index`_ in Confluence provides additional links
to developer documentation for this and other projects. The rest of the `Open
edX Development space`_ in Confluence provides additional documentation.
* User documentation and a more general Developer's Guide can be read on `Open
edX ReadTheDocs`_. The source for these guides can be found in the
`edx-documentation`_ repository.
.. _edx-platform docs directory: https://github.com/edx/edx-platform/tree/master/docs
.. _Developer Documentation Index: https://openedx.atlassian.net/wiki/display/OpenDev/Developer+Documentation
.. _Open edX Development space: https://openedx.atlassian.net/wiki/spaces/OpenDev/overview
.. _Open edX ReadTheDocs: http://docs.edx.org/
.. toctree::
:maxdepth: 2
guides
docstrings/docstrings
Change History
**************
* April, 2019: API and repository-specific documentation builds resumed.
* May, 2017: The local docs directory was cleared out to start fresh.
* January 13, 2015: The "edX Developer's Guide" was moved to
`edx-documentation`_.
* November 3, 2014: The documentation for several sub-projects were moved into
`edx-documentation`_.
.. _edx-documentation: https://github.com/edx/edx-documentation

35
docs/guides/make.bat Normal file
View File

@@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd

BIN
docs/guides/testing/test_pyramid.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

769
docs/guides/testing/testing.rst Normal file
View File

@@ -0,0 +1,769 @@
#######
Testing
#######
.. contents::
:local:
:depth: 3
Overview
========
We maintain three kinds of tests: unit tests, integration tests, and
acceptance tests.
Overall, you want to write the tests that **maximize coverage** while
**minimizing maintenance**. In practice, this usually means investing
heavily in unit tests, which tend to be the most robust to changes in
the code base.
.. figure:: test_pyramid.png
:alt: Test Pyramid
Test Pyramid
The pyramid above shows the relative number of unit tests, integration
tests, and acceptance tests. Most of our tests are unit tests or
integration tests.
Test Types
----------
Unit Tests
~~~~~~~~~~
- Each test case should be concise: setup, execute, check, and
teardown. If you find yourself writing tests with many steps,
consider refactoring the unit under tests into smaller units, and
then testing those individually.
- As a rule of thumb, your unit tests should cover every code branch.
- Mock or patch external dependencies. We use the voidspace `Mock Library`_.
- We unit test Python code (using `unittest`_) and Javascript (using
`Jasmine`_)
.. _Mock Library: http://www.voidspace.org.uk/python/mock/
.. _unittest: http://docs.python.org/2/library/unittest.html
.. _Jasmine: http://jasmine.github.io/
Integration Tests
~~~~~~~~~~~~~~~~~
- Test several units at the same time. Note that you can still mock or patch
dependencies that are not under test! For example, you might test that
``LoncapaProblem``, ``NumericalResponse``, and ``CorrectMap`` in the ``capa``
package work together, while still mocking out template rendering.
- Use integration tests to ensure that units are hooked up correctly. You do
not need to test every possible input--that's what unit tests are for.
Instead, focus on testing the "happy path" to verify that the components work
together correctly.
- Many of our tests use the `Django test client`_ to simulate HTTP requests to
the server.
.. _Django test client: https://docs.djangoproject.com/en/dev/topics/testing/overview/
UI Acceptance Tests
~~~~~~~~~~~~~~~~~~~
- There should be very few UI acceptance tests since they are generally slow and
flaky. Use these to test only bare minimum happy paths for necessary features.
- We use `Bok Choy`_ to write end-user acceptance tests directly in Python,
using the framework to maximize reliability and maintainability.
.. _Bok Choy: https://bok-choy.readthedocs.org/en/latest/tutorial.html
Test Locations
--------------
- Python unit and integration tests: Located in subpackages called
``tests``. For example, the tests for the ``capa`` package are
located in ``common/lib/capa/capa/tests``.
- Javascript unit tests: Located in ``spec`` folders. For example,
``common/lib/xmodule/xmodule/js/spec`` and
``{cms,lms}/static/js/spec`` For consistency, you should use the
same directory structure for implementation and test. For example,
the test for ``src/views/module.js`` should be written in
``spec/views/module_spec.js``.
- UI acceptance tests:
- Set up and helper methods, and stubs for external services:
``common/djangoapps/terrain``
- Bok Choy Acceptance Tests: located under ``common/test/acceptance/tests``
- Bok Choy Accessibility Tests: located under ``common/test/acceptance/tests`` and tagged with ``@attr("a11y")``
- Bok Choy PageObjects: located under ``common/test/acceptance/pages``
Running Tests
=============
You can run all of the unit-level tests using this command::
paver test
This includes python, JavaScript, and documentation tests. It does not,
however, run any acceptance tests.
Note -
`paver` is a scripting tool. To get information about various options, you can run the this command::
paver -h
Running Python Unit tests
-------------------------
We use `pytest`_ to run the test suite.
.. _pytest: https://pytest.org/
For example, this command runs all the python test scripts::
paver test_python
It also runs ``collectstatic``, which prepares the
static files used by the site (for example, compiling Sass to
CSS).
You can re-run all failed python tests by running this command (see note at end of
section)::
paver test_python --failed
To test lms python tests use this command::
paver test_system -s lms
To test cms python tests use this command::
paver test_system -s cms
To run these tests without ``collectstatic``, which is faster, append the following argument::
paver test_system -s lms --fasttest
To run cms python tests without ``collectstatic`` use this command::
paver test_system -s cms --fasttest
For the sake of speed, by default the python unit test database tables
are created directly from apps' models. If you want to run the tests
against a database created by applying the migrations instead, use the
``--enable-migrations`` option::
paver test_system -s lms --enable-migrations
To run a single django test class use this command::
paver test_system -t lms/djangoapps/courseware/tests/tests.py::ActivateLoginTest
Running a Single Test
~~~~~~~~~~~~~~~~~~~~~
When developing tests, it is often helpful to be able to really just run
one single test without the overhead of PIP installs, UX builds, etc. In
this case, it is helpful to look at the output of paver, and run just
the specific command (optionally, stripping away coverage metrics). At
the time of this writing, the command is the following::
pytest lms/djangoapps/courseware/tests/test_courses.py
To run a single test format the command like this::
paver test_system -t lms/djangoapps/courseware/tests/tests.py::ActivateLoginTest::test_activate_login
You can use ``--randomize`` to randomize the test case sequence. In the
short term, this is likely to reveal bugs in our test setup and teardown;
please fix (or at least file tickets for) any such issues you encounter.
You can also enable test concurrency with the ``--processes=N`` flag (where ``N``
is the number of processes to run tests with, and ``-1`` means one process per
available core). Note, however, that when running concurrently, breakpoints may
not work correctly.
For example::
# This will run all tests in the order that they appear in their files, serially
paver test_system -s lms --no-randomize --processes=0
# This will run using only 2 processes for tests
paver test_system -s lms --processes=2
To re-run all failing django tests from lms or cms, use the
``--failed``,\ ``-f`` flag (see note at end of section)::
paver test_system -s lms --failed
paver test_system -s cms --failed
There is also a ``--exitfirst``, ``-x`` option that will stop pytest
after the first failure.
common/lib tests are tested with the ``test_lib`` task, which also
accepts the ``--failed`` and ``--exitfirst`` options::
paver test_lib -l common/lib/xmodule
paver test_lib -l common/lib/xmodule --failed
For example, this command runs a single python unit test file::
pytest common/lib/xmodule/xmodule/tests/test_stringify.py
To select tests to run based on their name, provide an expression to the
`pytest -k option`_ which performs a substring match on test names::
pytest common/lib/xmodule/xmodule/tests/test_stringify.py -k test_stringify
.. _pytest -k option: https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name
.. _node ID: https://docs.pytest.org/en/latest/example/markers.html#node-id
Alternatively, you can select tests based on their `node ID`_ directly,
which is useful when you need to run only one of mutliple tests with the same
name in different classes or files.
This command runs any python unit test method that matches the substring
`test_stringify` within a specified TestCase class within a specified file::
pytest common/lib/xmodule/xmodule/tests/test_stringify.py::TestCase -k test_stringify
Note: if the method has an `@ddt.data` decorator, ddt will create multiple
methods with the same prefix name and each individual data input as the suffix
(e.g. `test_stringify_1_foo`). To test all of the ddt.data variations of the
same test method, pass the prefix name to the pytest `-k` option.
If you need to run only one of the test variations, you can the get the
name of all test methods in a class, file, or project, including all ddt.data
variations, by running pytest with `--collectonly`::
pytest common/lib/xmodule/xmodule/tests/test_stringify.py --collectonly
This is an example of how to run a single test and get stdout shown immediately, with proper env config::
pytest cms/djangoapps/contentstore/tests/test_import.py -s
How to output coverage locally
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These are examples of how to run a single test and get coverage::
pytest cms/djangoapps/contentstore/tests/test_import.py --cov --cov-conifg=.coveragerc-local # cms example
pytest lms/djangoapps/courseware/tests/test_module_render.py --cov --cov-conifg=.coveragerc-local # lms example
That ``--cov-conifg=.coveragerc-local`` option is important - without it, the coverage
tool will look for paths that exist on our jenkins test servers, but not on your local devstack.
How to spit out coverage for a single file with a list of each line that is missing coverage::
pytest lms/djangoapps/grades/tests/test_subsection_grade.py \
--cov=lms.djangoapps.grades.subsection_grade \
--cov-config=.coveragerc-local \
--cov-report=term-missing
---------- coverage: platform linux2, python 2.7.12-final-0 ----------
Name Stmts Miss Cover Missing
-------------------------------------------------------------------------
lms/djangoapps/grades/subsection_grade.py 125 38 70% 47-51, 57, 80-81, 85, 89, 99, 109, 113, [...]
Use this command to generate a coverage report (after previously running ``pytest``)::
coverage report
The above command looks for a test coverage data file in ``reports/.coverage`` - this file will
contain coverage data from your last run of ``pytest``. Coverage data is recorded for whichever
paths you specified in your ``--cov`` option, e.g.::
--cov=. # will track coverage for the entire project
--cov=path.to.your.module # will track coverage only for "module"
Use this command to generate an HTML report::
coverage html
The report is then saved in reports/common/lib/xmodule/cover/index.html
To run tests for stub servers, for example for `YouTube stub server`_, you can
run one of these commands::
paver test_system -s cms -t common/djangoapps/terrain/stubs/tests/test_youtube_stub.py
pytest common/djangoapps/terrain/stubs/tests/test_youtube_stub.py
.. _YouTube stub server: https://github.com/edx/edx-platform/blob/master/common/djangoapps/terrain/stubs/tests/test_youtube_stub.py
.. _the pdb documentation: http://docs.python.org/library/pdb.html
Very handy: if you pass the ``--pdb`` flag to a paver test function, or
uncomment the ``pdb=1`` line in ``setup.cfg``, the test runner will drop you
into pdb on error. This lets you go up and down the stack and see what the
values of the variables are. Check out `the pdb documentation`_. Note that
this only works if you aren't collecting coverage statistics (pdb and
coverage.py use the same mechanism to trace code execution).
Use this command to put a temporary debugging breakpoint in a test.
If you check this in, your tests will hang on jenkins::
import pdb; pdb.set_trace()
Note: More on the ``--failed`` functionality:
* In order to use this, you must run the tests first. If you haven't already
run the tests, or if no tests failed in the previous run, then using the
``--failed`` switch will result in **all** of the tests being run. See more
about this in the `pytest documentation
<https://docs.pytest.org/en/latest/cache.html>`__.
* Note that ``paver test_python`` calls pytest separately for cms and lms.
This means that if tests failed only in lms on the previous run, then calling
``paver test_python --failed`` will run **all of the tests for cms** in
addition to the previously failing lms tests. If you want it to run only the
failing tests for lms or cms, use the ``paver test_system -s lms --failed``
or ``paver test_system -s cms --failed`` commands.
Debugging Unittest Flakiness
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As we move over to running our unittests with Jenkins Pipelines and pytest-xdist,
there are new ways for tests to flake, which can sometimes be difficult to debug.
If you run into flakiness, check (and feel free to contribute to) this
`confluence document <https://openedx.atlassian.net/wiki/spaces/TE/pages/884998163/Debugging+test+failures+with+pytest-xdist>`__ for help.
Running Javascript Unit Tests
-----------------------------
We use Jasmine to run JavaScript unit tests. To run all the JavaScript
tests::
paver test_js
To run a specific set of JavaScript tests and print the results to the
console, run these commands::
paver test_js_run -s lms
paver test_js_run -s cms
paver test_js_run -s cms-squire
paver test_js_run -s xmodule
paver test_js_run -s common
paver test_js_run -s common-requirejs
To run JavaScript tests in a browser, run these commands::
paver test_js_dev -s lms
paver test_js_dev -s cms
paver test_js_dev -s cms-squire
paver test_js_dev -s xmodule
paver test_js_dev -s common
paver test_js_dev -s common-requirejs
To debug these tests on devstack in a local browser:
* first run the appropriate test_js_dev command from above which will open a browser using XQuartz
* open http://edx.devstack.lms:19876/debug.html in your host system's browser of choice
* this will run all the tests and show you the results including details of any failures
* you can click on an individually failing test and/or suite to re-run it by itself
* you can now use the browser's developer tools to debug as you would any other JavaScript code
Note: the port is also output to the console that you ran the tests from if you find that easier.
These paver commands call through to Karma. For more
info, see `karma-runner.github.io <https://karma-runner.github.io/>`__.
Running Bok Choy Acceptance Tests
---------------------------------
We use `Bok Choy`_ for acceptance testing. Bok Choy is a UI-level acceptance
test framework for writing robust `Selenium`_ tests in `Python`_. Bok Choy
makes your acceptance tests reliable and maintainable by utilizing the Page
Object and Promise design patterns.
**Prerequisites**:
These prerequisites are all automatically installed and available in
`Devstack`_, the supported development enviornment for the Open edX platform.
* Chromedriver and Chrome
* Mongo
* Memcache
* mySQL
To run all the bok choy acceptance tests run this command::
paver test_bokchoy
Once the database has been set up and the static files collected, you
can use the 'fast' option to skip those tasks. This option can also be
used with any of the test specs below::
paver test_bokchoy --fasttest
For example to run a single test, specify the name of the test file::
paver test_bokchoy -t lms/test_lms.py
Notice the test file location is relative to
common/test/acceptance/tests. This is another example::
paver test_bokchoy -t studio/test_studio_bad_data.py
To run a single test faster by not repeating setup tasks use the ``--fasttest`` option::
paver test_bokchoy -t studio/test_studio_bad_data.py --fasttest
To test only a certain feature, specify the file and the testcase class::
paver test_bokchoy -t studio/test_studio_bad_data.py::BadComponentTest
To execute only a certain test case, specify the file name, class, and
test case method::
paver test_bokchoy -t lms/test_lms.py::RegistrationTest::test_register
During acceptance test execution, log files and also screenshots of
failed tests are captured in test\_root/log.
Use this command to put a temporary debugging breakpoint in a test.
If you check this in, your tests will hang on jenkins::
import pdb; pdb.set_trace()
By default, all bokchoy tests are run with the 'split' ModuleStore. To
override the modulestore that is used, use the default\_store option.
The currently supported stores are: 'split'
(xmodule.modulestore.split\_mongo.split\_draft.DraftVersioningModuleStore)
and 'draft' (xmodule.modulestore.mongo.DraftMongoModuleStore). This is an example
for the 'draft' store::
paver test_bokchoy --default_store='draft'
Running Bok Choy Accessibility Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We use Bok Choy for `automated accessibility testing`_. Bok Choy, a UI-level
acceptance test framework for writing robust `Selenium`_ tests in `Python`_,
includes the ability to perform accessibility audits on web pages using `Google
Accessibility Developer Tools`_ or `Deque's aXe Core`_. For more details about
how to write accessibility tests, please read the `Bok Choy documentation`_ and
the `Automated Accessibility Tests`_ Open edX Confluence page.
.. _automated accessibility testing: https://bok-choy.readthedocs.org/en/latest/accessibility.html
.. _Selenium: http://docs.seleniumhq.org/
.. _Python: https://www.python.org/
.. _Google Accessibility Developer Tools: https://github.com/GoogleChrome/accessibility-developer-tools/
.. _Deque's aXe Core: https://github.com/dequelabs/axe-core/
.. _Bok Choy documentation: https://bok-choy.readthedocs.org/en/latest/accessibility.html
.. _Automated Accessibility Tests: https://openedx.atlassian.net/wiki/display/TE/Automated+Accessibility+Tests
**Prerequisites**:
These prerequisites are all automatically installed and available in
`Devstack`_ (since the Cypress release), the supported development environment
for the Open edX platform.
.. _Devstack: https://github.com/edx/configuration/wiki/edX-Developer-Stack
* Mongo
* Memcache
* mySQL
To run all the bok choy accessibility tests use this command::
paver test_a11y
To run specific tests, use the ``-t`` flag to specify a pytest-style test spec
relative to the ``common/test/acceptance/tests`` directory. This is an example for it::
paver test_a11y -t lms/test_lms_dashboard.py::LmsDashboardA11yTest::test_dashboard_course_listings_a11y
**Coverage**:
To generate the coverage report for the views run during accessibility tests::
paver a11y_coverage
Note that this coverage report is just a guideline to find areas that
are missing tests. If the view isn't 'covered', there definitely
isn't a test for it. If it is 'covered', we are loading that page
during the tests but not necessarily calling ``page.a11y_audit.check_for_accessibility_errors`` on it.
Options for Faster Development Cycles in Bok-Choy Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following are ways in which a developer could shorten the development
cycle for faster feedback. The options below can often be used together.
**Multiprocessing Mode**
Bok-choy tests can be threaded using the `-n` switch. Using 2 threads generally
reduces test cycles by 33%. The recommendation is to make sure the
number of threads is no more than the number of processors available. For
example, the Cypress release of devstack is provisioned by default with 2
processors. In that case, to run tests in multiprocess mode::
paver test_bokchoy -n 2
*Caveat*: Not all tests have been designed with multiprocessing in mind; some
testcases (approx 10%) will fail in multiprocess mode for various reasons
(e.g., shared fixtures, unexpected state, etc). If you have tests that fail
in multiprocessing mode, it may be worthwhile to run them in single-stream mode
to understand if you are encountering such a failure. With that noted, this
can speed development for most test classes.
**Leave Your Servers Running**
There are two additional switches available in the `paver test_bokchoy` task.
Used together, they can shorten the cycle between test runs. Similar to above,
there are a handful of tests that won't work with this approach, due to insufficient
teardown and other unmanaged state.
1. Start your servers in one terminal/ssh session::
paver test_bokchoy --serversonly
Note if setup has already been done, you can run::
paver test_bokchoy --serversonly --fasttest
2. Run your tests only in another terminal/ssh session::
paver test_bokchoy --testsonly --fasttest
You must run BOTH `--testsonly` and `--fasttest`.
3. When done, you can kill your servers in the first terminal/ssh session with
Control-C. *Warning*: Only hit Control-C one time so the pytest framework can
properly clean up.
Acceptance Test Techniques
~~~~~~~~~~~~~~~~~~~~~~~~~~
1. **Element existence on the page**: Do not use splinter's built-in browser
methods directly for determining if elements exist. Use the
world.is\_css\_present and world.is\_css\_not\_present wrapper
functions instead. Otherwise errors can arise if checks for the css
are performed before the page finishes loading. Also these wrapper
functions are optimized for the amount of wait time spent in both
cases of positive and negative expectation.
2. **Dealing with alerts**: Chrome can hang on javascripts alerts. If a
javascript alert/prompt/confirmation is expected, use the step 'I
will confirm all alerts', 'I will cancel all alerts' or 'I will anser
all prompts with "(.\*)"' before the step that causes the alert in
order to properly deal with it.
3. **Dealing with stale element reference exceptions**: These exceptions
happen if any part of the page is refreshed in between finding an
element and accessing the element. When possible, use any of the css
functions in common/djangoapps/terrain/ui\_helpers.py as they will
retry the action in case of this exception. If the functionality is
not there, wrap the function with world.retry\_on\_exception. This
function takes in a function and will retry and return the result of
the function if there was an exception.
4. **Scenario Level Constants**: If you want an object to be available for
the entire scenario, it can be stored in world.scenario\_dict. This
object is a dictionary that gets refreshed at the beginning on the
scenario. Currently, the current logged in user and the current
created course are stored under 'COURSE' and 'USER'. This will help
prevent strings from being hard coded so the acceptance tests can
become more flexible.
5. **Internal edX Jenkins considerations**: Acceptance tests are run in
Jenkins as part of the edX development workflow. They are broken into
shards and split across workers. Therefore if you add a new .feature
file, you need to define what shard they should be run in or else
they will not get executed. See someone from TestEng to help you
determine where they should go.
Also, the test results are rolled up in Jenkins for ease of
understanding, with the acceptance tests under the top level of "CMS"
and "LMS" when they follow this convention: name your feature in the
.feature file CMS or LMS with a single period and then no other
periods in the name. The name can contain spaces. E.g. "CMS.Sign Up"
Testing internationalization with dummy translations
----------------------------------------------------
Any text you add to the platform should be internationalized. To generate translations for your new strings, run the following command::
paver i18n_dummy
This command generates dummy translations for each dummy language in the
platform and puts the dummy strings in the appropriate language files.
You can then preview the dummy languages on your local machine and also in your sandbox, if and when you create one.
The dummy language files that are generated during this process can be
found in the following locations::
conf/locale/{LANG_CODE}
There are a few JavaScript files that are generated from this process. You can find those in the following locations::
lms/static/js/i18n/{LANG_CODE}
cms/static/js/i18n/{LANG_CODE}
Do not commit the ``.po``, ``.mo``, ``.js`` files that are generated
in the above locations during the dummy translation process!
Test Coverage and Quality
-------------------------
Viewing Test Coverage
~~~~~~~~~~~~~~~~~~~~~
We currently collect test coverage information for Python
unit/integration tests.
To view test coverage:
1. Run the test suite with this command::
paver test
2. Generate reports with this command::
paver coverage
3. Reports are located in the ``reports`` folder. The command generates
HTML and XML (Cobertura format) reports.
Python Code Style Quality
~~~~~~~~~~~~~~~~~~~~~~~~~
To view Python code style quality (including PEP 8 and pylint violations) run this command::
paver run_quality
More specific options are below.
- These commands run a particular quality report::
paver run_pep8
paver run_pylint
- This command runs a report, and sets it to fail if it exceeds a given number
of violations::
paver run_pep8 --limit=800
- The ``run_quality`` uses the underlying diff-quality tool (which is packaged
with `diff-cover`_). With that, the command can be set to fail if a certain
diff threshold is not met. For example, to cause the process to fail if
quality expectations are less than 100% when compared to master (or in other
words, if style quality is worse than what is already on master)::
paver run_quality --percentage=100
- Note that 'fixme' violations are not counted with run\_quality. To
see all 'TODO' lines, use this command::
paver find_fixme --system=lms
``system`` is an optional argument here. It defaults to
``cms,lms,common``.
.. _diff-cover: https://github.com/Bachmann1234/diff-cover
JavaScript Code Style Quality
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To view JavaScript code style quality run this command::
paver run_eslint
- This command also comes with a ``--limit`` switch, this is an example of that switch::
paver run_eslint --limit=50000
Code Complexity Tools
=====================
Two tools are available for evaluating complexity of edx-platform code:
- `radon <https://radon.readthedocs.org/en/latest/>`__ for Python code
complexity. To obtain complexity, run::
paver run_complexity
- `plato <https://github.com/es-analysis/plato>`__ for JavaScript code
complexity. Several options are available on the command line; see
documentation. Below, the following command will produce an HTML report in a
subdirectory called "jscomplexity"::
plato -q -x common/static/js/vendor/ -t common -e .eslintrc.json -r -d jscomplexity common/static/js/
Other Testing Tips
==================
Connecting to Browser
---------------------
If you want to see the browser being automated for JavaScript or bok-choy tests,
you can connect to the container running it via VNC.
+------------------------+----------------------+
| Browser | VNC connection |
+========================+======================+
| Firefox (Default) | vnc://0.0.0.0:25900 |
+------------------------+----------------------+
| Chrome (via Selenium) | vnc://0.0.0.0:15900 |
+------------------------+----------------------+
On macOS, enter the VNC connection string in Safari to connect via VNC. The VNC
passwords for both browsers are randomly generated and logged at container
startup, and can be found by running ``make vnc-passwords``.
Most tests are run in Firefox by default. To use Chrome for tests that normally
use Firefox instead, prefix the test command with
``SELENIUM_BROWSER=chrome SELENIUM_HOST=edx.devstack.chrome``
Factories
---------
Many tests delegate set-up to a "factory" class. For example, there are
factories for creating courses, problems, and users. This encapsulates
set-up logic from tests.
Factories are often implemented using `FactoryBoy`_.
In general, factories should be located close to the code they use. For
example, the factory for creating problem XML definitions is located in
``common/lib/capa/capa/tests/response_xml_factory.py`` because the
``capa`` package handles problem XML.
.. _FactoryBoy: https://readthedocs.org/projects/factoryboy/
Running Tests on Paver Scripts
------------------------------
To run tests on the scripts that power the various Paver commands, use the following command::
pytest pavelib
Testing using queue servers
---------------------------
When testing problems that use a queue server on AWS (e.g.
sandbox-xqueue.edx.org), you'll need to run your server on your public IP, like so::
./manage.py lms runserver 0.0.0.0:8000
When you connect to the LMS, you need to use the public ip. Use
``ifconfig`` to figure out the number, and connect e.g. to
``http://18.3.4.5:8000/``

BIN
docs/guides/v1_sass_pipeline.webp Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB