Merge branch 'master' into jkarni/docs-merge

Conflicts:
	docs/source/conf.py
This commit is contained in:
Julian Arni
2013-07-30 10:23:35 -04:00
1075 changed files with 42550 additions and 25704 deletions

View File

@@ -29,6 +29,7 @@ Specific Problem Types
course_data_formats/word_cloud/word_cloud.rst
course_data_formats/custom_response.rst
course_data_formats/symbolic_response.rst
course_data_formats/jsinput.rst
Internal Data Formats

View File

@@ -276,34 +276,6 @@ Psychoanalyze
:members:
:show-inheritance:
Simple wiki
===========
.. automodule:: simplewiki
:members:
:show-inheritance:
Models
------
.. automodule:: simplewiki.models
:members:
:show-inheritance:
Views
-----
.. automodule:: simplewiki.views
:members:
:show-inheritance:
Tests
-----
.. automodule:: simplewiki.tests
:members:
:show-inheritance:
Static template view
====================

View File

@@ -109,13 +109,6 @@ Progress
:members:
:show-inheritance:
Schematic
=========
.. automodule:: xmodule.schematic_module
:members:
:show-inheritance:
Sequence
========

View File

@@ -48,14 +48,14 @@ Both the LMS and Studio can be started using the following shortcut tasks
rake lms[cms.dev] # Start LMS to run alongside Studio
rake lms[cms.dev_preview] # Start LMS to run alongside Studio in preview mode
Under the hood, this executes `django-admin.py runserver --pythonpath=$WORKING_DIRECTORY --settings=lms.envs.dev`,
Under the hood, this executes `./manage.py {lms|cms} --settings $ENV runserver`,
which starts a local development server.
Both of these commands take arguments to start the servers in different environments
or with additional options:
# Start the LMS using the test configuration, on port 5000
rake lms[test,5000] # Executes django-admin.py runserver --pythonpath=$WORKING_DIRECTORY --setings=lms.envs.test 5000
rake lms[test,5000] # Executes ./manage.py lms --settings test runserver 5000
*N.B.* You may have to escape the `[` characters, depending on your shell: `rake "lms[test,5000]"`

View File

@@ -58,31 +58,29 @@ In the discussion service, notifications are handled asynchronously using a thir
bundle exec rake jobs:work
## Initialize roles and permissions
## From the edx-platform django app, initialize roles and permissions
To fully test the discussion forum, you might want to act as a moderator or an administrator. Currently, moderators can manage everything in the forum, and administrator can manage everything plus assigning and revoking moderator status of other users.
First make sure that the database is up-to-date:
rake django-admin[syncdb]
rake django-admin[migrate]
rake resetdb
For convenience, add the following environment variables to the terminal (assuming that you're using configuration set lms.envs.dev):
If you have created users in the edx-platform django apps when the comment service was not running, you will need to one-way sync the users into the comment service back end database:
export DJANGO_SETTINGS_MODULE=lms.envs.dev
export PYTHONPATH=.
./manage.py lms sync_user_info
Now initialzie roles and permissions, providing a course id eg.:
Now initialize roles and permissions, providing a course id. See the example below. Note that you do not need to do this for Studio-created courses, as the Studio application does this for you.
django-admin.py seed_permissions_roles "MITx/6.002x/2012_Fall"
./manage.py lms seed_permissions_roles "MITx/6.002x/2012_Fall"
To assign yourself as a moderator, use the following command (assuming your username is "test", and the course id is "MITx/6.002x/2012_Fall"):
django-admin.py assign_role test Moderator "MITx/6.002x/2012_Fall"
./manage.py lms assign_role test Moderator "MITx/6.002x/2012_Fall"
To assign yourself as an administrator, use the following command
django-admin.py assign_role test Administrator "MITx/6.002x/2012_Fall"
./manage.py lms assign_role test Administrator "MITx/6.002x/2012_Fall"
## Some other useful commands
@@ -152,8 +150,8 @@ You can use the following command to launch a console within the service environ
Use the following command to see the roles and permissions of a user in a given course (assuming, again, that the username is "test"):
django-admin.py show_permissions moderator
./manage.py lms show_permissions moderator
You need to make sure that the environment variables are exported. Otherwise you would need to do
django-admin.py show_permissions moderator --settings=lms.envs.dev --pythonpath=.
./manage.py lms show_permissions moderator

View File

@@ -64,6 +64,12 @@ You should be familiar with the following. If you're not, go read some docs...
from a Location object, and the ModuleSystem knows how to render things,
track events, and complain about 404s
- XModules and XModuleDescriptors are uniquely identified by a Location object, encoding the organization, course, category, name, and possibly revision of the module.
- XModule initialization: XModules are instantiated by the `XModuleDescriptor.xmodule` method, and given a ModuleSystem, the descriptor which instantiated it, and their relevant model data.
- XModuleDescriptor initialization: If an XModuleDescriptor is loaded from an XML-based course, the XML data is passed into its `from_xml` method, which is responsible for instantiating a descriptor with the correct attributes. If it's in Mongo, the descriptor is instantiated directly. The module's attributes will be present in the `model_data` dict.
- `course.xml` format. We use python setuptools to connect supported tags with the descriptors that handle them. See `common/lib/xmodule/setup.py`. There are checking and validation tools in `common/validate`.
- the xml import+export functionality is in `xml_module.py:XmlDescriptor`, which is a mixin class that's used by the actual descriptor classes.
@@ -122,11 +128,6 @@ In production, the django `collectstatic` command recompiles everything and puts
In development, we don't use collectstatic, instead accessing the files in place. The auto-compilation is run via `common/djangoapps/pipeline_mako/templates/static_content.html`. Details: templates include `<%namespace name='static' file='static_content.html'/>`, then something like `<%static:css group='application'/>` to call the functions in `common/djangoapps/pipeline_mako/__init__.py`, which call the `django-pipeline` compilers.
### Other modules
- Wiki -- in `lms/djangoapps/simplewiki`. Has some markdown extentions for embedding circuits, videos, etc.
## Testing
See `testing.md`.

View File

@@ -175,6 +175,8 @@ Use `rake -T` to get a list of all available subsystems
**Troubleshooting**: If you get an error message while running the `rake` task,
try running `bundle install` to install the required ruby gems.
Unit tests can be run in parallel to each other and while acceptance tests are running
### Running Acceptance Tests
We use [Lettuce](http://lettuce.it/) for acceptance testing.
@@ -203,6 +205,10 @@ To start the debugger on failure, add the `--pdb` option:
To run tests faster by not collecting static files, you can use
`rake fasttest_acceptance_lms` and `rake fasttest_acceptance_cms`.
Acceptance tests will run on a randomized port and can be run in the background of rake cms and lms or unit tests.
To specify the port, change the LETTUCE_SERVER_PORT constant in cms/envs/acceptance.py and lms/envs/acceptance.py
as well as the port listed in cms/djangoapps/contentstore/feature/upload.py
**Note**: The acceptance tests can *not* currently run in parallel.
## Viewing Test Coverage
@@ -227,6 +233,33 @@ generates HTML and XML (Cobertura format) reports.
When testing problems that use a queue server on AWS (e.g. sandbox-xqueue.edx.org), you'll need to run your server on your public IP, like so.
`django-admin.py runserver --settings=lms.envs.dev --pythonpath=. 0.0.0.0:8000`
`./manage.py lms runserver 0.0.0.0:8000`
When you connect to the LMS, you need to use the public ip. Use `ifconfig` to figure out the number, and connect e.g. to `http://18.3.4.5:8000/`
## Acceptance Test Techniques
1. Do not assert not if possible for css. Use world.is_css_present and is_css_not_present
Errors can arise if checks for the css are performed before the page finishes loading.
To get around this, there are functions that will wait a period of time for the css to appear
before returning and return immediately if they are there. There is a reverse to this function as well.
It will wait for the css to not appear and returns if it isn't there.
All css functions can utilize this timeout to ensure that the page is fully loaded
2. Dealing with alerts
Chrome can hang on javascripts alerts. If a javascript alert/prompt/confirmation is expected, use the step
'I will confirm all alerts', 'I will cancel all alerts' or 'I will anser all prompts with "(.*)"' before the step
that causes the alert in order to properly deal with it.
3. Dealing with stale element reference exceptions
These exceptions happen if any part of the page is refreshed in between finding an element and accessing the element.
When possible, use any of the css functions in common/djangoapps/terrain/ui_helpers.py as they will retry the action
in case of this exception. If the functionality is not there, wrap the function with world.retry_on_exception. This function takes in a function and will retry and return the result of the function if there was an exception
4. Scenario Level Constants
If you want an object to be available for the entire scenario, it can be stored in world.scenario_dict. This object
is a dictionary that gets refreshed at the beginning on the scenario. Currently, the current logged in user and the current created course are stored under 'COURSE' and 'USER'. This will help prevent strings from being hard coded so the
acceptance tests can become more flexible.

338
docs/source/conf.py Normal file
View File

@@ -0,0 +1,338 @@
# -*- coding: utf-8 -*-
"""
EdX documentation build configuration file
"""
#pylint: disable=C0103
#pylint: disable=W0622
#pylint: disable=W0212
#pylint: disable=W0613
import sys
import os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# sys.path.insert(0, os.path.abspath('.'))
root = os.path.abspath('../..')
sys.path.append(root)
sys.path.append(os.path.join(root, "common/djangoapps"))
sys.path.append(os.path.join(root, "common/lib"))
sys.path.append(os.path.join(root, "common/lib/sandbox-packages"))
sys.path.append(os.path.join(root, "lms/djangoapps"))
sys.path.append(os.path.join(root, "lms/lib"))
sys.path.append(os.path.join(root, "cms/djangoapps"))
sys.path.append(os.path.join(root, "cms/lib"))
# django configuration - careful here
os.environ['DJANGO_SETTINGS_MODULE'] = 'lms.envs.test'
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.coverage',
'sphinx.ext.pngmath', 'sphinx.ext.mathjax', 'sphinx.ext.viewcode']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'EdX Dev Data'
copyright = u'2012-13, EdX team'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '0.2'
# The full version, including alpha/beta/rc tags.
release = '0.2'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['build']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'sphinxdoc'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'edXDocs'
# -- Options for LaTeX output --------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'edXDocs.tex', u'EdX Dev Data Documentation',
u'EdX Team', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output --------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'edxdocs', u'EdX Dev Data Documentation',
[u'EdX Team'], 1)
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output ------------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'EdXDocs', u'EdX Dev Data Documentation',
u'EdX Team', 'EdXDocs', 'One line description of project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'http://docs.python.org/': None}
# from http://djangosnippets.org/snippets/2533/
# autogenerate models definitions
import inspect
from django.utils.html import strip_tags
from django.utils.encoding import force_unicode
def process_docstring(app, what, name, obj, options, lines):
"""Autodoc django models"""
# This causes import errors if left outside the function
from django.db import models
# If you want extract docs from django forms:
# from django import forms
# from django.forms.models import BaseInlineFormSet
# Only look at objects that inherit from Django's base MODEL class
if inspect.isclass(obj) and issubclass(obj, models.Model):
# Grab the field list from the meta class
fields = obj._meta._fields()
for field in fields:
# Decode and strip any html out of the field's help text
help_text = strip_tags(force_unicode(field.help_text))
# Decode and capitalize the verbose name, for use if there isn't
# any help text
verbose_name = force_unicode(field.verbose_name).capitalize()
if help_text:
# Add the model field to the end of the docstring as a param
# using the help text as the description
lines.append(u':param %s: %s' % (field.attname, help_text))
else:
# Add the model field to the end of the docstring as a param
# using the verbose name as the description
lines.append(u':param %s: %s' % (field.attname, verbose_name))
# Add the field's type to the docstring
lines.append(u':type %s: %s' % (field.attname, type(field).__name__))
# Only look at objects that inherit from Django's base FORM class
# elif (inspect.isclass(obj) and issubclass(obj, forms.ModelForm) or issubclass(obj, forms.ModelForm) or issubclass(obj, BaseInlineFormSet)):
# pass
# # Grab the field list from the meta class
# import ipdb; ipdb.set_trace()
# fields = obj._meta._fields()
# import ipdb; ipdb.set_trace()
# for field in fields:
# import ipdb; ipdb.set_trace()
# # Decode and strip any html out of the field's help text
# help_text = strip_tags(force_unicode(field.help_text))
# # Decode and capitalize the verbose name, for use if there isn't
# # any help text
# verbose_name = force_unicode(field.verbose_name).capitalize()
# if help_text:
# # Add the model field to the end of the docstring as a param
# # using the help text as the description
# lines.append(u':param %s: %s' % (field.attname, help_text))
# else:
# # Add the model field to the end of the docstring as a param
# # using the verbose name as the description
# lines.append(u':param %s: %s' % (field.attname, verbose_name))
# # Add the field's type to the docstring
# lines.append(u':type %s: %s' % (field.attname, type(field).__name__))
# Return the extended docstring
return lines
def setup(app):
"""Setup docsting processors"""
#Register the docstring processor with sphinx
app.connect('autodoc-process-docstring', process_docstring)

658
docs/source/persistence.rst Normal file
View File

@@ -0,0 +1,658 @@
This document describes the split mongostore representation which
separates course structure from content where each course run can have
its own structure. It does not describe the original mongostore
representation which combined structure and content and used the key
to distinguish draft from published elements.
This document does not describe mongo nor its operations. See
`http://www.mongodb.org/`_ for information on Mongo.
Product Goals and Discussion
----------------------------
(Mark Chang)
This work was instigated by the studio team's need to correctly do
metadata inheritance. As we moved from an on-startup load of the
courseware, the system was able to inflate and perform an inheritance
calculation step such that the intended properties of children could
be set through inheritance. While not strictly a requirement from the
studio authoring approach, where inheritance really rears its head is
on import of existing courseware that was designed assuming
inheritance.
A short term patch was applied that allowed inheritance to act
correctly, but it was felt that it was insufficient and this would be
an opportunity to make a more clean datastore representation. After
much difficulty with how draft objects would work, Calen Pennington
worked through a split data store model ala FAT filesystem (Mark's
metaphor, not Cale's) to split the structure from the content. The
goal would be a sea of content documents that would not know about the
structure they were utilized within. Cale began the work and handed it
off to Don Mitchell.
In the interim, great discussion was had at the Architect's Council
that firmed up the design and strategy for implementation, adding
great richness and completeness to the new data structure.
The immediate
needs are two, and only two.
#. functioning metadata inheritance
#. good groundwork for versioning
While the discussions of the atomic unit of courseware available for
sharing, how these are shared, and how they refer back to the parent
definition are all valuable, they will not be built in the near term. I
understand and expect there to be many refactorings, improvements, and
migrations in the future.
I fully anticipate much more detail to be uncovered even in this first
thin implementation. When that happens, we will need as much advice
from those watching this page to make sure we move in the right
direction. We also must have the right design artifacts to document
where we stand relative to the overall design that has loftier goals.
Representation
--------------
The xmodule collections:
+ `modulestore.active_versions`: this collection maps the org, course,
and run to the current draft and published versions of the course.
+ `modulestore.structures`: this collection has one entry per course
run and one for the template.
+ `modulestore.definitions`: this collection has one entry per
"module" or "block" version.
modulestore.active_versions: 2 simple maps for dereferencing the
correct course from the structures collection. Every course run will
have a draft version. Not every course run will have a published
version. No course run will have more than one of each of these.
::
{ '_id' : uniqueid,
'versions' : { <versionName> : versionGuid, ..}
'creator' : user_id,
'created' : date (native mongo rep)
}
::
+ `id` is a unique id for finding this course run. It's a
location-reference string, like 'edu.mit.eng.eecs.6002x.industry.spring2013'.
+ `versions`: These are references to `modulestore.structures`. A
location-reference like
`edu.mit.eng.eecs.6002x.industry.spring2013;draft` refers to the value
associated with `draft` for this document.
+ `versionName` is `draft`, `published`, or another user-defined
string.
+ `versionGuid` is a system generated globally unique id (hash). It
points to the entry in `modulestore.structures` ` `
`draftVersion`: the design will try to generate a new draft version
for each change to the course object: that is, for each move,
deletion, node creation, or metadata change. Cloning a course
(creating a new run of a course or such) will create a new entry in
this table with just a `draftVersion` and will cause a copy of the
corresponding entry in `modulestore.structures`. The entry in
`structures` will point to its version parent in the source course.
modulestore.structures : the entries in this collection follow this
definition:
::
{ '_id' : course_guid,
'blocks' :
{ block_guid : // the guid is an arbitrary id to represent this node in the course tree
{ 'children' : [ block_guid* ],
'metadata' : { property map },
'definition' : definition_guid,
'category' : 'section' | 'sequence' | ... }
::
...// more guids
::
},
'root' : block_guid,
'original' : course_guid, // the first version of this course from which all others were derived
'previous' : course_guid | null, // the previous revision of this course (null if this is the original)
'version_entry' : uniqueid, // from the active_versions collection
'creator' : user_idÂ
}
+ `blocks`: each block is a node in the course such as the course, a
section, a subsection, a unit, or a component. The block ids remain
the same over edits (they're not versioned).
+ `root`: the true top of the course. Not all nodes without parents
are truly roots. Some are orphans.
+ `course_guid, block_guid, definition_guid` are not those specific
strings but instead some system generated globally unique id.
+ The one which gets passed around and pointed to by urls is the
`block_guid`; so, it will be the one the system ensures is readable.
Unlike the other guids, this one stays the same over revisions and can
even be the same between course runs (although the course run
contextualizes it to distinguish its instantiated version).
+ `definition` points to the specific revision of the given element in
`modulestore.definitions` which this version of the course includes.
+ `children` lists the block_guids which are the children of this node
in the course tree. It's an error if the guid in the `children` list
does not occur in the `blocks` dictionary.
+ `metadata` is the node's explicitly defined metadata some of which
may be inherited by its children
For debugging purposes, there may be value in adding a courseId field
(org, course, run) for use via db browsers.
modulestore.definitions : the data associated with each version of
each node in the structures. Many courses may point to the same
definition or may point to different versions derived from the same
original definition.
::
{ '_id' : guid,
'data' : ..,
'default_settings' : {'display_name':..,..}, // a starting point for new uses of this definition
'category' : xblocktype, // the xmodule/xblock type such as course, problem, html, video, about
'original' : guid, // the first kept version of this definition from which all others were derived
'previous' : guid | null, // the previous revision of this definition (null if this is the original)
'creator' : user_id // the id of whomever pressed the draft or publish button
}
+ `_id`: a guid to uniquely identify the definition.
+ `data` is the payload used by the xmodule and following the
xmodule's data representation.
+ `category` is the xmodule type and used to figure out which xmodule
to instantiate.
There may be some debugging value to adding a courseId field, but it
may also be misleading if the element is used in more than one course.
Templates
~~~~~~~~~
(I'm refactoring templates quite a bit from their representation prior
to this design)
All field defaults will be defined through the xblock field.default
mechanism. Templates, otoh, are for representing optional boilerplate
usually for examples such as a multiple-choice problem or a video
component with the fields all filled in. Templates are stored in yaml
files which provide a template name, sorting and filtering information
(e.g., requires advanced editor v allows simple editor), and then
field: value pairs for setting xblocks' fields upon template
selection.
Most of the pre-existing templates including all of the 'empty' ones
will go away. The ones which will stay are the ones truly just giving
examples or starting points for variants. This change will require
that the template choice code provide a default 'blank' choice to the
user which just instantiates the model w/ its defaults versus a choice
of the boilerplates. The client can therefore populate its own model
of the xblock and then send a create-item request to the server when
the user says he/she's ready to save it.
Import/export
~~~~~~~~~~~~~
Export should allow the user to select the version of the course to
export which can be any of the draft or published versions. At a
minimum, the user should choose between draft or published.
Import should import the course as a draft course regardless of
whether it was exported as a published or draft one, I believe. If
there's already a draft for the same course, in the best of all
worlds, it would have the guid to see if the guid exists in the
structures collection, and, if so, just make that the current
draftVersion (don't do any actual data changes). If there's no guid or
the guid doesn't exist in the structures collection, then we'll need
to work out the logic for how to decide what definitions to create v
update v point to.
Course ID
~~~~~~~~~
Currently, we use a triple to identify a run of a course. The triple
is organization, course name, and run identity (e.g., 2013Q1). The
system does not care what the id consists of only that it uniquely
identify an edition of the course. The system uses this id to organize
the course composition and find the course elements. It distinguishes
between a current being-edited version (aka, draft) and publicly
viewable version (published). Not every course has a published
version, but every course will have a draft version. The application
specifies whether it wants the draft or published version. This system
allows the application to easily switch between the 2; however, it
will have a configuration in which it's impossible to access the draft
so that we can add access optimizations and extraction filtering later
if needed.
Location
~~~~~~~~
The purpose of `Location` is to identify content. That is, to be able
to locate content by providing sufficient addressing. The `Location`
object is ubiquitous throughout the current code and thus will be
difficult to adapt and make more flexible. Right now, it's a very
simple `namedtuple` and a lot of code presumes this. This refactoring
generalizes and subclasses it to handle various addressing schemes and
remove direct manipulations.
Our code needs to locate several types of things and should probably
use several different types of locators for these. These are the types
of things we need to address. Some of these can be the same as others,
but I wanted to lay them out fairly fine grained here before proposing
my distinctions:
#. Courses: an object representing a course as an offering but not any
of its content. Used for dashboards and other such navigators. These
may specify a version or merely reference the idea of the course's
existence.
#. Course structures: the names (and other metadata), `Locations`, and
children pointers but not definitions for all the blocks in a course
or a subtree of a course. Our applications often display contextual,
outline, or other such structural information which do not need to
include definitions but need to show display names, graded as, and
other status info. This document's design makes fetching these a
single document fetch; however, if it has to fetch the full course, it
will require far more work (getting all definitions too) than the apps
need.
#. Blocks (uses of definitions within a version of a course including
metadata, pointers to children, and type specific content)
#. Definitions: use independent definitions of content without
metadata (and currently w/o pointers to children).
#. Version trees Fetching the time history portrayal of a definition,
course, or block including branching.
#. Collections of courses, definitions, or blocks matching some
partial descriptors (e.g., all courses for org x, all definitions of
type foo, all blocks in course y of type x, all currently accessible
courses (published with startdate < today and enddate > today)).
#. Fetching of courses, blocks, or definitions via "human readable"
urls.
#. (partial descriptors) may suffice for this as human readable
does not guarantee uniqueness.
Some of these differ not so much in how to address them but in what
should be returned. The content should be up to the functions not the
addressing scheme. So, I think the addressable things are:
#. Course as in #1 above: usually a specific offering of a course.
Often used as a context for the other queries.
#. Blocks (aka usages) as in #3 above: a specific block contextualized
in a course
#. Definitions (#4): a specific definition
#. Collections of courses, blocks within a specific course, or
definitions matching a partial descriptor
Course locator (course_loc)
```````````````````````````
There are 3 ways to locate a course:
#. By its unique id in the `active_versions` collection with an
implied or specified selection of draft or published version.
#. By its unique id in the `structures` collection.
Block locator (block_loc)
`````````````````````````
A block locator finds a specific node in a specific version of a
course. Thus, it needs a course locator plus a `usage_id`.
Definition locator (definition_loc)
```````````````````````````````````
Just a `guid`.
Partial descriptor collections locators (partial)
`````````````````````````````````````````````````
In the most general case, and to simplify implementation, these can be
any payload passable to mongo for doing the lookup. The specification
of which collection to look into can be implied by which lookup
function your code calls (get_courses, get_blocks, get_definitions) or
we could add it as another property. For now, I will leave this as
merely a search string. Thus, to find all courses for org = mitx,
`{"org": "mitx"}`. To find all blocks in a course whose display name
contains "circuit example", call `get_blocks` with the course locator
plus `{"metadata.display_name" : /circuit example/i}` (the i makes it
case insensitive and is just an example). To find if a definition is
used in a course, call get_blocks with the course locator plus
`{definition : definition_guid}`. Note, this looks for a specific
version of the definition. If you wanted to see if it used any of a
set of versions, use `{definition : {"$in" : [definition_guid*]}}`
i4x locator
```````````
To support existing xml based courses and any urls, we need to
support i4x locators. These are tuples of `(org course category id
['draft'])`. The trouble with these is that they don't uniquely
identify a course run from which to dereference the element. There's
also no requirement that `id` have any uniqueness outside the scope of
the other elements. There's some debate as to whether these address
blocks or definitions. To mean, they seem to address blocks; however,
in the current system there is no distinction between blocks and
definitions; so, either could be argued.
This version will define an `i4x_location` class for representing
these and using them for xml based courses if necessary.
Current code munges strings to make them 'acceptable' by replacing
'illegal' chars with underscores. I'd like to suggest leaving strings
as is and using url escaping to make acceptable urls. As to making
human readable names from display strings, that should be the
responsibility of the naming module not the Location representation,
imo.
Use cases (expository)
~~~~~~~~~~~~~~~~~~~~~~
There's a section below walking through a specific use case. This one
just tries to review potential functionality.
Inheritance
```````````
Our system has the notion of policies which should control the
behavior of whole courses or subtrees within courses. Such policies
include graceperiods, discussion forum controls, dates, whether to
show answers, how to randomize, etc. It's important that the course
authors' intent propagates to all relevant course sections. The
desired behavior is that (some? all?) metadata attributes on modules
flow down to all children unless overridden.
This design addresses inheritance by making course structure and
metadata separate from content thus enabling a single or small number
of db queries to get these and then compute the inheritance.
Separating editing from live production
```````````````````````````````````````
Course authors should be able to make changes in isolation from
production and then push out consistent chunks of changes for all
students to see as atomic and consistent. The current system allows
authors to change text and content without affecting production but
not metadata nor course structure. This design separates all changes
from production until pushed.
Sharing of content, part 1
``````````````````````````
Authors want to share content between course runs and even between
different courses. The current system requires copying all such
content and losing the providence information which could be used to
take advantage of other peoples' changes. This design allows multiple
courses and multiple places within a course to point to the same
definitions and thus potentially, at some day, see other changes to
the content.
Sharing of content, part 2: course structure
````````````````````````````````````````````
Because courses structures are separate from their identities, courses
can share structure and track changes in the same way as definitions.
That is, a new course run can point to an existing course instance
with its version history and then branch it from there.
Sharing of content, part 3: modules
```````````````````````````````````
Suppose a course includes a soldering tutorial (or a required lab
safety lesson). Other courses want to use the same tutorial and
possibly allow the student to skip it if the student succeeded at it
in another course. As the tutorial updates, other courses may want to
track the updates or choose to move to the updates without having to
copy the modules from the module's authoritative parent course.
This design enables sharing of composed modules but it does not track
the revisions of those modules separately from their courses. It does
not adequately address this but may be extendible enough to do so.
That is, we could represent these shared units as separate "courses"
and allow ids in block.children[] to point to courses as well as other
blocks in the same course.
We should decide on the behaviors we want. Such as, some times the
student has to repeat the content or the student never has to repeat
it or? progress should be tracked by the owning course or as a stand
alone minicourse type element? Because it's a safety lesson, all
courses should track the current published head and not have their own
heads or they should choose when to promote the head?
Are these shared elements rare and large grained enough to make the
indirection not expensive or will it result in devolving to the
current one entry per module design for deducing course structure?
Functional differences from existing modulestore:
-------------------------------------------------
+ Courses and definitions support trees of versions knowing from where
they were derived. For now, I will not implement the server functions
for retrieving and manipulating these version trees and will leave
those for a future effort. I will only implement functions which
extend the trees.
+ Changes to course structure don't immediately affect production:
note, we need to figure out the granularity of the user's publish
behavior for pushing out these actions. That is, do they publish a
whole subtree which may include new children in order to make these
effective, do they publish all structural (deletion, move) changes
under a subtree but not insertions as an action, do they publish each
action individually, or what? How do they know that any of these are
not yet published? Do we have phantom placeholders for deleted nodes
w/ "publish deletion" buttons?
+ Element deletion
+ Element move
+ metadata changes
+ No location objects used as ids! This implementation will use guids
instead. There's a reasonable objection to guids as being too ugly,
long, and indecipherable. I will check mongy, pymongo, and python guid
generation mechanisms to find out if there's a way to make ones which
include a prepended string (such as course and run or an explicitly
stated prepend string) and minimize guid length (e.g., by using
sequential serial # from a global or local pool).
Use case walkthrough:
---------------------
Simple course creation with no precursor course: Note, this shows that
publishing creates subsets and side copies not in line versions of
nodes.
user db create course for org, course id, run id
active_versions.draftVersion: add entry
definitions: add entry C w/ category = 'course', no data
structures: add entry w/ 1 child C, original = self, no previous,
author = user
add section S copy structures entry, new one points to old as original
and previous
active_versions.draftVersion points to new
definitions: add entry S w/ category = 'section'
structures entry:
+ add S to children of the course block,
+ add S to blocks w/ no children
add subsection T copy structures entry, new one points to old as
original and previous
active_versions.draftVersion points to new
definitions: add entry T w/ category = 'sequential'
structures entry:
+ add T to children of the S block entry,
+ add T to blocks w/ no children
add unit U copy structures entry, new one points to old as original
and previous
active_versions.draftVersion points to new
definitions: add entry U w/ category = 'vertical'
structures entry:
+ add U to children of the T block entry,
+ add U to blocks w/ no children
publish U
create structures entry, new one points to self as original (no
pointer to draft course b/c it's not really a clone)
active_versions.publishedVersion points to new
block: add U, T, S, C pointers with each as respective child
(regardless of other children they may have in draft), and their
metadata
add units V, W, X under T copy structures entry of the draftVersion,
new one points to old as original and previous
active_versions.draftVersion points to new
definitions: add entries V, W, X w/ category = 'vertical'
structures entry:
+ add V, W, X to children of the T block entry,
+ add V, W, X to blocks w/ no children
edit U copy structures entry, new one points to old as original and
previous
active_versions.draftVersion points to new
definitions: copy entry U to U_2 w/ updates, U_2 points to U as
original and previous
structures entry:
+ replace U w/ U_2 in children of the T block entry,
+ copy entry U in blocks to entry U_2 and remove U
add subsection Z under S copy structures entry, new one points to old
as original and previous
active_versions.draftVersion points to new
definitions: add entry Z w/ category = 'sequential'
structures entry:
+ add Z to children of the S block entry,
+ add Z to blocks w/ no children
edit S's name (metadata) copy structures entry, new one points to old
as original and previous
active_versions.draftVersion points to new
structures entry: update S's metadata w/ new name publish U, V copy
publishedCourse structures entry, new one points to old published as
original and previous
active_versions.publishedVersion points to new
block: update T to point to new U & V and not old U
Note: does not update S's name publish C copy publishedCourse
structures entry, new one points to old published as original and
previous
active_versions.publishedVersion points to new
blocks: note that C child S == published(S) but metadata !=, update
metadata
note that S has unpublished children: publish them (recurse on this)
note that Z is unpublished: add pointer to blocks and children of S
note that W, X unpublished: add to blocks, add to children of T edit C
metadata (e.g., graceperiod) copy draft structures entry, new one
points to old as original and previous
active_versions.draftVersion points to new
structures entry: update C's metadata add Y under Z ... publish C's
metadata change copy publishedCourse structures entry, new one points
to old published as original and previous
active_versions.publishedVersion points to new
blocks: update C's metadata
Note: no copying of Y or any other changes to published move X under Z
copy draft structures entry, new one points to old as original and
previous
active_versions.draftVersion points to new
structures entry: remove X from T's children and add to Z's
Note: making it persistently clear to the user that X still exists
under T in the published version will be crucial delete W copy draft
structures entry, new one points to old as original and previous
active_versions.draftVersion points to new
structures entry: remove W from T's children and remove W from blocks
Note: no actual deletion of W, just no longer reachable w/in the draft
course, but still in published; so, need to keep user aware of that.
publish Z Note: the interesting thing here is that X cannot occur
under both Z and T, but the user's not publishing T, here's where
having a consistent definition of original may help. If the original
of a new element == original of an existing, then it's an update?
copy publishedCourse entry...
definitions: add Y, copy/update Z, X if either have any data changes
(they don't)
blocks: remove X from T's children and add to Z's, add Y to Z, add Y
publish deletion of W copy publishedCourse entry...
structures entry: remove W from T's children and remove W from blocks
Conflict detection:
Need a scenario where 2 authors make edits to different parts of
course, to parts while parents being moved, while parents being
deleted, to same parts, ...
.. _http://www.mongodb.org/: http://www.mongodb.org/