* Consolidates and renames the runtime used as a base for all the others:
* Before: `xmodule.x_module:DescriptorSystem` and
`xmodule.mako_block:MakoDescriptorSystem`.
* After: `xmodule.x_module:ModuleStoreRuntime`.
* Co-locates and renames the runtimes for importing course OLX:
* Before: `xmodule.x_module:XMLParsingSystem` and
`xmodule.modulestore.xml:ImportSystem`.
* After: `xmodule.modulestore.xml:XMLParsingModuleStoreRuntime` and
`xmodule.modulestore.xml:XMLImportingModuleStoreRuntime`.
* Note: I would have liked to consolidate these, but it would have
involved nontrivial test refactoring.
* Renames the stub Old Mongo runtime:
* Before: `xmodule.modulestore.mongo.base:CachingDescriptorSystem`.
* After: `xmodule.modulestore.mongo.base:OldModuleStoreRuntime`.
* Renames the Split Mongo runtime, the which is what runs courses in LMS and CMS:
* Before: `xmodule.modulestore.split_mongo.caching_descriptor_system:CachingDescriptorSystem`.
* After: `xmodule.modulestore.split_mongo.runtime:SplitModuleStoreRuntime`.
* Renames some of the dummy runtimes used only in unit tests.
BREAKING CHANGE: This removes the following deprecated shims from the runtime:
`replace_urls`, `replace_course_urls`, `replace_jump_to_id_urls`. XBlocks need
to use the `replace_urls` service instead.
Deprecates the following attributes from ModuleSystem:
* replace_urls
* replace_course_urls
* replace_jump_to_id_urls
A new ReplaceURLService is created as replacement with a unified replace_urls method
django-not-configured is an error raised by pylint (with
the pylint-django plugin) when it's not correctly configured.
We should not be applying lint amnesty for such a violation.
warnings.py:109 - /edx/app/edxapp/edx-platform/common/djangoapps/static_replace/__init__.py:76: DeprecationWarning: Flags not at the start of the expression '\n (?x) ' (truncated)
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
* fix type mismatch in course_goals migrations
* fix type mismatch in experiments migrations
* fix type mismatch in xblock_django migrations
* fix type mismatch in catalog migrations
* fix type mismatch in static_replace migrations
* fix type mismatch in bulk_email migrations
* fix type mismatch in course_overviews migrations
Update the tests around static_file code to use byte streams instead of
string streams for generating static content. This is a fix to get
these tests passing on python 3.
rewriting them.
In PERF-341, we adjusted the static_replace middleware to try and
exclude static XBlock resource URLs from being interpreted as the marker
URLs used to signify course assets in course content. Since they both
started with /static, this could, and did, cause issues where linking
directly to the assets of an XBlock within, say, one of its templates,
would lead to that link being rewritten and ultimately being incorrect.
The fix attempted to see if the link started with the prefix that all
static XBlock resource URLs start, and if so, it returned them
unmodified.
We incorrectly assumed that our testing captured all cases, and since
we're here, we know that this was wrong. We weren't accounting for cases
when the URLs being generated had the STATIC_URL configuration value
prefixed -- https://example.com/static/xblock/.... -- and so our direct
check of seeing if such a URL started with "/static/xblock" would always
fail, leading to the erroneous rewriting and nonsensical output.
This fix checks if the link either starts with the prefix OR if it
starts with the STATIC_URL value and contains the prefix overall. There
is a small overlap between the STATIC_URL and the prefix we check for,
so an inconsistency could arise down the line if we changed our
STATIC_URL to use a difference base directory, but our tests will at
least catch the issue now.
This version component reflects the "version" of the StaticContent
objects which we cache server-side. If the layout of those objects
changes between releases, errors occur when loading them from cache.
By using a separate version value, which can be incremented on its own
after a change has been made to the StaticContent class, we can avoid
loading older cached content and in turn take advantage of these changes
faster, without needing to intervene operationally.
For XBlocks that used their static.public resources in the rendered
output -- for example, a link to a bundled image -- those URLs would be
treated as course assets using the '/static/' prefix trick, and thus,
rewritten. These rewritten URLs don't work because they aren't course
assets.
When course content authors are creating their courses, we provide them
a special shorthand way of writing URLs that reference assets they have
uploaded to the course. If they uploaded a file called my-lil-pony.mp4,
and they wanted to provide a link to it when users view the course, they
would use /static/my-lil-pony.mp4. This special prefix -- /static/ --
signals to the static_replace middleware that it's a course asset and we
should write /static/my-lil-pony.mp4 to a URL that properly references
the asset based on the course ID, and things like the configured asset
CDN, etc.
Thus, the URL /static/my-lil-pony.mp4 gets turned into something like:
/assets/courseware/<md5hash>/asset-v1:edX+Demo+2016T1+type@asset+block/my-lil-pony.mp4
when viewed in the courseware.
Now, we also serve actual static assets from a prefix of /static/. This
is stuff like our JavaScript and CSS, and the JS/CSS/etc of
XBlocks/XModules. These paths look like:
/static/js/lms-main_vendor.46d6a8c02600.js
or
/static/xblock/resources/xmodule.vertical_block/public/js/vertical_student_view.43727a907769.js
Normally, these paths are caught by nginx, before they reach the LMS,
and are served straight from the filesystem. However, if you were to
have one of these paths in your course content, the static_replace
middleware would see the /static/ at the front and immediately think
it's a course asset, and would dutifully rewrite the URL to something
like:
/assets/courseware/<md5hash>/asset-v1:edX+Demo+2016T1+type@asset+block/static_xblock_resources_xmodule.vertical_block_public_js_vertical_student_view.43727a907769.js
which is not a course asset, and so it will always fail to load.
Long story short, I changed the static_replace middleware to
specifically check to see if the path being matched starts with
/static/xblock/, and if so, it keeps the original instead of rewriting
it.
To be able to more easily deal with same-origin issues, due to some course assets being loaded in an iframe, we're adding the ability to specify file extensions that should be excluded from URL canonicalization. The default value is simply 'html', which means we won't rewrite asset links that point to files ending in .html. Thus, they'll be loaded from the same origin as the parent page, and voila, no SOP issues.
A base URL can now be configured which is, potentially, prepended to an
asset URL. This allows a CDN, or caching server, to front static asset
requests, taking load off of the contentstore and speeding up page load
times.
Asset URL generation respects locked vs unlocked assets, and will not
generate links to locked assets that would traverse a CDN (even though
the authorization component of the contentserver middleware wouldn't
allow those links to work anyways).
