rewriting them.
In PERF-341, we adjusted the static_replace middleware to try and
exclude static XBlock resource URLs from being interpreted as the marker
URLs used to signify course assets in course content. Since they both
started with /static, this could, and did, cause issues where linking
directly to the assets of an XBlock within, say, one of its templates,
would lead to that link being rewritten and ultimately being incorrect.
The fix attempted to see if the link started with the prefix that all
static XBlock resource URLs start, and if so, it returned them
unmodified.
We incorrectly assumed that our testing captured all cases, and since
we're here, we know that this was wrong. We weren't accounting for cases
when the URLs being generated had the STATIC_URL configuration value
prefixed -- https://example.com/static/xblock/.... -- and so our direct
check of seeing if such a URL started with "/static/xblock" would always
fail, leading to the erroneous rewriting and nonsensical output.
This fix checks if the link either starts with the prefix OR if it
starts with the STATIC_URL value and contains the prefix overall. There
is a small overlap between the STATIC_URL and the prefix we check for,
so an inconsistency could arise down the line if we changed our
STATIC_URL to use a difference base directory, but our tests will at
least catch the issue now.
This version component reflects the "version" of the StaticContent
objects which we cache server-side. If the layout of those objects
changes between releases, errors occur when loading them from cache.
By using a separate version value, which can be incremented on its own
after a change has been made to the StaticContent class, we can avoid
loading older cached content and in turn take advantage of these changes
faster, without needing to intervene operationally.
For XBlocks that used their static.public resources in the rendered
output -- for example, a link to a bundled image -- those URLs would be
treated as course assets using the '/static/' prefix trick, and thus,
rewritten. These rewritten URLs don't work because they aren't course
assets.
When course content authors are creating their courses, we provide them
a special shorthand way of writing URLs that reference assets they have
uploaded to the course. If they uploaded a file called my-lil-pony.mp4,
and they wanted to provide a link to it when users view the course, they
would use /static/my-lil-pony.mp4. This special prefix -- /static/ --
signals to the static_replace middleware that it's a course asset and we
should write /static/my-lil-pony.mp4 to a URL that properly references
the asset based on the course ID, and things like the configured asset
CDN, etc.
Thus, the URL /static/my-lil-pony.mp4 gets turned into something like:
/assets/courseware/<md5hash>/asset-v1:edX+Demo+2016T1+type@asset+block/my-lil-pony.mp4
when viewed in the courseware.
Now, we also serve actual static assets from a prefix of /static/. This
is stuff like our JavaScript and CSS, and the JS/CSS/etc of
XBlocks/XModules. These paths look like:
/static/js/lms-main_vendor.46d6a8c02600.js
or
/static/xblock/resources/xmodule.vertical_block/public/js/vertical_student_view.43727a907769.js
Normally, these paths are caught by nginx, before they reach the LMS,
and are served straight from the filesystem. However, if you were to
have one of these paths in your course content, the static_replace
middleware would see the /static/ at the front and immediately think
it's a course asset, and would dutifully rewrite the URL to something
like:
/assets/courseware/<md5hash>/asset-v1:edX+Demo+2016T1+type@asset+block/static_xblock_resources_xmodule.vertical_block_public_js_vertical_student_view.43727a907769.js
which is not a course asset, and so it will always fail to load.
Long story short, I changed the static_replace middleware to
specifically check to see if the path being matched starts with
/static/xblock/, and if so, it keeps the original instead of rewriting
it.
To be able to more easily deal with same-origin issues, due to some course assets being loaded in an iframe, we're adding the ability to specify file extensions that should be excluded from URL canonicalization. The default value is simply 'html', which means we won't rewrite asset links that point to files ending in .html. Thus, they'll be loaded from the same origin as the parent page, and voila, no SOP issues.
A base URL can now be configured which is, potentially, prepended to an
asset URL. This allows a CDN, or caching server, to front static asset
requests, taking load off of the contentstore and speeding up page load
times.
Asset URL generation respects locked vs unlocked assets, and will not
generate links to locked assets that would traverse a CDN (even though
the authorization component of the contentserver middleware wouldn't
allow those links to work anyways).
Query params with a value that starts with /static should have their values converted to the full location.
Query params with a value that does not start with /static should be left unchanged.
This commit updates common/djangoapps.
These keys are now objects with a limited interface, and the particular
internal representation is managed by the data storage layer (the
modulestore).
For the LMS, there should be no outward-facing changes to the system.
The keys are, for now, a change to internal representation only. For
Studio, the new serialized form of the keys is used in urls, to allow
for further migration in the future.
Co-Author: Andy Armstrong <andya@edx.org>
Co-Author: Christina Roberts <christina@edx.org>
Co-Author: David Baumgold <db@edx.org>
Co-Author: Diana Huang <dkh@edx.org>
Co-Author: Don Mitchell <dmitchell@edx.org>
Co-Author: Julia Hansbrough <julia@edx.org>
Co-Author: Nimisha Asthagiri <nasthagiri@edx.org>
Co-Author: Sarina Canelake <sarina@edx.org>
[LMS-2370]
fixing unit tests
fixing merge error
fixing xqueue submission issue with unicode url (trial 0.1)
fixing fotmats as commented upon
removing yaml file language selection
Unicode changes to support QRF
removed unnecessary pass in modulestore/init.py
fixing merge error
fixing fotmats as commented upon
removing yaml file language selection
fixing pep8 violations
- fixing pylint violations
pylint violation
fixing line spaces and formats
ignore pylint E1101
remove empty line
fixing pylint violations
pep8 violations
bulk mail unicode/decode
fix migration error
fix pep8 just to push again
more unicode/decode
Final changes to comments and error messages.
