Final planned portion of https://github.com/openedx/edx-platform/issues/34702 -- waffle flag and middleware are removed.
Commits:
1. Feature change
- Delete `content_server.use_view` waffle flag in favor of always using view
- Delete the husk of `StaticContentServerMiddleware` and references to it
- Update views module docstring
2. Refactor (move)
- Move contentserver implementation into views.py
3. Lint cleanup
- Fix import ordering (from refactor debris + amnestied lint)
This repo is no longer using USE-JWT-COOKIE header,
since it has the required edx-drf-extensions>10.2.0,
where it was fully removed.
This is final clean-up for this repo.
See "[DEPR]: USE-JWT-COOKIE header" for more details:
- https://github.com/openedx/edx-drf-extensions/issues/371
See https://github.com/openedx/edx-platform/issues/34702
This necessarily involves switching from calling
`StaticContent.is_versioned_asset_path` to determine whether to handle the
request to having a hardcoded urlpattern. I've made the choice to hardcode
the other two patterns similarly rather than using imported constants. The
mapping of URL patterns to database records should be explicit (even though
we don't expect those constants to change out from under us.)
I've renamed the middleware rather than choosing a new name for the
implementation because there are other references in tests and other code.
This was the smaller change.
A note on HTTP methods: The middleware currently completely ignores the
request's HTTP method, so I wanted to confirm that only GETs were being
used in practice. This query reveals that 99.8% of requests that this
middleware handles are GET, with just a smattering of PROPFIND and OPTIONS
and a tiny number of HEAD and POST:
```
from Transaction select count(*) facet request.method
where name = 'WebTransaction/Function/openedx.core.djangoapps.contentserver.middleware:StaticContentServer'
since 4 weeks ago
```
Open edX implements its a JwtAuthentication class in edx-drf-extensions
(in edx_rest_framework_extensions.auth.jwt.authentication). This class
updates the local User database entry to match certain values in the
token. It's used as a way to automatically provision and update users
with their LMS user information on other Open edX services like
ecommerce.
Since LMS and Studio keep the record of truth in its database tables,
they should *not* update their database user information based on the
JWT. Doing so would allow stale JWTs to incorrectly reset user values
after they had been changed in the LMS. This is done by having the
EDX_DRF_EXTENSIONS['JWT_PAYLOAD_USER_ATTRIBUTE_MAPPING'] setting be an
empty dictionary, and was set correctly for the LMS in its common.py env
settings module. Unfortunately, this was *not* being set for Studio.
This commit adds the same setting to Studio's common settings module.
Prior to this commit, it was possible for a stale JWT to reset user
attributes if the user hit a Studio API endpoint that used JWT for auth
(e.g. endpoints used by the Course Authoring MFE). This opened up a
potential security issue where a global staff user (is_staff=True) that
had their global staff status removed (is_staff=False) could have up to
a one hour window in which they could use their stale-but-still-valid
global-staff JWT token to regain global staff status by calling a Studio
endpoint with their browser.
Blockstore and all of its (experimental) functionality has been replaced with
openedx-learning, aka "Learning Core". This commit uninstalls the now-unused
openedx-blockstore package and removes all dangling references to it.
Note: This also removes the `copy_library_from_v1_to_v2` management command,
which has been broken ever since we switched from Blockstore to Learning Core.
Part of this DEPR: https://github.com/openedx/public-engineering/issues/238
Introduce emission of the COURSE_PASSING_STATUS_UPDATED as well as CCX_COURSE_PASSING_STATUS_UPDATED events, that are groundwork for the new Credly integration and the future badging initiative.
Product GH ticket for tracking - openedx/platform-roadmap#280
The LMS and Studio need to set values for CORS_ALLOW_HEADERS so that the
MFEs can work properly, since preflight requests will need to send over
extra headers. Prior to this commit, CORS_ALLOW_HEADERS was being
redefined in multiple places in edx-platform and again in Tutor's config
because it was only being conditionally set if ENABLE_CORS_HEADERS was
True (which was a policy setting). But CORS_ALLOW_HEADERS is application
logic in that the value is determined by what the view needs, and won't
vary by deployment.
By consolidating this to always be defined in the common.py files, we
make sure that deployment environments don't have to define it. An
example of where this bit us was when course import in the course
authoring MFE did not work because Tutor was using an outdated value for
this setting.
A followup to this would be to just rip out the ENABLE_CORS_HEADERS
setting entirely, and just always have it on. But that would benefit
from a little more discovery to make sure there's no weird use case that
still requires it to be False (maybe something in the test suite?).
Reverts #34554, which causes compilation of edX.org's
legacy comprehensive theme to be skipped in their deployment pipeline.
We have not determined the precise cause yet, but it seems like the
compile_sass management command is not correctly getting the
list of comprehensive theme directories from Django settings.
Together, these changes make it so that all features of the Paver-based
asset compilation system are supported with drop-in Paver-free
replacements. The remaining Paver asset functions are trivial wrappers,
which can be comfortably deleted before Sumac.
* Turn `./manage.py ... compile_sass` into a simple wrapper around `npm
run compile-sass`
* Turn `paver webpack` into a simple wrapper around `npm run webpack`
* Turn `pavelib.assets:collect_assets` into a simple wrapper around
`./manage.py ... collectstatic`
* Add/improve deprecation warnings for all Paver asset commands.
* Load defaults for asset-related Django settings from environment
variables. This allows the build to work without Python. For the
settings which will be removed in Sumac, I've added deprecation
warnings.
* Change EDX_PLATFORM_THEME_DIRS env var to COMPREHENSIVE_THEME_DIRS.
This simplifies the migration instructions, because all the new env
vars now match their corresponding Django settings. This amends an
ADR, but it should not be a breaking change because the env var was
recently added (since Quince) and nobody should be using it yet.
* Future-proof the static assets ADR with links. The linked pages will
be kept up-to-date even if the ADR isn't.
Part of: https://github.com/openedx/edx-platform/issues/34467
A new field in the Problem settings for choosing a Grading Method. Currently, the only Grading Method is the Last Score. From now on, when turning the feature flag on, the new grading methods available for configuration in Studio are:
- Last Score (Default): The last score made is taken for grading.
- First Score: The first score made is taken for grading.
- Highest Score: The highest score made is taken for grading.
- Average Score: The average of all scores made is taken for grading.
This moves the Content Libraries V2 backend from Blockstore [1] over to
Learning Core [2] For high-level overview and rationale of this move, see
the Blockstore DEPR [3]. There are several follow-up tasks [4], most notably
adding support for static assets in libraries.
BREAKING CHANGE: Existing V2 libraries, backed by Blockstore, will stop
working. They will continue to be listed in Studio, but their content
will be unavailable. They need to be deleted (via Django admin) or manually
migrated to Learning Core. We do not expect production sites to be in
this situation, as the feature has never left "experimental" status.
[1] https://github.com/openedx-unsupported/blockstore
[2] https://github.com/openedx/openedx-learning/
[3] https://github.com/openedx/public-engineering/issues/238
[4] https://github.com/openedx/edx-platform/issues/34283
See discussion here: https://github.com/overhangio/tutor/pull/984
This is a breaking change that will explicitely set the timeout of
course structure cache entries to 1 week, instead of being unlimited. If
you wish to revert to the former behaviour, you should set
`course_structure_cache["TIMEOUT"] = None`.
The course structure cache keys were previously set with an explicit
timeout of `None` which was overriding the cache default timeout of 2h.
This was OK in environments where the cache is configured with a maximum
memory limit and an automatic key eviction strategy. But in others (such
as Tutor), the course structure cache could grow infinitely.
It was agreed that course structure cache keys should be long-lived but
should respect the default cache structure timeout. Thus, we set here
the TTL to 1 week.
We can also configure Tutor to use a cache eviction policy. But that
means we need to set a `maxmemory` value in Redis. It's not possible to
set a value that will be appropriate for everyone:
- if it's higher than the total memory (e.g: in small instances), server
will crash before the cache is filled.
- if it's too low (e.g: in large instances), the whole platform will abruptly
slow down as many cache entries are suddenly evicted.
That question of whether Tutor should define a cache eviction policy is
still under discussion, but it can be resolved independently of this
change.
BREAKING CHANGE: Removes all code, tests, and settings related to
indexing of V2 (blockstore-backed) content libraries in elasticsearch.
This includes indexing of top-level library metadata as well as indexing
of library block metadata. Operators who enabled the experimental
Library Authoring MFE *and* the experimental ENABLE_CONTENT_LIBRARY_INDEX
feature may notice that sorting, filtering, and searching of V2
libraries and their blocks may now be slower and/or less powerful.
The ENABLE_CONTENT_LIBRARY_INDEX feature was already disabled by
default, so most/all operators (including edx.org) should not notice
any difference.
Removed settings include:
* FEATURES['ENABLE_CONTENT_LIBRARY_INDEX']
* ENABLE_ELASTICSEARCH_FOR_TESTS
* TEST_ELASTICSEARCH_USE_SSL
* TEST_ELASTICSEARCH_HOST
* TEST_ELASTICSEARCH_PORT
For rationale, see the updated "Status" section of:
./openedx/core/djangoapps/content_libraries/docs/decisions/0001-index-libraries-in-elasticsearch.rst
Originally, Blockstore was an independent micro-service, accessed via a REST API.
Then, we changed Blockstore so it could be installed as an in-process Django app.
To support both modes, there existed a blockstore_api wrapper library in edx-platform,
with toggles controlling whether the wrapper called out to the micro-service's REST API versus the
Django app's Python API. Now that the micro-service Blockstore implementation is deprecated,
though, this wrapper library and toggles are just unnecessary complexity.
As a first step towards cleanup, we:
* remove several toggles and settings (details below);
* remove the blocokstore_api wrapper methods which called the REST API and
marshalled them back into Python objects; and
* remove all test cases which relied on the Blockstore micro-service (and were skippped in CI).
In the future, we will remove the content libraries indexer,
clean up the remaining bits of blockstore_api, and flatten out all
the Blockstore-related test class hierarchies which are no longer nceessary.
BREAKING CHANGE:
* These Django settings are removed:
* BLOCKSTORE_PUBLIC_URL_ROOT
* BLOCKSTORE_API_URL
* BLOCKSTORE_API_AUTH_TOKEN
* BLOCKSTORE_USE_BLOCKSTORE_APP_API
* The blockstore.use_blockstore_app_api Waffle switch is removed.
* edx-platform will act as it did when the DJango setting BLOCKSTORE_USE_BLOCKSTORE_APP_API
or the Waffle switch blockstore.use_blockstore_app_api were enabled. That is, any running Blockstore
micro-service instance will be ignored, and the Blockstore package which is installed into edx-platform
will be used instead.
Ref: https://github.com/openedx/blockstore/issues/296
Refactors and reworks the LibraryContentBlock so that its
sync-from-library operations are asynchronous and work with
V2 content libraries. This also required us to make
library_content block duplication asynchronous, as that
involves syncing from the source library.
For the sake of clarity, this PR includes two major method renames:
* update_children(...) -> sync_from_library(...)
* refresh_library(...) -> sync_from_library(upgrade_to_latest=True, ...)
an an XBlock HTTP handler rename:
/refresh_children -> /upgrade_and_sync
There are still a couple issues with import or duplication
of library_content blocks referencing V2 libraries other than
latest. These will be resolved in an upcoming PR.
Part of: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3820617729/Spec+Memo+Content+Library+Authoring+Experience+V2
Follow-up work: https://github.com/openedx/edx-platform/issues/33640
Co-authored-by: Connor Haugh <chaugh@2u.com>
Co-authored-by: Eugene Dyudyunov <evgen.dyudyunov@raccoongang.com>
