This moves the Content Libraries V2 backend from Blockstore [1] over to
Learning Core [2] For high-level overview and rationale of this move, see
the Blockstore DEPR [3]. There are several follow-up tasks [4], most notably
adding support for static assets in libraries.
BREAKING CHANGE: Existing V2 libraries, backed by Blockstore, will stop
working. They will continue to be listed in Studio, but their content
will be unavailable. They need to be deleted (via Django admin) or manually
migrated to Learning Core. We do not expect production sites to be in
this situation, as the feature has never left "experimental" status.
[1] https://github.com/openedx-unsupported/blockstore
[2] https://github.com/openedx/openedx-learning/
[3] https://github.com/openedx/public-engineering/issues/238
[4] https://github.com/openedx/edx-platform/issues/34283
Originally, Blockstore was an independent micro-service, accessed via a REST API.
Then, we changed Blockstore so it could be installed as an in-process Django app.
To support both modes, there existed a blockstore_api wrapper library in edx-platform,
with toggles controlling whether the wrapper called out to the micro-service's REST API versus the
Django app's Python API. Now that the micro-service Blockstore implementation is deprecated,
though, this wrapper library and toggles are just unnecessary complexity.
As a first step towards cleanup, we:
* remove several toggles and settings (details below);
* remove the blocokstore_api wrapper methods which called the REST API and
marshalled them back into Python objects; and
* remove all test cases which relied on the Blockstore micro-service (and were skippped in CI).
In the future, we will remove the content libraries indexer,
clean up the remaining bits of blockstore_api, and flatten out all
the Blockstore-related test class hierarchies which are no longer nceessary.
BREAKING CHANGE:
* These Django settings are removed:
* BLOCKSTORE_PUBLIC_URL_ROOT
* BLOCKSTORE_API_URL
* BLOCKSTORE_API_AUTH_TOKEN
* BLOCKSTORE_USE_BLOCKSTORE_APP_API
* The blockstore.use_blockstore_app_api Waffle switch is removed.
* edx-platform will act as it did when the DJango setting BLOCKSTORE_USE_BLOCKSTORE_APP_API
or the Waffle switch blockstore.use_blockstore_app_api were enabled. That is, any running Blockstore
micro-service instance will be ignored, and the Blockstore package which is installed into edx-platform
will be used instead.
Ref: https://github.com/openedx/blockstore/issues/296
Refactors and reworks the LibraryContentBlock so that its
sync-from-library operations are asynchronous and work with
V2 content libraries. This also required us to make
library_content block duplication asynchronous, as that
involves syncing from the source library.
For the sake of clarity, this PR includes two major method renames:
* update_children(...) -> sync_from_library(...)
* refresh_library(...) -> sync_from_library(upgrade_to_latest=True, ...)
an an XBlock HTTP handler rename:
/refresh_children -> /upgrade_and_sync
There are still a couple issues with import or duplication
of library_content blocks referencing V2 libraries other than
latest. These will be resolved in an upcoming PR.
Part of: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3820617729/Spec+Memo+Content+Library+Authoring+Experience+V2
Follow-up work: https://github.com/openedx/edx-platform/issues/33640
Co-authored-by: Connor Haugh <chaugh@2u.com>
Co-authored-by: Eugene Dyudyunov <evgen.dyudyunov@raccoongang.com>
Removed BasicAuthentication as a default from DRF
endpoints that have not overridden the authentication
classes. It appears this is not in use, and was just
implicitly a default because it came from DRF's
defaults.
See DEPR: https://github.com/openedx/edx-platform/issues/33028
Added new authentication classes to be used in
DEFAULT_AUTHENTICATION_CLASSES that include
observability. This will enable us to see more
about the endpoints using the defaults, to help
us make choices about changes in the defaults.
We make the DRF default of Session and Basic
Authentication explicit by setting
DEFAULT_AUTHENTICATION_CLASSES explicitly.
* test: warn about dependencies from cms->openedx->lms and vice versa
* test: warn about importing from package's internal implementation code
* chore: Update some imports to use public APIs only
* chore: Update 'bookmarks' app to have stricter public API
* fix: we are sharing 'adapters' from olx_rest_api to content_staging
Tests which @requires_blockstore (i.e. the Blockstore service) have
been made to run as a unit test using the installed Blockstore app, and
will be run by the platform CI.
The Blockstore service tests can still be run manually by setting
EDXAPP_RUN_BLOCKSTORE_TESTS=1
Related fixes:
* adds blockstore bundle storage settings
* let the studio devstack and test servers serve static files from
the /media URL This allows the blockstore/content libraries API to
serve blockstore assets in dev.
* Wrap ContentLibrary creation in an atomic transaction, so that if it
fails, the related bundle can be deleted directly from the database
during the exception handler. (Previously, we called a REST API which
deleted it as part of a separate service.)
django-not-configured is an error raised by pylint (with
the pylint-django plugin) when it's not correctly configured.
We should not be applying lint amnesty for such a violation.
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
First off, this fixes a bug where BundleCache would only set caches for
the duration of the default timeout, defeating the cache versioning
strategy.
Second, this adds an optimization so that bundle draft files are cached
as soon as possible.
This includes an optimization to the get_bundle_version_files_cached method, which is used very often when loading blockstore data; it was previously being cached only in a process-local cache (lru_cache). My hunch is that in production, with many appservers and LMS workers and frequent deployments and a large number of bundles, the process-local cache is not being hit very often.
I also increased the MAX_BLOCKSTORE_CACHE_DELAY from 60s to 300s; this reduces the frequency with which we check if either (A) an external system modified the blockstore bundle and/or (B) we have a cache invalidation bug somewhere. I am increasing it because that check is more expensive than I thought (calling blockstore API to ascertain latest version of a particular bundle), and I haven't seen any cache invalidation errors that this would help to work around. (Plus, increasing this will make such bugs more obvious.)
Also improve usefulness of some blockstore runtime logs for debugging
Context:
Sometimes when trying to load an XBlock's XML file from Amazon S3, AWS will return a 4xx or 5xx response along with error XML like:
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>foo/bar</Key>...</Error>
A bug in the get_bundle_file_data_with_cache method would cause this XML to be returned to the runtime anyways, as if it were the expected OLX. This would then (obviously) lead to strange parsing bugs, e.g. when trying to interpret <Code> as an <xblock-include>.
This fixes the bug and improves the logging, both to make this sort of issue easier to debug in the future and to return whatever detailed error code S3 provides (or Blockstore, if S3 is not being used).
This includes an optimization to the get_bundle_version_files_cached method, which is used very often when loading blockstore data; it was previously being cached only in a process-local cache (lru_cache). My hunch is that in production, with many appservers and LMS workers and frequent deployments and a large number of bundles, the process-local cache is not being hit very often.
I also increased the MAX_BLOCKSTORE_CACHE_DELAY from 60s to 300s; this reduces the frequency with which we check if either (A) an external system modified the blockstore bundle and/or (B) we have a cache invalidation bug somewhere. I am increasing it because that check is more expensive than I thought (calling blockstore API to ascertain latest version of a particular bundle), and I haven't seen any cache invalidation errors that this would help to work around. (Plus, increasing this will make such bugs more obvious.)
Also improve usefulness of some blockstore runtime logs for debugging
Context:
Sometimes when trying to load an XBlock's XML file from Amazon S3, AWS will return a 4xx or 5xx response along with error XML like:
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>foo/bar</Key>...</Error>
A bug in the get_bundle_file_data_with_cache method would cause this XML to be returned to the runtime anyways, as if it were the expected OLX. This would then (obviously) lead to strange parsing bugs, e.g. when trying to interpret <Code> as an <xblock-include>.
This fixes the bug and improves the logging, both to make this sort of issue easier to debug in the future and to return whatever detailed error code S3 provides (or Blockstore, if S3 is not being used).
* Removing from provider imports from openedx
* removed all uses of retire_dop_oauth2_models
* Removing provider library from lms, common, and cms
Created/copied function short_token(from django-oauth-provider) and create_hash256 to help with conversion
Scheduled emails show "unsubscribe" link if waffle switch
`schedules.course_update_show_unsubscribe` is enabled, and
settings.ACE_ENABLED_POLICIES respects `bulk_email_optout`.
API endpoint allows GET/POST requests, which:
* GET asks for confirmation of opt-out
* POST accepts "unsubscribe" or "cancel", where "unsubscribe" creates the
Optout entry, and "cancel" does nothing.
Fixes flaky tests:
* The resolvers handle users in "bins", which are groups that depend on the user ID.
* The test user ID varies depending on the test order.
* This change ensures that the bin requested matches the user for the test.
