Version 0.16.3 of openedx-learning adds support for calling
create_next_component_version with the bytes data of the files to be
added, instead of just the ID of the Content object. This lets us
simplify the code in edx-platform a bit.
PII Annotations are very out of date, this commit adds most that were
missing in edx-platform, and some additional annotations to the
safelist. It is not comprehensive, several other upstream Open edX
packages also need to be updated. It also does not include removing
annotations that have been moved upstream, or been removed entirely.
Those are separate follow-on tasks.
At one point, we envisioned having different kinds of libraries, e.g.
a "Video" library would be distinct from a "Problem" library. Later on,
we decided on a more generalized form of Libraries, where any given
library can hold any combination of content–which would then be
organized using collections and tagging.
Due to this shift in perspective, these values haven't actually been
used for a long time. This is just getting rid of them altogether.
The biggest challenge is dealing with the mismatch between how Libraries store
assets (per-Component) and how Courses store assets (global Files and Uploads
space). To bridge this, we're going to kludge a component-local namespace in
Files and Uploads by making use of the obscure feature that you can create
folders there at an API level, even if no such UI exists.
In this commit:
* Assets work when copy-pasting between library components.
* Assets work when copy-pasting from a library to a course, with the convention
being to put that file in a subdirectory of the form:
components/{block_type}/{block_id}/file.
Note that the Studio course Files page still just shows the filename.
* Assets work when copy-pasting from a course to a library.
Top level assets are put into a static folder in the Component, per Learning
Core conventions.
Limitations:
* Roundtrips don't work properly.
* There's no normalized form, so directories will start nesting if you copy
from library and paste into course, then copy the pasted thing and paste back
into library, etc. This was deemed acceptable for Sumac.
Low level stuff:
* XBlockSerializerForLearningCore has been removed, with the url_name stripping
functionality added as an optional param to XBlockSerializer (the other stuff
was for children and "vertical" -> "unit" conversion, neither of which are
relevant now).
* url_name is now stripped out of anything added to the clipboard, so that we
don't end up writing it in block.xml when it is redundant (and would be
stripped out with the next write anyway).
For the Libraries Relaunch Beta. This should not affect any site which
has kept New Libraries disabled.
Issue: https://github.com/openedx/frontend-app-authoring/issues/1170
Adds a new Studio-only libraries static asset endpoint at
/library_assets/blocks/{usage_key}. This endpoint will serve assets
only from the Draft branch, and is only available to people who have
read permission to the containing library.
This also moves the existing library asset endpoint that did lookups
by Component Version to /library_assets/component_versions/{uuid}
This change was motivated by the desire to make it easier to make the
editor preview work for images by having a single URL that will
consistently point to the latest version of the asset, rather than
having a new URL after every save (which the Component Version lookup
* fix: problem block could not be used with versioned handler URls
* refactor: simplify REST API handling of usage keys
* feat: add more version awareness and update tests
* fix: make the preview changes modal bigger as requested
* refactor: parse version at the urlconf layer too
Add support for displaying static assets in the Learing Core XBlock
runtime via "/static/asset-name" style substitutions in the OLX. This is
currently used for new Content Library components.
Static asset display is version-aware, so viewing older versions of the
XBlock content via the embed view will show the appropriate assets for
that version.
This introdues the idea of "upstream" and "downstream" content,
where downstreams (like course components) can pull content updates from
upstreams (like learning core-backed content library blocks). This
supports the upcoming Content Libraries Relaunch Beta for Sumac.
New features include:
* A new XBlockMixin: UpstreamSyncMixin.
* A new CMS Python API: cms.lib.xblock.upstream_sync
* A new CMS JSON API: /api/contentstore/v2/downstreams
* A temporary, very basic UI for syncing from Content Library blocks
Implements:
https://github.com/kdmccormick/edx-platform/blob/kdmccormick/upstream-proto/docs/decisions/0020-upstream-block.rst
Co-authored-by: Braden MacDonald <braden@opencraft.com>
This commit adds a new view to serve static assets for content
libraries, along with Content Library API calls to add, delete, and get
metadata about these assets. These assets come from Learning Core and
should ONLY BE ACCESSED FROM STUDIO. Users must have read access to the
library in order to see an asset in that library.
This also re-implements video transcript support for content libraries
and re-enables some previously disabled tests around it.
The V2 libraries project had a few past iterations which were never
launched. This commit cleans up pieces from those which we don't need
for the real Libraries Relaunch MVP in Sumac:
* Remove ENABLE_LIBRARY_AUTHORING_MICROFRONTEND,
LIBRARY_AUTHORING_FRONTEND_URL, and
REDIRECT_TO_LIBRARY_AUTHORING_MICROFRONTEND, all of which are obsolete
now that library authoring has been merged into
https://github.com/openedx/frontend-app-authoring.
More details on the new Content Libraries configuration settings are
here: https://github.com/openedx/frontend-app-authoring/issues/1334
* Remove dangling support for syncing V2 (learning core-backed) library
content using the LibraryContentBlock. This code was all based on an
older understanding of V2 Content Libraries, where the libraries were
smaller and versioned as a whole rather then versioned by-item.
Reference to V2 libraries will be done on a per-block basis using
the upstream/downstream system, described here:
https://github.com/openedx/edx-platform/blob/master/docs/decisions/0020-upstream-downstream.rst
It's important that we remove this support now so that OLX course
authors don't stuble upon it and use it, which would be buggy and
complicate future migrations.
* Remove the "mode" parameter from LibraryContentBlock. The only
supported mode was and is "random". We will not be adding any further
modes. Going forward for V2, we will have an ItemBank block for
randomizing items (regardless of source), which can be synthesized
with upstream referenced as described above. Existing
LibraryContentBlocks will be migrated.
* Finally, some renamings:
* LibraryContentBlock -> LegacyLibraryContentBlock
* LibraryToolsService -> LegacyLibraryToolsService
* LibrarySummary -> LegacyLibrarySummary
Module names and the old OLX tag (library_content) are unchanged.
Closes: https://github.com/openedx/frontend-app-authoring/issues/1115
* refactor: use django signals to trigger LIBRARY_COLLECTION events
* refactor: use collection usage_key as search document id
This change standardises the search document "id" to be a meilisearch ID
generated from the usage key, for all types of indexed objects.
This is important for collections so we can locate the collection
document in the search index solely from the data provided by the
LIBRARY_COLLECTION_DELETED event (library_key + collection_key), even if
the collection has been deleted from the database.
* refactor: avoid fetching more data than we have to.
* get_library_collection_usage_key and
searchable_doc_tags_for_collection do not need a Collection object;
the usage key can be created from the library_key and collection_key.
* updated searchable_doc_for_collection to require the parts of the
collection usage key + an optional collection. This allows us to
identify the collection's search document from its usage key without
requiring an existing Collection object (in case it's been deleted).
Also removes the edge case for indexing Collections not associated
with a ContentLibrary -- this won't ever really happen.
* feat: remove soft- and hard-deleted collections from search index
* feat: adds library_component_usage_key to content_libraries.api
* refactor: send CONTENT_OBJECT_ASSOCIATON_CHANGED on django model signals
so that added/removed collections are removed/re-added to component documents.
Special case: When a collection is soft-deleted/restored, we detect this
in the search index and update the collection's component documents
directly, without a CONTENT_OBJECT_ASSOCIATON_CHANGED signal.
* chore: bumps openedx-learning to 0.13.0
When serializing to OLX, the Learning Core runtime wraps HTML content in
CDATA to avoid having to escape every individual `<`, `>`, and `&`. The
runtime also puts newlines around the content within the CDATA,
So, given HTML content `...`, we get `<![CDATA[\n...\n]]>`.
The problem is that every time you serialize an HTML block to OLX, it
adds another pair of newlines. These newlines aren't visible to the end
users, but they do make it so that importing and exporting content never
reached a stable, aka "canonical" form. It also makes unit testing
difficult, because the value of `html_block.data` becomes a moving
target.
We do not believe these newlines are necessary, so we have removed them
from the `CDATA` block, and added a unit test to ensure that HTML blocks
having a canonical serialization.
Closes: https://github.com/openedx/edx-platform/issues/35525
* feat: Add Library Collections REST endpoints
* test: Add tests for Collections REST APIs
* chore: Add missing __init__ files
* docs: Add warning about unstable REST APIs
* feat: emit CONTENT_OBJECT_ASSOCIATIONS_CHANGED
whenever a content object's tags or collections have changed,
and handle that event in content/search.
The deprecated CONTENT_OBJECT_TAGS_CHANGED event is still emitted when
tags change; to be removed after Sumac.
* docs: replaces CONTENT_OBJECT_TAGS_CHANGED with CONTENT_OBJECT_ASSOCIATIONS_CHANGED
* chore: updates openedx-events==9.14.0
* chore: updates openedx-learning==0.11.4
Co-authored-by: Yusuf Musleh <yusuf@opencraft.com>
Co-authored-by: Chris Chávez <xnpiochv@gmail.com>
Co-authored-by: Rômulo Penido <romulo@thinkdash.dev>
Blockstore and all of its (experimental) functionality has been replaced with
openedx-learning, aka "Learning Core". This commit uninstalls the now-unused
openedx-blockstore package and removes all dangling references to it.
Note: This also removes the `copy_library_from_v1_to_v2` management command,
which has been broken ever since we switched from Blockstore to Learning Core.
Part of this DEPR: https://github.com/openedx/public-engineering/issues/238
This moves the Content Libraries V2 backend from Blockstore [1] over to
Learning Core [2] For high-level overview and rationale of this move, see
the Blockstore DEPR [3]. There are several follow-up tasks [4], most notably
adding support for static assets in libraries.
BREAKING CHANGE: Existing V2 libraries, backed by Blockstore, will stop
working. They will continue to be listed in Studio, but their content
will be unavailable. They need to be deleted (via Django admin) or manually
migrated to Learning Core. We do not expect production sites to be in
this situation, as the feature has never left "experimental" status.
[1] https://github.com/openedx-unsupported/blockstore
[2] https://github.com/openedx/openedx-learning/
[3] https://github.com/openedx/public-engineering/issues/238
[4] https://github.com/openedx/edx-platform/issues/34283
After we merged this PR: https://github.com/openedx/edx-platform/pull/33920
this error began popping up in logs:
Unable to load XBlock 'staffgradedxblock'
....
ImportError: cannot import name 'get_course_blocks' from
partially initialized module 'lms.djangoapps.course_blocks.api'
(most likely due to a circular import) ...
The root cause was the new imports of `derived_key` and `BlockKey` into
xmodule/library_content_block.py. Those new imports come from
xmodule/modulestore/store_utilities.py, which runs
`XBlock.load_classes()` at the module level, which fails because we are
still in the process of loading xmodule/library_content_block.
As a solution, we move both `derived_key` and `BlockKey` to
xmodule/util/keys.py. We could potentially move that file to opaque-keys
eventually, depending on how well we think that those concepts generalize.
Also:
* We rename the function from derived_key to derive_key, as
functions should be verbs.
* We combine the first to parameters of derive_key (a source ContextKey
and a source BlockKey) into a single parameter (a source UsageKey). In
my opinion, this makes the function call easier to understand.
BREAKING CHANGE: Removes all code, tests, and settings related to
indexing of V2 (blockstore-backed) content libraries in elasticsearch.
This includes indexing of top-level library metadata as well as indexing
of library block metadata. Operators who enabled the experimental
Library Authoring MFE *and* the experimental ENABLE_CONTENT_LIBRARY_INDEX
feature may notice that sorting, filtering, and searching of V2
libraries and their blocks may now be slower and/or less powerful.
The ENABLE_CONTENT_LIBRARY_INDEX feature was already disabled by
default, so most/all operators (including edx.org) should not notice
any difference.
Removed settings include:
* FEATURES['ENABLE_CONTENT_LIBRARY_INDEX']
* ENABLE_ELASTICSEARCH_FOR_TESTS
* TEST_ELASTICSEARCH_USE_SSL
* TEST_ELASTICSEARCH_HOST
* TEST_ELASTICSEARCH_PORT
For rationale, see the updated "Status" section of:
./openedx/core/djangoapps/content_libraries/docs/decisions/0001-index-libraries-in-elasticsearch.rst
Originally, Blockstore was an independent micro-service, accessed via a REST API.
Then, we changed Blockstore so it could be installed as an in-process Django app.
To support both modes, there existed a blockstore_api wrapper library in edx-platform,
with toggles controlling whether the wrapper called out to the micro-service's REST API versus the
Django app's Python API. Now that the micro-service Blockstore implementation is deprecated,
though, this wrapper library and toggles are just unnecessary complexity.
As a first step towards cleanup, we:
* remove several toggles and settings (details below);
* remove the blocokstore_api wrapper methods which called the REST API and
marshalled them back into Python objects; and
* remove all test cases which relied on the Blockstore micro-service (and were skippped in CI).
In the future, we will remove the content libraries indexer,
clean up the remaining bits of blockstore_api, and flatten out all
the Blockstore-related test class hierarchies which are no longer nceessary.
BREAKING CHANGE:
* These Django settings are removed:
* BLOCKSTORE_PUBLIC_URL_ROOT
* BLOCKSTORE_API_URL
* BLOCKSTORE_API_AUTH_TOKEN
* BLOCKSTORE_USE_BLOCKSTORE_APP_API
* The blockstore.use_blockstore_app_api Waffle switch is removed.
* edx-platform will act as it did when the DJango setting BLOCKSTORE_USE_BLOCKSTORE_APP_API
or the Waffle switch blockstore.use_blockstore_app_api were enabled. That is, any running Blockstore
micro-service instance will be ignored, and the Blockstore package which is installed into edx-platform
will be used instead.
Ref: https://github.com/openedx/blockstore/issues/296
After refactoring the library_content block to use asynchronous tasks
for syncing and duplicating children, we are seeing an error arise
during library_content duplication process on edx.org:
Traceback (most recent call last):
File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/content_libraries/tasks.py", line 305, in duplicate_children
_copy_overrides(store=store, user_id=user_id, source_block=source_block, dest_block=dest_block)
File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/content_libraries/tasks.py", line 383, in _copy_overrides
dest_block=dest_block.runtime.get_block(dest_child_key),
File "/edx/app/edxapp/edx-platform/xmodule/x_module.py", line 1401, in get_block
block = self.load_item(usage_id, for_parent=for_parent)
File "/edx/app/edxapp/edx-platform/xmodule/modulestore/split_mongo/caching_descriptor_system.py", line 127, in _load_item
block_data = self.get_module_data(block_key, course_key)
File "/edx/app/edxapp/edx-platform/xmodule/modulestore/split_mongo/caching_descriptor_system.py", line 154, in get_module_data
raise ItemNotFoundError(block_key)
xmodule.modulestore.exceptions.ItemNotFoundError: BlockKey(...)
We cannot reproduce the issue locally.
We are not entirely certain the cause of this, but we think it might
have do with caching. Specifically, the `store.get_item` and
`source_block.runtime.get_block` methods might use a different cache
than `dest_block.runtime.get_block`. It's possible that writes to
Mongo are sitting in one of those caches, causing reads from the
`dest_block.runtime`'s cache to fail to find dest_block's children.
We attempt to fix this by using the same "block getting" method
consistently. So instead of using a mix of `store.get_item`,
`source_block.runtime.get_block`, and `dest_block.runtime.get_block`,
we just use `store.get_item` everywhere.
