The V2 libraries project had a few past iterations which were never
launched. This commit cleans up pieces from those which we don't need
for the real Libraries Relaunch MVP in Sumac:
* Remove ENABLE_LIBRARY_AUTHORING_MICROFRONTEND,
LIBRARY_AUTHORING_FRONTEND_URL, and
REDIRECT_TO_LIBRARY_AUTHORING_MICROFRONTEND, all of which are obsolete
now that library authoring has been merged into
https://github.com/openedx/frontend-app-authoring.
More details on the new Content Libraries configuration settings are
here: https://github.com/openedx/frontend-app-authoring/issues/1334
* Remove dangling support for syncing V2 (learning core-backed) library
content using the LibraryContentBlock. This code was all based on an
older understanding of V2 Content Libraries, where the libraries were
smaller and versioned as a whole rather then versioned by-item.
Reference to V2 libraries will be done on a per-block basis using
the upstream/downstream system, described here:
https://github.com/openedx/edx-platform/blob/master/docs/decisions/0020-upstream-downstream.rst
It's important that we remove this support now so that OLX course
authors don't stuble upon it and use it, which would be buggy and
complicate future migrations.
* Remove the "mode" parameter from LibraryContentBlock. The only
supported mode was and is "random". We will not be adding any further
modes. Going forward for V2, we will have an ItemBank block for
randomizing items (regardless of source), which can be synthesized
with upstream referenced as described above. Existing
LibraryContentBlocks will be migrated.
* Finally, some renamings:
* LibraryContentBlock -> LegacyLibraryContentBlock
* LibraryToolsService -> LegacyLibraryToolsService
* LibrarySummary -> LegacyLibrarySummary
Module names and the old OLX tag (library_content) are unchanged.
Closes: https://github.com/openedx/frontend-app-authoring/issues/1115
When serializing to OLX, the Learning Core runtime wraps HTML content in
CDATA to avoid having to escape every individual `<`, `>`, and `&`. The
runtime also puts newlines around the content within the CDATA,
So, given HTML content `...`, we get `<![CDATA[\n...\n]]>`.
The problem is that every time you serialize an HTML block to OLX, it
adds another pair of newlines. These newlines aren't visible to the end
users, but they do make it so that importing and exporting content never
reached a stable, aka "canonical" form. It also makes unit testing
difficult, because the value of `html_block.data` becomes a moving
target.
We do not believe these newlines are necessary, so we have removed them
from the `CDATA` block, and added a unit test to ensure that HTML blocks
having a canonical serialization.
Closes: https://github.com/openedx/edx-platform/issues/35525
Blockstore and all of its (experimental) functionality has been replaced with
openedx-learning, aka "Learning Core". This commit uninstalls the now-unused
openedx-blockstore package and removes all dangling references to it.
Note: This also removes the `copy_library_from_v1_to_v2` management command,
which has been broken ever since we switched from Blockstore to Learning Core.
Part of this DEPR: https://github.com/openedx/public-engineering/issues/238
Implements the connection from the teams feature to the content groups feature. This implementation uses the dynamic partition generator extension point to associate content groups with the users that belong to a Team.
This implementation was heavily inspired by the enrollment tracks dynamic partitions.
The Hashable object was moved in python 3.3 and support for the old
location is dropped in python 3.10 the new location is available in
python 3.8 so we can just update this and it should work with both
python 3.8 and 3.11
https://docs.python.org/3.8/library/collections.html
Given code like the following
```
class Foo:
@process_cached
def bar(self):
pass
```
In Python 3.8 referencing `bar` would not call its `__get__` method.
```
x = Foo().bar
```
However in Python 3.11, making the same call would call the `__get__`
method, permanently replacing the underlying `process_cached` object
with the partial function that references it.
This meant that code to clear the cache would work in Python 3.8 but
would break in 3.11
```
Foo().bar.cache.clear() # Works in 3.8 but not in 3.11
```
In 3.11 this results in the following error:
```
E AttributeError: 'functools.partial' object has no attribute 'cache'
```
To make this compatible in both version, we just add the cache as an
accessible attribute on the partial we generate for our wrapped
function.
This moves the Content Libraries V2 backend from Blockstore [1] over to
Learning Core [2] For high-level overview and rationale of this move, see
the Blockstore DEPR [3]. There are several follow-up tasks [4], most notably
adding support for static assets in libraries.
BREAKING CHANGE: Existing V2 libraries, backed by Blockstore, will stop
working. They will continue to be listed in Studio, but their content
will be unavailable. They need to be deleted (via Django admin) or manually
migrated to Learning Core. We do not expect production sites to be in
this situation, as the feature has never left "experimental" status.
[1] https://github.com/openedx-unsupported/blockstore
[2] https://github.com/openedx/openedx-learning/
[3] https://github.com/openedx/public-engineering/issues/238
[4] https://github.com/openedx/edx-platform/issues/34283
To determine whether or not we need to revoke any program certificates, we first run get_revokable_program_uuids. This calls get_certified_programs, which calls get_credentials, which uses the OpenEdx Core utility method get_api_data. get_api_data makes the API call inside a try block, does raise_for_status also inside the try block, and then catches the exception, logs it, and returns an empty result to the caller.
This means that on a failure to call the credentials API, get_credentials can’t tell the difference between a failure to hit the API (because credentials is, as it sometimes is during a notify_programs run, overloaded), or a learner with no program certificates. In this particular case, this is absolute failure, incorrect behavior.
* Adds a new flag, `raise_on_error` which will make `get_api_data` log the exception and then re-raise the HTTPError, defaulting to false in order to avoid changing the behavior on any other callers
* Also: my editor reformatted all of the touched files to our modern code standards, and it seemed appropriate to let it do that.
* Also: added type hints in some cases, because they helped me write the code and debug. Our test suite definitely reports mypy results on type errors so we are verifying that hints are correct.
FIXES: APER-3146
Originally, Blockstore was an independent micro-service, accessed via a REST API.
Then, we changed Blockstore so it could be installed as an in-process Django app.
To support both modes, there existed a blockstore_api wrapper library in edx-platform,
with toggles controlling whether the wrapper called out to the micro-service's REST API versus the
Django app's Python API. Now that the micro-service Blockstore implementation is deprecated,
though, this wrapper library and toggles are just unnecessary complexity.
As a first step towards cleanup, we:
* remove several toggles and settings (details below);
* remove the blocokstore_api wrapper methods which called the REST API and
marshalled them back into Python objects; and
* remove all test cases which relied on the Blockstore micro-service (and were skippped in CI).
In the future, we will remove the content libraries indexer,
clean up the remaining bits of blockstore_api, and flatten out all
the Blockstore-related test class hierarchies which are no longer nceessary.
BREAKING CHANGE:
* These Django settings are removed:
* BLOCKSTORE_PUBLIC_URL_ROOT
* BLOCKSTORE_API_URL
* BLOCKSTORE_API_AUTH_TOKEN
* BLOCKSTORE_USE_BLOCKSTORE_APP_API
* The blockstore.use_blockstore_app_api Waffle switch is removed.
* edx-platform will act as it did when the DJango setting BLOCKSTORE_USE_BLOCKSTORE_APP_API
or the Waffle switch blockstore.use_blockstore_app_api were enabled. That is, any running Blockstore
micro-service instance will be ignored, and the Blockstore package which is installed into edx-platform
will be used instead.
Ref: https://github.com/openedx/blockstore/issues/296
This switch has been kept disabled in edx.org for well over a year with no
trouble, and the migration to `CLOSEST_CLIENT_IP_FROM_HEADERS`
was introduced in Nutmeg.
DEPR issue: https://github.com/openedx/edx-platform/issues/33733
Refactors and reworks the LibraryContentBlock so that its
sync-from-library operations are asynchronous and work with
V2 content libraries. This also required us to make
library_content block duplication asynchronous, as that
involves syncing from the source library.
For the sake of clarity, this PR includes two major method renames:
* update_children(...) -> sync_from_library(...)
* refresh_library(...) -> sync_from_library(upgrade_to_latest=True, ...)
an an XBlock HTTP handler rename:
/refresh_children -> /upgrade_and_sync
There are still a couple issues with import or duplication
of library_content blocks referencing V2 libraries other than
latest. These will be resolved in an upcoming PR.
Part of: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3820617729/Spec+Memo+Content+Library+Authoring+Experience+V2
Follow-up work: https://github.com/openedx/edx-platform/issues/33640
Co-authored-by: Connor Haugh <chaugh@2u.com>
Co-authored-by: Eugene Dyudyunov <evgen.dyudyunov@raccoongang.com>
When including `JwtAuthentication`, the auth_header becomes `JWT
realm="api"`. Without it, it is `None`. This changes the behavior of the
code in DRF and returns a slightly different auth response.
Relevant Code: 56946fac8f/rest_framework/views.py (L456C3-L456C3)
Removes expected part of EXPECTED_ERRORS with a variety of changes.
- In many placed in the code, "expected" was used to mean
"ignored and expected", and all such instances are renamed to "ignored".
- The setting ``EXPECTED_ERRORS`` is renamed to ``IGNORED_ERRORS``,
which better matches how it was being used in the first place.
- The setting ``EXPECTED_ERRORS[REASON_EXPECTED]`` is renamed to ``IGNORED_ERRORS[REASON_IGNORED]``.
- The setting toggle ``EXPECTED_ERRORS[IS_IGNORED]`` is removed,
because it will now always be True.
- The how-to will is renamed to how_tos/logging-and-monitoring-ignored-errors.rst.
See 0002-logging-and-monitoring-expected-errors-removed.rst for more details.
Implements DEPR: https://github.com/openedx/edx-platform/issues/32405
**BREAKING CHANGE:** The rename of the setting ``EXPECTED_ERRORS`` to
``IGNORED_ERRORS``, and ``REASON_EXPECTED`` to ``REASON_IGNORED``,
was implemented without backward compatibility. Simply copy the old settings
with the new name as an expand phase before deleting the old names in the
contract phase.
This fixes a bug in xblock serialization when trying to copy a
unit/component that contains an HTML xblock with the characters "]]>"
which is a special character in CDATA.
Originally, we planned to add support for V2 libraries and for static
(hand-selected) library block reference via new block type: library_sourced.
We have since decided that it would be better to add those capabilities
in-place to the existing library_content block. This will ease V1->V2
library migration and make adoption of the new features easier for current
library users. It will also avoid duplication of logic between two block types,
we we fear would be error-prone. For details, see this ADR:
https://github.com/openedx/edx-platform/pull/33231
So, we are removing the library_sourced block.
This block has existed in edx-platform for a few years now, but was not
enabled by default and never officially supported. It was only usable via the
experimental V2 content library feature. Operators who added library_sourced
blocks to their course will now see them render as `HiddenBlock` instances, i.e.:
> ERROR: "library_sourced" is an unknown component type...
This should not impact other component types in such courses and should not
impact import/export.
This fixes the issue when pasting a copied unit that contains
python_lib.zip file into another course. The python_lib.zip file was not
being correctly copied over to the new course's Files & Uploads page.
This takes into account the extra files that are usually required when
copying problems containing JSInputs. Static files such as additional
CSS and JS files needed to interact and style the problem.
In ~Palm and earlier, all built-in XBlock Sass was included into CMS
(and LMS) styles before being compiled. So, if a site theme was meant to
affect built-in XBlock styling, those changes would be manifested
directly in the base CMS CSS that is included into every single Studio
page. When the user provided the `?site_theme` querystring parameter,
which is intended to allow devs & admins to view Studio through a given
theme, CMS would look up the given theme and serve the corresponding
base CMS CSS, which would affect the built-in XBlocks views (as
expected).
After ~Palm, built-in XBlocks styles are handled more similarly to to
pure XBlock styles, in that they are only requested when CMS tries to
render the block. In Studio, blocks are not rendered by the original
request, but by a subsequent AJAX request to the `/container_preview`
enpoint. Thus, passing the `?site_theme` query parameter to the original
request will apply the given theme to Studio's chrome, but the theme
will _not_ apply to built-in XBlock views, whose CSS is now loaded via
async request.
To fix this, we simply pass Studio's querystring parameters (including
`?site_theme`) along to the `/container_view` AJAX request. This will
cause CMS to correctly serve the built-in XBlock CSS from the theme
specified by `?site_theme`, rather than whatever the current theme is.
Part of: https://github.com/openedx/edx-platform/issues/32292
In ~Palm and earlier, all built-in XBlock Sass was included into LMS and CMS
styles before being compiled. The generated CSS was coupled together with
broader LMS/CMS CSS. This means that comprehensive themes have been able to
modify built-in XBlock appearance by setting certain Sass variables. We say that
built-in XBlock Sass was, and is expected to be, "theme-aware".
Shortly after Palm, we decoupled XBlock Sass from LMS and CMS Sass [1]. Each
built-in block's Sass is now compiled into two separate CSS targets, one for
block editing and one for block display. The CSS, now located at
`common/static/css/xmodule`, is injected into the running Webpack context with
the new `XModuleWebpackLoader`. Built-in XBlocks already used
`add_webpack_to_fragment` in order to add JS Webpack bundles to their view
fragments, so when CSS was added to Webpack, it Just Worked.
This unlocked a slieu of simplifications for static asset processing [2];
however, it accidentally made XBlock Sass theme-*unaware*, or perhaps
theme-confused, since the CSS was targeted at `common/static/css/xmodule`
regardless of the theme. The result of this is that **built-in XBlock views will
use CSS based on the Sass variables _last theme to be compiled._** Sass
variables are only used in a handful of places in XBlocks, so the bug is subtle,
but it is there for those running off of master. For example, using edX.org's
theme on master, we can see that there is a default blue underline in the Studio
sequence nav [3]. With this bugfix, it becomes the standard edX.org
greenish-black [4].
This commit makes several changes, firstly to fix the bug, and secondly to leave
ourselves with a more comprehensible asset setup in the `xmodule/` directory.
* We remove the `XModuleWebpackLoader`, thus taking built-in XBlock Sass back
out of Webpack.
* We compile XBlock Sass not to `common/static/css/xmodule`, but to:
* `[lms|cms]/static/css` for the default theme, and
* `<THEME_ROOT>/[lms|cms]/static/css`, for any custom theme.
This is where the comprehensive theming system expects to find themable
assets. Unfortunately, this does mean that the Sass is compiled twice, both
for LMS and CMS. We would have liked to compile it once to somewhere in the
`common/`, but comprehensive theming does not consider `common/` assets to be
themable.
* We split `add_webpack_to_fragment` into two more specialized functions:
* `add_webpack_js_to_fragment` , for adding *just* JS from a Webpack bundle,
and
* `add_sass_to_fragment`, for adding static links to CSS compiled themable
Sass (not Webpack). Both these functions are moved to a new module
`xmodule/util/builtin_assets.py`, since the original module
(`xmodule/util/xmodule_django.py`) didn't make a ton of sense.
* In an orthogonal bugfix, we merge Sass `CourseInfoBlock`, `StaticTabBlock`,
`AboutBlock` into the `HtmlBlock` Sass files. The first three were never used,
as their styling was handled by `HtmlBlock` (their shared parent class).
* As a refactoring, we change Webpack bundle names and Sass module names to be
less misleading:
* student_view, public_view, and author_view: was `<Name>BlockPreview`, is now
`<Name>BlockDisplay`.
* studio_view: was `<Name>BlockStudio`, is now `<Name>BlockEditor`.
* As a refactoring, we move the contents of `xmodule/static` into the existing
`xmodule/assets` directory, and adopt its simper structure. We now have:
* `xmodule/assets/*.scss`: Top-level compiled Sass modules. These could be
collapsed away in a future refactoring.
* `xmodule/assets/<blocktype>/*`: Resources for each block, including both JS
modules and Sass includes (underscore-prefixed so that they aren't
compiled). This structure maps closely with what externally-defined XBlocks
do.
* `xmodule/js` still exists, but it will soon be folded into the
`xmodule/assets`.
* We add a new README [4] to explain the new structure, and also update a
docstring in `openedx/lib/xblock/utils` which had fallen out of date with
reality.
* Side note: We avoid the term "XModule" in all of this, because that's
(thankfully) become a much less useful/accurate way to describe these blocks.
Instead, we say "built-in XBlocks".
Refs:
1. https://github.com/openedx/edx-platform/pull/32018
2. https://github.com/openedx/edx-platform/issues/32292
3. https://github.com/openedx/edx-platform/assets/3628148/8b44545d-0f71-4357-9385-69d6e1cca86f
4. https://github.com/openedx/edx-platform/assets/3628148/d0b7b309-b8a4-4697-920a-8a520e903e06
5. https://github.com/openedx/edx-platform/tree/master/xmodule/assets#readme
Part of: https://github.com/openedx/edx-platform/issues/32292
* refactor: improve typing of StaticFile named tuple
* feat: copy static asset files into the clipboard
* feat: paste static assets
* feat: show notification in studio about pasted assets
* fix: HTML XBlocks would lose the editor="raw" setting when copy-pasted.
* feat: copy python_lib.zip to the clipboard when it seems to be in use
We were using an old version of the python3-saml library,
which was causing issues with newer versions of social-core.
The reason it was pinned was because our etree implementation
didn't support several fields that the saml library did, so
we are now importing those entities as well.
