BREAKING CHANGE: this forces course IDs in modulestore to be unique (case insensitive). This was always supposed to be the case, but it wasn't working properly on MySQL. Upgrading past this commit may cause a migration failure if you have conflicting course IDs - see the migration 0004 docstring for details.
There is a singleton SplitMongoModuleStore instance that is returned
whenever we call the ubiquitous modulestore() function (wrapped in a
MixedModuleStore). During initialization, SplitMongoModuleStore sets
up a small handful of XBlock runtime services that are intended to be
shared globally: i18n, fs, cache.
When we get an individual block back from the store using get_item(),
SplitMongoModuleStore creates a SplitModuleStoreRuntime using
SplitMongoModuleStore.create_runtime(). These runtimes are intended to
be modified on a per-item, and later per-user basis (using
prepare_runtime_for_user()).
Prior to this commit, the create_runtime() method was assigning the
globally shared SplitMongoModuleStore.services dict directly to the
newly instantiated SplitModuleStoreRuntime. This meant that even though
each block had its own _services dict, they were all in fact pointing
to the same underlying object. This exposed us to a risk of multiple
threads contaminating each other's SplitModuleStoreRuntime services
when deployed under load in multithreaded mode. We believe this led to
a race condition that caused student submissions to be mis-scored in
some cases.
This commit makes a copy of the SplitMongoModuleStore.services dict for
each SplitModuleStoreRuntime. The baseline global services are still
shared, but other per-item and per-user services are now better
isolated from each other.
This commit also includes a small modification to the PartitionService,
which up until this point had relied on the (incorrect) shared instance
behavior. The details are provided in the comments in the
PartitionService __init__().
It's worth noting that the historical rationale for having a singleton
ModuleStore instance is that the ModuleStore used to be extremely
expensive to initialize. This was because at one point, the init
process required reading entire XML-based courses into memory, or
pre-computing complex field inheritance caches. This is no longer the
case, and SplitMongoModuleStore initialization is in the 1-2 ms range,
with most of that being for PyMongo's connection setup. We should try
to fully remove the global singleton in the Verawood release cycle in
order to make this kind of bug less likely.
* Consolidates and renames the runtime used as a base for all the others:
* Before: `xmodule.x_module:DescriptorSystem` and
`xmodule.mako_block:MakoDescriptorSystem`.
* After: `xmodule.x_module:ModuleStoreRuntime`.
* Co-locates and renames the runtimes for importing course OLX:
* Before: `xmodule.x_module:XMLParsingSystem` and
`xmodule.modulestore.xml:ImportSystem`.
* After: `xmodule.modulestore.xml:XMLParsingModuleStoreRuntime` and
`xmodule.modulestore.xml:XMLImportingModuleStoreRuntime`.
* Note: I would have liked to consolidate these, but it would have
involved nontrivial test refactoring.
* Renames the stub Old Mongo runtime:
* Before: `xmodule.modulestore.mongo.base:CachingDescriptorSystem`.
* After: `xmodule.modulestore.mongo.base:OldModuleStoreRuntime`.
* Renames the Split Mongo runtime, the which is what runs courses in LMS and CMS:
* Before: `xmodule.modulestore.split_mongo.caching_descriptor_system:CachingDescriptorSystem`.
* After: `xmodule.modulestore.split_mongo.runtime:SplitModuleStoreRuntime`.
* Renames some of the dummy runtimes used only in unit tests.
This PR removes the usage of the custom `mongodb_proxy` package from
edx-platform, replacing it with MongoDB's built-in retry functionality via
`retryReads=True`.
Changes:
* Removed all references to `mongodb_proxy.MongoProxy` from connection logic.
* Updated `connect_to_mongodb()` to always use `retryReads=True`.
* Uninstalled `openedx-mongodbproxy` from edx-platform requirements.
Fixes: https://github.com/openedx/edx-platform/issues/35210
The V2 libraries project had a few past iterations which were never
launched. This commit cleans up pieces from those which we don't need
for the real Libraries Relaunch MVP in Sumac:
* Remove ENABLE_LIBRARY_AUTHORING_MICROFRONTEND,
LIBRARY_AUTHORING_FRONTEND_URL, and
REDIRECT_TO_LIBRARY_AUTHORING_MICROFRONTEND, all of which are obsolete
now that library authoring has been merged into
https://github.com/openedx/frontend-app-authoring.
More details on the new Content Libraries configuration settings are
here: https://github.com/openedx/frontend-app-authoring/issues/1334
* Remove dangling support for syncing V2 (learning core-backed) library
content using the LibraryContentBlock. This code was all based on an
older understanding of V2 Content Libraries, where the libraries were
smaller and versioned as a whole rather then versioned by-item.
Reference to V2 libraries will be done on a per-block basis using
the upstream/downstream system, described here:
https://github.com/openedx/edx-platform/blob/master/docs/decisions/0020-upstream-downstream.rst
It's important that we remove this support now so that OLX course
authors don't stuble upon it and use it, which would be buggy and
complicate future migrations.
* Remove the "mode" parameter from LibraryContentBlock. The only
supported mode was and is "random". We will not be adding any further
modes. Going forward for V2, we will have an ItemBank block for
randomizing items (regardless of source), which can be synthesized
with upstream referenced as described above. Existing
LibraryContentBlocks will be migrated.
* Finally, some renamings:
* LibraryContentBlock -> LegacyLibraryContentBlock
* LibraryToolsService -> LegacyLibraryToolsService
* LibrarySummary -> LegacyLibrarySummary
Module names and the old OLX tag (library_content) are unchanged.
Closes: https://github.com/openedx/frontend-app-authoring/issues/1115
We migrated the source of truth for what the active draft and published
versions of course and v1 library content are to the
SplitModulestoreCourseIndex Django model. But the contentpruning
code (structures.py) that was developed in tubular and will be moved to
edx-platform is not aware of this newer model, and still only pulls its
source of truth from MongoDB. So we *must* continue to do writes to
MongoDB, or the pruning code will start pruning live versions.
The longer term fix for this is to make the pruning code aware of
SplitModulestoreCourseIndex, which will be easier once it's moved into
edx-platform.
See discussion here: https://github.com/overhangio/tutor/pull/984
This is a breaking change that will explicitely set the timeout of
course structure cache entries to 1 week, instead of being unlimited. If
you wish to revert to the former behaviour, you should set
`course_structure_cache["TIMEOUT"] = None`.
The course structure cache keys were previously set with an explicit
timeout of `None` which was overriding the cache default timeout of 2h.
This was OK in environments where the cache is configured with a maximum
memory limit and an automatic key eviction strategy. But in others (such
as Tutor), the course structure cache could grow infinitely.
It was agreed that course structure cache keys should be long-lived but
should respect the default cache structure timeout. Thus, we set here
the TTL to 1 week.
We can also configure Tutor to use a cache eviction policy. But that
means we need to set a `maxmemory` value in Redis. It's not possible to
set a value that will be appropriate for everyone:
- if it's higher than the total memory (e.g: in small instances), server
will crash before the cache is filled.
- if it's too low (e.g: in large instances), the whole platform will abruptly
slow down as many cache entries are suddenly evicted.
That question of whether Tutor should define a cache eviction policy is
still under discussion, but it can be resolved independently of this
change.
`remove` method of Collection objects has been deprecated in favour
of either `delete_one` or `delete_many`.
This change will address 36 deprecation warnings that are generated from
test runs.
Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>
After we merged this PR: https://github.com/openedx/edx-platform/pull/33920
this error began popping up in logs:
Unable to load XBlock 'staffgradedxblock'
....
ImportError: cannot import name 'get_course_blocks' from
partially initialized module 'lms.djangoapps.course_blocks.api'
(most likely due to a circular import) ...
The root cause was the new imports of `derived_key` and `BlockKey` into
xmodule/library_content_block.py. Those new imports come from
xmodule/modulestore/store_utilities.py, which runs
`XBlock.load_classes()` at the module level, which fails because we are
still in the process of loading xmodule/library_content_block.
As a solution, we move both `derived_key` and `BlockKey` to
xmodule/util/keys.py. We could potentially move that file to opaque-keys
eventually, depending on how well we think that those concepts generalize.
Also:
* We rename the function from derived_key to derive_key, as
functions should be verbs.
* We combine the first to parameters of derive_key (a source ContextKey
and a source BlockKey) into a single parameter (a source UsageKey). In
my opinion, this makes the function call easier to understand.
Refactors and reworks the LibraryContentBlock so that its
sync-from-library operations are asynchronous and work with
V2 content libraries. This also required us to make
library_content block duplication asynchronous, as that
involves syncing from the source library.
For the sake of clarity, this PR includes two major method renames:
* update_children(...) -> sync_from_library(...)
* refresh_library(...) -> sync_from_library(upgrade_to_latest=True, ...)
an an XBlock HTTP handler rename:
/refresh_children -> /upgrade_and_sync
There are still a couple issues with import or duplication
of library_content blocks referencing V2 libraries other than
latest. These will be resolved in an upcoming PR.
Part of: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3820617729/Spec+Memo+Content+Library+Authoring+Experience+V2
Follow-up work: https://github.com/openedx/edx-platform/issues/33640
Co-authored-by: Connor Haugh <chaugh@2u.com>
Co-authored-by: Eugene Dyudyunov <evgen.dyudyunov@raccoongang.com>
This PR adds a management command to, given a mapping of V1 content libraries to matching v2 content libraries, replaces references to v1 libs in courses (in library source xblocks) with V2 libraries. It does so by manipulating the mongo document directly.
it also offers some improvements to the management command which copies all v1 libraries into v2 libraries.
-> bind_for_student() and friends change the user_id
-> DraftVersioningModuleStore.update_item() mutates .location which mutates scope_ids.def_id and scope_ids.usage_id in a way that seems totally wrong
This makes a couple of changes to the xblock handler in the CMS. These changes
add a handful of utility functions and modify the existing ones to make reuse
of existing blocks easier. With these changes, it is possible to copy an
entire section from one course to another, and then later refresh that section,
and all of its children, without destroying the blocks next to it.
The existing _duplicate_block function was modified to have a shallow keyword
to avoid copying children, and the update_from_source function was added to
make it easy to copy attributes over from one block to another. These functions
can be used alongside copy_from_template in the modulestore to copy over blocks
and their children without requiring them to be within any particular container
(other than a library or course root)-- thus allowing library-like inclusion
without the library content block. This is especially useful for cases like
copying sections rather than unit content.
* feat: Implement paste button
* chore: improve docs and add tests for python API
* fix: drive-by fix to use a better API for comparing XML
* feat: track which XBlock something was copied from
* feat: add tests
* feat: enable import linter so content_staging's public API is respected
* fix: error seen when trying to paste drag-and-drop-v2 blocks
* fix: use strip_text=True consistently for XML comparisons
* refactor: rename get_user_clipboard_status to get_user_clipboard
* feat: Better error reporting when pasting in Studio
* chore: convert new test suite to pytest assertions
* refactor: push READY status check into the API per review suggestion
* fix: use strip_text=True consistently for XML comparisons
* fix: store "copied_from_block" as a string to avoid Reference field issues
* fix: minor lint error
* refactor: move data types to data.py per OEP-49
YT: https://youtrack.raccoongang.com/issue/EDX_BLND_CLI-87
- V2 libraries are available for selection in the Random Block edit modal;
- selected V2 library blocks are copied to the modulestore and saved as children of the Random Block;
- V2 library version validation works the same as for the V1 libraries (with possibility to update block with the latest version);
- filtering by problem type can't be done for V2 the same as for V1 because the v2 library problems are not divided by types;
- the problem type field is hidden for v2 libraries in the edit mode;
- unit tests added/updated.
We have found that for many courses on edx.org, the "current version"
of the course that's stored in MongoDB doesn't match what's stored in
MySQL. Although this doesn't directly cause any problems as the
"current version" (course index) is not normally read from MongoDB,
it's not supposed to happen - and it can cause problems if some other
extra-platform components (like a pruner script) read the course
indexes out of MongoDB.
We still aren't sure what can cause MongoDB and MySQL to get out of
sync in the first place; this won't necessarily fix that issue. What
this does fix is a bug that seems to be: once they get out of sync,
they stay out of sync and mongo will stop receiving writes.
The Mongo code in most cases will only write a new record if the
"current" record's last_update matches the last_update value before
the change was made. e.g. last_update is "10am", user makes a change,
Mongo gets updated only if the current row's last_update is still
"10am". Otherwise it's considered a "collision" and silently ignored.
Once Mongo and MySQL somehow become out of sync, they may stay out of
sync because any new writes will have a last_update value read from
MySQL, which is newer than the value in MongoDB, so the MongoDB writes
will all be rejected as "collisions".
This should fix the issue by making mongo writes always match the MySQL
writes, instead of letting the Mongo code "decide on its own" when to
write course index updates or not.
