We migrated the source of truth for what the active draft and published
versions of course and v1 library content are to the
SplitModulestoreCourseIndex Django model. But the contentpruning
code (structures.py) that was developed in tubular and will be moved to
edx-platform is not aware of this newer model, and still only pulls its
source of truth from MongoDB. So we *must* continue to do writes to
MongoDB, or the pruning code will start pruning live versions.
The longer term fix for this is to make the pruning code aware of
SplitModulestoreCourseIndex, which will be easier once it's moved into
edx-platform.
See discussion here: https://github.com/overhangio/tutor/pull/984
This is a breaking change that will explicitely set the timeout of
course structure cache entries to 1 week, instead of being unlimited. If
you wish to revert to the former behaviour, you should set
`course_structure_cache["TIMEOUT"] = None`.
The course structure cache keys were previously set with an explicit
timeout of `None` which was overriding the cache default timeout of 2h.
This was OK in environments where the cache is configured with a maximum
memory limit and an automatic key eviction strategy. But in others (such
as Tutor), the course structure cache could grow infinitely.
It was agreed that course structure cache keys should be long-lived but
should respect the default cache structure timeout. Thus, we set here
the TTL to 1 week.
We can also configure Tutor to use a cache eviction policy. But that
means we need to set a `maxmemory` value in Redis. It's not possible to
set a value that will be appropriate for everyone:
- if it's higher than the total memory (e.g: in small instances), server
will crash before the cache is filled.
- if it's too low (e.g: in large instances), the whole platform will abruptly
slow down as many cache entries are suddenly evicted.
That question of whether Tutor should define a cache eviction policy is
still under discussion, but it can be resolved independently of this
change.
`remove` method of Collection objects has been deprecated in favour
of either `delete_one` or `delete_many`.
This change will address 36 deprecation warnings that are generated from
test runs.
Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>
We have found that for many courses on edx.org, the "current version"
of the course that's stored in MongoDB doesn't match what's stored in
MySQL. Although this doesn't directly cause any problems as the
"current version" (course index) is not normally read from MongoDB,
it's not supposed to happen - and it can cause problems if some other
extra-platform components (like a pruner script) read the course
indexes out of MongoDB.
We still aren't sure what can cause MongoDB and MySQL to get out of
sync in the first place; this won't necessarily fix that issue. What
this does fix is a bug that seems to be: once they get out of sync,
they stay out of sync and mongo will stop receiving writes.
The Mongo code in most cases will only write a new record if the
"current" record's last_update matches the last_update value before
the change was made. e.g. last_update is "10am", user makes a change,
Mongo gets updated only if the current row's last_update is still
"10am". Otherwise it's considered a "collision" and silently ignored.
Once Mongo and MySQL somehow become out of sync, they may stay out of
sync because any new writes will have a last_update value read from
MySQL, which is newer than the value in MongoDB, so the MongoDB writes
will all be rejected as "collisions".
This should fix the issue by making mongo writes always match the MySQL
writes, instead of letting the Mongo code "decide on its own" when to
write course index updates or not.
- Moving xmodule folder to root as we're dissolving sub-projects of common folder in edx-platform
- More info: https://openedx.atlassian.net/browse/BOM-2579
- -e common/lib/xmodule has been removed from the requirements as xmodule has itself become the part of edx-platform and not being installed through requirements
- The test files common/lib/xmodule/test_files/ have been removed as they are not being used anymore
