The only way to access the legacy courseware is now through the
Studio preview feature (and at some point, when the MFE supports a
preview mode, we can then remove even that).
This drops the courseware.use_legacy_frontend waffle.
edx/edx-platform#24365 has changed the completion mode of these blocks.
Before Koa, it was sufficient to view the block to get a completion checkmark.
Since Koa, all children of the block must be completed.
This adds a toggle to change the completion behavior back to the previous one
so that the user experience can be consistent if needed.
Description
This is a follow up to #29058 and #29413. This is the next step in moving part of the modulestore data (the course indexes / "active versions" table) from MongoDB to MySQL.
There are four steps planned in moving course index data to MySQL:
Step 1: create the tables in MySQL, start writing to MySQL + MongoDB ✅ done
Step 2: migrate all remaining courses to MySQL ✅ done
Step 3: switch reads from MongoDB to MySQL (this PR)
Step 4 (much later, once we know this is working well): stop writing to MongoDB altogether.
Supporting information
OpenCraft Jira ticket: MNG-2557
Status
✅ Tested with a large Open edX instance is in progress.
Testing instructions
Try making changes in Studio and verify that they work fine.
Deadline
None
Convert more tests from MONGO_AMNESTY to SPLIT modulestores.
This is in preparation for just wholesale denying access to Old
Mongo, so I either converted tests to split or just deleted some
test variants that were Old Mongo specific. (e.g. ddt lines)
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
This commit starts writing course indexes (active_versions) to both MySQL and Mongo, but continues to read from MongoDB only.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
Does 3 things:
(1) Use django for modulestore tests
(2) Use normal LMS settings for modulestore tests instead of openedx/tests/settings.py
(3) Simplify some TestCase subclasses by converting them to use ModuleStoreTestCase
Details and rationale:
(1) Currently parts of the modulestore test suite are designed to run "without django", although there is still a lot of django functionality imported at times, and many of the tests do in fact use django. But for the upcoming PR #27565 (moving split's course indexes from MongoDB to MySQL), we will need to always have Django enabled. So this commit paves the way for that change.
(2) The previous tests that did use Django used a special settings file, openedx/tests/settings.py which made some debugging confusing because those tests had quite different django settings than other tests. This change deletes that file and runs the tests using the LMS test settings.
(3) The test suite also contains many different ways of initializing and testing a modulestore, with significant differences in their configuration, and also a lot of repetition. I find this makes understanding, debugging and writing tests more difficult. So this commit also reduces the number of different "test case using modulestore" base classes:
* Simplifies MixedWithOptionsTestCase and MixedSplitTestCase by making them simple subclasses of ModuleStoreTestCase.
* Removes PureModulestoreTestCase.
Deprecated edx-platform import paths
(for example, `student` instead of
`common.djangoapps.student`) currently raise
warnings when used. We want to fully remove
support for those paths.
As an easily reversible way to initially remove
support, we add a new setting to LMS and Studio
called `ERROR_ON_DEPRECATED_EDX_PLATFORM_IMPORTS`,
defaulting to False. We set it to True for devstack
and will set it to True in Stage and Production
soon. If critical errors occur, we can easily
flip the setting back to False.
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
Some tests were still relying on deprecated
WaffleSwitchNamespace.override method, which was working before because
we were importing WaffleSwitchNamespace from waffle_utils.__init__. This
no longer works after we import WaffleSwitchNamespace from edx-toggles.
The old folder name is somewhat confusing, because the
folder contains shims to _compensate for the removal
of sys.path hacks_, but does not contain the sys.path
hacks themselves.
Furthermore, this import_shims/ system could also be
used for other import path changes, such as turning
the locally-installed packages in common/lib/
into regular, importable modules
(e.g. `from common.lib.xmodule import abc` instead of
`from xmodule import abc`). So, a name that is not
specific to the sys.path hacks may be better
in the medium-to-long term.
Along the same lines, we also rename SysPathHackWarning
to DeprecatedEdxPlatformImportWarning.
This change will prevent Library Content from being marked as
complete on view and the corresponding version bump to
edx-completion contains code that will start looking at the
children of the library content for completeness.
Fixing 56 GuessedAtParserWarnings, in commit edx#24098
Background: BeautifulSoup automatically picks the fastest parser available. By default, it picks the "lxml" parser.
Per the [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser) documentation:
> Beautiful Soup supports the HTML parser included in Python’s standard library, but it also supports a number of third-party Python parsers. One is the lxml parser. Depending on your setup, you might install lxml with one of these commands.
> Another alternative is the pure-Python html5lib parser, which parses HTML the way a web browser does.
Context: We changed two statements, one in lms and another in openedx. Both statements fire up BeautifulSoup. Now we explicitly ask for "lxml," following the recommendation on BeautifulSoup's documentation:
> If you can, I recommend you install and use lxml for speed. If you’re using a very old version of Python – earlier than 2.7.3 or 3.2.2 – it’s essential that you install lxml or html5lib. Python’s built-in HTML parser is just not very good in those old versions.
Before:
`soup = BeautifulSoup(content)`
After:
`soup = BeautifulSoup(markup=content, features="lxml")`
The warnings are gone, tests are passing in local.
1. Created a new celery queue with key `SOFTWARE_SECURE_VERIFICATION_ROUTING_KEY`.
2. Added a celery task with retry logic.
3. sorted imports with isort.
4. Changed deprecated `log.warn` => `log.warning`.
