Andal.LND/edx-platform
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: store split modulestore's course indexes in Django/MySQL

Browse Source 
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.

By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
This commit is contained in:
Braden MacDonald
2021-09-29 15:55:52 -07:00
committed by David Ormsbee
parent d20a769f73
commit 96e5ff8dce
52 changed files with 729 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
lms/djangoapps/ccx/tests/test_field_override_performance.py
View File

@@ -271,22 +271,22 @@ class TestFieldOverrideSplitPerformance(FieldOverridePerformanceTestCase):
__test__ = True
# TODO: decrease query count as part of REVO-28
QUERY_COUNT = 33
QUERY_COUNT = 34
TEST_DATA = {
('no_overrides', 1, True, False): (QUERY_COUNT, 3),
('no_overrides', 2, True, False): (QUERY_COUNT, 3),
('no_overrides', 3, True, False): (QUERY_COUNT, 3),
('ccx', 1, True, False): (QUERY_COUNT, 3),
('ccx', 2, True, False): (QUERY_COUNT, 3),
('ccx', 3, True, False): (QUERY_COUNT, 3),
('ccx', 1, True, True): (QUERY_COUNT + 2, 3),
('ccx', 2, True, True): (QUERY_COUNT + 2, 3),
('ccx', 3, True, True): (QUERY_COUNT + 2, 3),
('no_overrides', 1, False, False): (QUERY_COUNT, 3),
('no_overrides', 2, False, False): (QUERY_COUNT, 3),
('no_overrides', 3, False, False): (QUERY_COUNT, 3),
('ccx', 1, False, False): (QUERY_COUNT, 3),
('ccx', 2, False, False): (QUERY_COUNT, 3),
('ccx', 3, False, False): (QUERY_COUNT, 3),
('no_overrides', 1, True, False): (QUERY_COUNT, 2),
('no_overrides', 2, True, False): (QUERY_COUNT, 2),
('no_overrides', 3, True, False): (QUERY_COUNT, 2),
('ccx', 1, True, False): (QUERY_COUNT, 2),
('ccx', 2, True, False): (QUERY_COUNT, 2),
('ccx', 3, True, False): (QUERY_COUNT, 2),
('ccx', 1, True, True): (QUERY_COUNT + 3, 2),
('ccx', 2, True, True): (QUERY_COUNT + 3, 2),
('ccx', 3, True, True): (QUERY_COUNT + 3, 2),
('no_overrides', 1, False, False): (QUERY_COUNT, 2),
('no_overrides', 2, False, False): (QUERY_COUNT, 2),
('no_overrides', 3, False, False): (QUERY_COUNT, 2),
('ccx', 1, False, False): (QUERY_COUNT, 2),
('ccx', 2, False, False): (QUERY_COUNT, 2),
('ccx', 3, False, False): (QUERY_COUNT, 2),
}

6
lms/djangoapps/ccx/tests/test_models.py
View File

@@ -46,7 +46,7 @@ class TestCCX(ModuleStoreTestCase):
def test_ccx_course_caching(self):
"""verify that caching the propery works to limit queries"""
with check_mongo_calls(3):
with check_mongo_calls(2):
# these statements are used entirely to demonstrate the
# instance-level caching of these values on CCX objects. The
# check_mongo_calls context is the point here.
@@ -72,7 +72,7 @@ class TestCCX(ModuleStoreTestCase):
"""verify that caching the start property works to limit queries"""
now = datetime.now(utc)
self.set_ccx_override('start', now)
with check_mongo_calls(3):
with check_mongo_calls(2):
# these statements are used entirely to demonstrate the
# instance-level caching of these values on CCX objects. The
# check_mongo_calls context is the point here.
@@ -97,7 +97,7 @@ class TestCCX(ModuleStoreTestCase):
"""verify that caching the due property works to limit queries"""
expected = datetime.now(utc)
self.set_ccx_override('due', expected)
with check_mongo_calls(3):
with check_mongo_calls(2):
# these statements are used entirely to demonstrate the
# instance-level caching of these values on CCX objects. The
# check_mongo_calls context is the point here.