It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
This commit starts writing course indexes (active_versions) to both MySQL and Mongo, but continues to read from MongoDB only.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
This commit removes several waffle toggles that have been enabled
on edx.org for years. It's time to remove the rollout gating for
these features and enable them by default.
This doesn't directly change any behavior. But it does create new
database objects by default now and allows for enabling other
schedule based features more easily.
Specifically, the following toggles were affected.
schedules.create_schedules_for_course
- Waffle flag removed as always-enabled
- We now always create a schedule when an enrollment is created
schedules.send_updates_for_course
- Waffle flag removed as always-enabled
- Course update emails are sent as long as the ScheduleConfig
allows it.
- This is not a change in default behavior, because ScheduleConfig
is off by default.
dynamic_pacing.studio_course_update
- Waffle switch removed as always-enabled
- Course teams can now always edit course updates directly in Studio
ScheduleConfig.create_schedules
ScheduleConfig.hold_back_ratio
- Model fields for rolling out the schedules feature
- Schedules are now always created
- This commit only removes references to these fields, they still
exist in the database. A future commit will remove them entirely
This commit also adds a new has_highlights field to CourseOverview.
This is used to cache whether a course has highlights, used to
decide which course update email behavior they get. Previously every
enrollment had to dig into the modulestore to determine that.
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
- Have it load dates from edx-when, not just write to it. This
fixes self-paced courses where edx-when is only place dates are
kept.
- Have it read original date for a homework from edx-when when
resetting a date. This fixes the message it gives the instructor
about whether it was successfully reset.
- Have it recursively set a date, rather than assuming that dates
are only ever set on the subsection layer. This fixes setting
dates on self-paced courses (where dates are set all the way
down) and just in case somebody somewhere edits the course xml
to have a date where it's not expected.
Also added reset_test_case() and @modifies_courseware to SharedModuleStoreTestCase.
Revert "More verbose test builds in Jenkins, for debugging."
This reverts commit 58cade4cc4288335026649470a48b7bbca969ee8.
Original Commit Messages:
use edx's own get_parent method, rather than our own.
add field to unique constraint to avoid MultipleObjectsReturned in case of multiple browser clicks on submit
fix 0011 migration, inherit from TimeStampedField and add composite index (migration only)
fix bug where adding an already registered user to a ccx would cause a crash due to an undefined variable
add assertNumQueries tests to test modules where override field providers are used
remove unnecessary teardown
implement recommended style for checking empty list
import utility methods rather than use duplicate code
added comment explaining date conversion to string for json
add logging for invalid users or emails when enrolling students
add comment about xmodule user state
avoid using get_or_create, which seems to be causing a race condition on schedule change save
relocate badly placed edvent handlers to fix multiple submit problem
individual students, and a reimplementation of the individual due date
feature.
This work introduces an architecture, used with the 'authored_data'
portion of LmsFieldData, which allows arbitrary field overrides to be
made for fields that are part of the course content or settings (Mongo
data). The basic architecture is extensible by means of writing and
configuring arbitrary field override providers.
One concrete implementation of a field override provider is provided
which allows for overrides to be for individual students. This provider
is then used as a basis for reimplementing the individual due date
extensions feature as a proof of concept for the design.
One can imagine writing override providers that provide overrides based
on a student's membership in a cohort or other similar idea. This work
is being done, in fact, to pave the way for the Personal Online Courses
feature being developed by MIT, which will use an override provider very
much long those lines.
