Call into the exam service instead of the edx-proctoring plugin on course publish if the course_apps.exams_ida course waffle flag is enabled. This is an early step in moving away from edx-proctoring
This commit removes code that was used to copy Old Mongo courses into
new Split Mongo courses. This includes both the migrate_to_split
management command, as well as the backend code that would be invoked
to re-run Old Mongo courses as Split courses using Studio (the UI for
this was already removed in b429e55c).
This is a part of the Old Mongo removal effort tracked in:
https://github.com/openedx/public-engineering/issues/62
This removes user-facing Studio edit support for Old Mongo courses
(courses that have a CourseKey of the format {org}/{course}/{run}).
This does not affect our normal courses, which have CourseKeys
starting with "course-v1:".
After this commit:
* Old Mongo courses will continue to appear on the Studio course
listing page, but are not clickable.
* Any attempt to directly access an Old Mongo course in Studio via URL
fail with a 404 error.
* Course certificates will still be available for Old Mongo courses.
* Old Mongo courses will continue to be returned by CourseOverviews
and get_course_summaries() calls.
We decided against removing Old Mongo courses from the listing entirely
because that would require very expensive CourseOverviews query to
filter them out. Making that query more efficient would involve a
database migration to add appropriate indexing, which is something else
that we are looking to avoid. CourseOverviews are used everywhere in
the system, so we want to avoid changing how they work so that we can
minimize risk.
This is part of the Old Mongo Modulestore deprecation effort:
https://github.com/openedx/public-engineering/issues/62
* fix: studio registration using the LMS SSO
Add the social-core settings:
```
INACTIVE_USER_LOGIN = True
INACTIVE_USER_URL = 'http://localhost:18010'
```
Change the registration link's `next` parameter to trigger SSO login
after the registration.
Improving Studio homepage performance for users with course access role with no course_id
Fixing unit tests
Added create CourseOverviewFactory after creating course to course listing test
Fix order import for `from openedx.core.djangoapps.content.course_overviews.tests.factories import CourseOverviewFactory
`
(cherry picked from commit 997a0ff770744309f0ee84f3c0696a80310c5f2d)
Remove temporary FutureCourseWaffleFlag class;
Update ora2 and edx-toggles to versions cleaned from the
LegacyWaffle* classes;
Replace `override_flag`s with `override_waffle_flag`;
Replace `override_switch`s with `override_waffle_switch` (where it's possible).
This is a first stage for removing the LegacyWaffle* classes.
LegacyWaffleFlag usage replaced with WaffleFlag;
LegacyWaffleSwitche usage replaced with WaffleSwitch;
New CourseWaffleFlag added to the temporary module __future__ as FutureCourseWaffleFlag;
Updated all the imports to use CourseWaffleFlag from the __future__ module;
BREAKING CHANGE: A number of toggle related constants (e.g. ENABLE_ACCESSIBILITY_POLICY_PAGE)
changed types. They were strings, and are now toggle instances (e.g. WaffleSwitch). Although the entire
refactor should be self-contained in edx-platform, if any plugins or dependencies were directly
using these constants, they will break. If this is the case, try to find a better publicized way of
exposing those toggles.
Description
This is a follow up to #29058 and #29413. This is the next step in moving part of the modulestore data (the course indexes / "active versions" table) from MongoDB to MySQL.
There are four steps planned in moving course index data to MySQL:
Step 1: create the tables in MySQL, start writing to MySQL + MongoDB ✅ done
Step 2: migrate all remaining courses to MySQL ✅ done
Step 3: switch reads from MongoDB to MySQL (this PR)
Step 4 (much later, once we know this is working well): stop writing to MongoDB altogether.
Supporting information
OpenCraft Jira ticket: MNG-2557
Status
✅ Tested with a large Open edX instance is in progress.
Testing instructions
Try making changes in Studio and verify that they work fine.
Deadline
None
`videosequence` and `problemset` have been replaced with `sequential`.
`problemset` and `videosequence` are old-but-not-entirely-unused aliases to the `sequential` block type (in Studio-speak, "Subsection").
Since [these block types have been removed from the 6 courses that used them](https://openedx.atlassian.net/browse/DEPR-151?focusedCommentId=588197), this ticket removes the support for the `problemset` and `videosequence` block-types.
For more information, see ticket: [DEPR-151](https://openedx.atlassian.net/browse/DEPR-151)
Convert more tests from MONGO_AMNESTY to SPLIT modulestores.
This is in preparation for just wholesale denying access to Old
Mongo, so I either converted tests to split or just deleted some
test variants that were Old Mongo specific. (e.g. ddt lines)
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
This commit starts writing course indexes (active_versions) to both MySQL and Mongo, but continues to read from MongoDB only.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
- Update migration instructions
- Changes regarding redirect URLs and cookie domain are to permit the
site to run on multiple domains.
- Set LOGIN_URL in common so that it can be unset in environment overrides
This bypasses the "redirect to LMS" login/signup code, but does not yet
remove it; removal is covered by DEPR-166 so that this remains a
configuration-only change for now.
There should have no user-visible effect.
ref: ARCHBOM-1890
Current State (before this commit):
Studio, as of today doesn't have a way to restrict a user to
create a course in a particular organization. What Studio
provides right now is a CourseCreator permission which gives
an Admin the power to grant a user the permission to create
a course.
For example: If the Admin has given a user Spiderman the
permission to create courses, Spiderman can now create courses
in any organization i.e Marvel as well as DC.
There is no way to restrict Spiderman from creating courses
under DC.
Purpose of this commit:
The changes done here gives Admin the ability to restrict a
user on an Organization level from creating courses via the
Course Creators section of the Studio Django administration
panel.
For example: Now, the Admin can give the user Spiderman the
privilege of creating courses only under Marvel organization.
The moment Spiderman tries to create a course under some
other organization(i.e DC), Studio will show an error message.
This change is available to all Studio instances that
enable the FEATURES['ENABLE_CREATOR_GROUP'] flag.
Regardless of the flag, it will not affect any instances that choose
not to use it.
BB-3622
Does 3 things:
(1) Use django for modulestore tests
(2) Use normal LMS settings for modulestore tests instead of openedx/tests/settings.py
(3) Simplify some TestCase subclasses by converting them to use ModuleStoreTestCase
Details and rationale:
(1) Currently parts of the modulestore test suite are designed to run "without django", although there is still a lot of django functionality imported at times, and many of the tests do in fact use django. But for the upcoming PR #27565 (moving split's course indexes from MongoDB to MySQL), we will need to always have Django enabled. So this commit paves the way for that change.
(2) The previous tests that did use Django used a special settings file, openedx/tests/settings.py which made some debugging confusing because those tests had quite different django settings than other tests. This change deletes that file and runs the tests using the LMS test settings.
(3) The test suite also contains many different ways of initializing and testing a modulestore, with significant differences in their configuration, and also a lot of repetition. I find this makes understanding, debugging and writing tests more difficult. So this commit also reduces the number of different "test case using modulestore" base classes:
* Simplifies MixedWithOptionsTestCase and MixedSplitTestCase by making them simple subclasses of ModuleStoreTestCase.
* Removes PureModulestoreTestCase.
Some actions in split modulestore record the user ID who requested the action. Currently, split modulestore doesn't care what data type you use for those user IDs. Most of the codebase uses integers, but some tests used username or email address strings.
My upcoming PR #27565 will move split modulestore's "course index" data from MongoDB into MySQL. In doing so, it requires that user IDs are always numeric. So this PR paves the way for that one by using numeric IDs consistently in all test cases. I believe the actual non-test code was already consistently using integer IDs.
Instead of having json errors in transcript acquisition and conversion cause errors, have transcription conversion and acquisition simply return an error message in the transcription which can prompt a change from the user.
Although not uploading a transcript is handled, transcripts can often cause errors in edit, export, and other activities due to json errors. These errors block the entire use of these features, so to allow for reupload, etc, we add an error message instead of transcript and log the event.
In response to [TNL-8539](https://openedx.atlassian.net/secure/RapidBoard.jspa?rapidView=580&projectKey=TNL&modal=detail&selectedIssue=TNL-8539)
Testing: Unit tests coverage is included in the PR. Upload, import, and export of courses with transcriptions is also easily hand-testable. Just create a video in studio, add an irrelevant transcript. Then try to import, export, and edit the problem. Expected behavior is success.
This PR fixes Course Outline DAGs with duplicate sequences. Previously
when a course outline had duplicate sequences, the outline would not generate
and raise a ValueError. There were no checks for duplicate sequences
before the generation of the course outline because it is not possible
to create duplicate sequences in Studio, but is possible when a Course
Author imports a course. Now before the course outline is generated, it
will be checked for duplicate sequences. If a duplicate sequence is
found an error will be logged for Partner Support to see in the Django
Admin and the duplicate will be deleted. This change will impact the
Course Author.
Code in ./common/lib/xmodule/xmodule should
be imported as `from xmodule`, since `xmodule`
is a locally-installed package.
This is weird, but as long as it is the case,
we should be consistent.
(In BOM-2584, I propose moving the files to
./xmodule, which would quell this confusion.)
This commit addresses MST-793. In MST-757, exam registration was moved to an asychronous task to improve the performance of saving content in Studio. This feature was gated behind a waffle flag, ENABLE_ASYNC_REGISTER_EXAMS, to avoid interrupting course teams during the testing and roll out of this feature.
This feature has been tested and rolled out, and so the aforementioned waffle flag is ready to be removed.
The effect of this change is that asynchronous exam registration will no longer be gated by the aforementioned waffle flag, and this feature will be rolled out across all of the edX platform.
MST-362. Previously instructors were unable to modify the zendesk ticket field, although that field should change based on the proctoring provider. This backend change allows course instructors to modify the field so that it is appropriate for whateverproctoring provider they choose.
Modify the existing login api in a way that
it will allow the user to login via username as well.
currently it is only allowing email to log the user in.
VAN-445
In dfb80acc1, we allowed group_access settings to bubble up from
Units to Sequences in the case where all Units had the same group
settings for a particular user partition. This caused an issue
where the group being bubbled up could be empty if the Unit was
misconfigured in OLX (this is already caught if done directly on
the Sequence). This commit fixes things so that empty partition
group settings never bubble up.
We don't want empty partition groups to bubble up because it would
mean that no groups are allowed to access the Sequence, which is
probably an error. (There is a separate visible_to_staff_only flag
when you want to hide content from all non-staff users.)
NOTE: This will require a forced backfill of course outlines to update
the course content data in learning_sequences:
python manage.py cms backfill_course_outlines --force
Without this backfill, the learning_sequences API will continue to serve
stale content data that has no user partition group data. It won't cause
errors, but it won't do the exclusions properly.
Commit summary:
* Created EnrollmentTrackPartitionGroupsOutlineProcessor to process the
enrollment_track User Partition Group, allowing Sequences and Sections
to be removed based on their group_access settings.
* Added user_partition_groups attribute to CourseLearningSequenceData
and CourseSectionData in learning_sequences/data.py, along with
backing model data.
* get_outline_from_modulestore now extracts group_access settings from
Sections and Sequences. It also bubbles up group_access settings from
Units, meaning that if a Sequence with no group_access setting has
Units that are all set to show only to the Verified enrollment track,
then the Sequence will only show to the Verified enrollment track.
This commit adds model-level support for all user partition groups by
capturing all the content group associations (group_access), but it only
implements the code checks for the enrollment track partition. It's not
clear that we want to generalize, since there's only one other partition
type (A/B testing) that is applicable at the outline level.
It's important to note that there is no way to set the group_access for
a Section or Sequence in Studio today. It's only possible by direct
editing of the OLX for import. That being said, the block structures
framework supports applying course groups at this level, and this commit
moves learning_sequences closer to feature parity.
The bubbling up from Units to the parent Sequence was done to mitigate
confusion when a Sequence is entirely composed of Units that are not
visible to the user because of content group restrictions. It's not
clear whether this is something we want to do in the long term, since it
would simplify the code to always specify group_access at the Sequence
level. This first pass is done partially to collect better data about
places in our courses where this kind of usage is already happening.
Most of the EnrollmentTrackPartitionGroupsOutlineProcessor code and its
tests were written by @schenedx.
Synchronously registering proctored exams while saving content to studio is causing a significant slow down. The function that registers the exams has been moved to an async task. In addition, a signal handler on_course_publish has also been moved to the async task, as it relies on exam registration being complete before being executed.
