Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
This test was failing for Django 3 and during investigation it found that it isn't working as per expected due to introduction of caching. So fixed the test case to avoid caching to mimic race condition.
BOM-2799
course outline, which don't need block dates below the subsection level of a course.
Pass the course's published version to all the appropriate places where edx-when's API
is called - to allow edx-when to more efficiently cache queried/processed results.
TNL-8061
Reverting because of errors seen in proctoring. The error from splunk:
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/edx_proctoring/views.py", line 590, in get
data['onboarding_release_date'] = effective_start.isoformat()
AttributeError: 'NoneType' object has no attribute 'isoformat'
The effective date is coming from learning_sequences, which ultimately
comes from edx-when.
course outline, which don't need block dates below the subsection level of a course.
Pass the course's published version to all the appropriate places where edx-when's API
is called - to allow edx-when to more efficiently cache queried/processed results.
TNL-8061
* Add unsubscribe_token uuid field to CourseGoal model
* Add endpoint to unsubcribe from just a token (no login needed)
* Add admin page for the course_goals djangoapp
* Add get_course_overview_or_404 utility method
* Clean up URL handling in course_home_api
AA-907
feat: reimagine certificate display settings
The course settings `certificate_available_date` (CAD) and
`certificates_display_behavior` (CDB) were previously
acting indedependantly of one another. They now work in
tandem. This change:
- limits CDB to a dropdown
- removes "early_with_info" and adds "end_with_date"
- only takes CAD into account if "end_with_date" is selected
- Moves CDB to the main course schedule settings page
- updates CourseOverview model and CourseDetails objects to
validate these fields and choose sane defaults if they aren't
expected values
This work was previously done in bd9e7dd (complete with bugs), so this
version is toggleable via the ENABLE_V2_CERT_DISPLAY_SETTINGS setting
We control rollout of the Learning Sequences REST API with the
USE_FOR_OUTLINES (learning_sequences.use_for_outlines) course waffle
flag. But sometimes we're going to want to be able to take a look at
production data output of this API before explicitly turning the API on
(and switching over to it) for the Courseware MFE. To do this, we're
going to make the availability base itself purely on the waffle flag and
course type (Old Mongo courses are not supported), and allow people to
"peek" at what the API would have given if it was rolled out by passing
an explicit "force_on=1" querystring param.
This is work to support the rollout of TNL-8330.
A bug was reported of a learner accessing content that should have been hidden
due to the hide after due setting in Studio by the course team by clicking on
the Progress tab. This takes into account that value and will now hide the URL
on the Progress tab in that scenario.
The `may_certify` function is used to determine whether a learner should
be able to see that their certificate is ready. It is therefore entirely
in the certificates domain. This remove may_certifiy and
may_certify_for_course from the xmodule and course_overview
apps and into the certificates app. The xmodule may_certify was not called
outside of tests prior to this, and the other's were easily moved.
In order to avoid circular imports, this also moved
certificate_info_for_user out of the model code and into an API
This adds support for course staff masquerading as any other user when
viewing their own course, via user={username} querystring parameter.
Rules:
* Anonymous users are repesented by a blank "user" value.
* If there is no "user" parameter at all, render for the user making the
request.
* Global staff can view any course as any user.
* Course staff and instructors can view their own course as any user
(including other staff, the anonymous user, or users not enrolled in
their course).
This commit supports TNL-8330 (switching the frontend-app-learning MFE
to use the Learning Sequences API instead if Course Blocks).
and CourseSectionSequence to ensure that cascading deletes will occur to delete
the relation upon deletion of a foreign-keyed object.
This commit is phase 3 of 4 in order to ensure a smooth deploy. The phases:
1) Add separate through models for user partition groups, add fields to point to the
separate models, and start writing to those fields as well. (COMPLETE)
2) After all models have been re-generated, switch code over to use separate through
model fields backed by the separate through models *and* stop writing to the original
ManyToManyField field. (COMPLETE)
3) After phase 2 is deployed smoothly, remove the original ManyToManyField fields. (This PR!)
4) After phase 3 is deployed smoothly, rename the new fields to have the same name
as the old fields, so new_user_partition_groups -> user_partition_groups.
This PR is designed to be able to easily be rolled-back if anything goes wrong.
[MICROBA-1289]
* Remove use of the CourseOverview model directly in the Certificates app
* Introduce a few new Python API functions in the `course_overview` and `catalog` django apps to replace functionality in the Certificates app.
and CourseSectionSequence to ensure that cascading deletes will occur to delete
the relation upon deletion of a foreign-keyed object.
This commit is phase 2 of 3 in order to ensure a smooth deploy. The phases:
1) Add separate through models for user partition groups, add fields to point to the
separate models, and start writing to those fields as well. (COMPLETE)
2) After all models have been re-generated, switch code over to use separate through
model fields backed by the separate through models *and* stop writing to the original
ManyToManyField field. (This PR!)
3) After phase 2 is deployed smoothly, remove the original ManyToManyField fields
and rename the new fields to have the same name as the old fields.
This PR is designed to be able to easily be rolled-back if anything goes wrong.
The course settings `certificate_available_date` (CAD) and
`certificates_display_behavior` (CDB) were previously acting indedependantly
of one another. They now work in tandem. This change:
- limits CDB to a dropdown
- removes "early_with_info" and adds "end_with_date"
- only takes CAD into account if "end_with_date" is selected
- Moves CDB to the main course schedule settings page
- updates CourseOverview to validate these fields and choose sane
defaults if they aren't expected values
Certificates will now show under the following circumstances:
"Immediately upon passing"
certificate_availability_date = null
certificates_display_behavior = "early_no_info"
"End date of course"
certificate_availability_date = null
certificates_display_behavior = "end"
"A date after the course end date"
certificate_availability_date = <date>
certificates_display_behavior = "end_with_date"
and CourseSectionSequence to ensure that cascading deletes will occur to delete
the relation upon deletion of a foreign-keyed object.
This commit is phase 1 of 3 in order to ensure a smooth deploy. The phases:
1) Add separate through models for user partition groups, add fields to point to the
separate models, and start writing to those fields as well.
2) After all models have been re-generated, switch code over to use separate through
model fields backed by the separate through models.
3) After phase 2 is deployed smoothly, remove the original ManyToManyField fields
and rename the new fields to have the same name as the old fields.
* feat: Course Apps API
This adds a new concept called course apps. These are exposed via a new
"openedx.course_app" entrypoint, which helps the LMS and studio discover such
apps and list them in a new rest api for the same.
These course apps will drive the pages and resources view in the course authoring
MFE. This system will track which apps are enabled and which are disabled. It
also allows third-party apps to be listed here by using the plugin entrypoint.
* Apply feedback from review
A common usage pattern is to make an exam that is restricted to a
particular enrollment track. Since there is no UI in Studio to do this
at the sequence/subsection level, course authors instead restrict every
unit in the sequence to that track (e.g. "verified"). The legacy
courseware experience could handle this, but the Course Blocks API does
not. It is likely that we'll want to permit toggling this at the
sequence level, but regardless of whether we do that going forward, we
still have to deal with this kind of content data in existing courses.
An entirely empty sequence is useless, and currently breaks the
Courseware MFE browsing experience. This commit introduces the first
part of that fix, invoking the new Learning Sequences API to remove
sequences that the user shouldn't be allowed to know about. The plan is
to switch over the entire course outline from the Course Blocks API to
the Learning Sequences API, and this is the first small step in that
direction.
This also creates CourseWaffleFlag learning_sequences.use_for_outlines
to control the gradual rollout of this feature. This will start as a
very limited rollout to address courses that show this specific bug
(TNL-8377).
New learning_sequences public API call: public_api_available
Code in ./common/lib/xmodule/xmodule should
be imported as `from xmodule`, since `xmodule`
is a locally-installed package.
This is weird, but as long as it is the case,
we should be consistent.
(In BOM-2584, I propose moving the files to
./xmodule, which would quell this confusion.)
In order to upgrade to Django > 2.2.20, we can't continue to use
absolute paths in the block_structure's data FileField. This used to
work for S3, but it will not work going forward due to a security fix
in Django 2.2.21.
This data migration will remove the starting '/' from any paths in the
block_structure table. The resulting locations in S3 should be
unaffected.
The edx-platform codebase already includes quite a few type annotations, but
they were not regularly checked. This may cause problems, when the annotations
themselves include errors (as we found out in some of the learning_sequences
annotations). So here, we add mypy as a dev requirement and introduce a make
command to run mypy regularly. Mypy runs on a very small portion of the total
edx-platform, as configured in mypy.ini. Our hope is that developers will add
more and more modules to this configuration file, until we can eventually run
mypy on the full code base.
See discussion: https://discuss.openedx.org/t/dev-notes-running-mypy-on-edx-platform/4860
In DE-1822, we believed we needed to switch to start_date and end_date.
It was determined this was not the case, so this updates the comment
to ensure future users use the correct fields (start and end) and
updates any pieces of code that may have used start_date or end_date.
We already had function traces to see how long each processor took when
generating a UserCourseOutlineData. This adds a similar tracking around
making a UserCourseOutlineDetailsData, to help track down performance
issues observed in what I think is the special exams processor.
[MICROBA-1178]
- remove modulestore usage in `generation_handler.py`
- add duplicate functions that utilize a CourseKey or CourseOverview to remove dependence on modulestore (this will be cleaned up (if possible) at a later part of this refactor)
- add python API function to `content`/`course_overview` app that will retrieve a single CourseOverview (rather than a serialized list of dicts of CourseOverview data)
NOTE: This will require a forced backfill of course outlines to update
the course content data in learning_sequences:
python manage.py cms backfill_course_outlines --force
Without this backfill, the learning_sequences API will continue to serve
stale content data that has no user partition group data. It won't cause
errors, but it won't do the exclusions properly.
Commit summary:
* Created EnrollmentTrackPartitionGroupsOutlineProcessor to process the
enrollment_track User Partition Group, allowing Sequences and Sections
to be removed based on their group_access settings.
* Added user_partition_groups attribute to CourseLearningSequenceData
and CourseSectionData in learning_sequences/data.py, along with
backing model data.
* get_outline_from_modulestore now extracts group_access settings from
Sections and Sequences. It also bubbles up group_access settings from
Units, meaning that if a Sequence with no group_access setting has
Units that are all set to show only to the Verified enrollment track,
then the Sequence will only show to the Verified enrollment track.
This commit adds model-level support for all user partition groups by
capturing all the content group associations (group_access), but it only
implements the code checks for the enrollment track partition. It's not
clear that we want to generalize, since there's only one other partition
type (A/B testing) that is applicable at the outline level.
It's important to note that there is no way to set the group_access for
a Section or Sequence in Studio today. It's only possible by direct
editing of the OLX for import. That being said, the block structures
framework supports applying course groups at this level, and this commit
moves learning_sequences closer to feature parity.
The bubbling up from Units to the parent Sequence was done to mitigate
confusion when a Sequence is entirely composed of Units that are not
visible to the user because of content group restrictions. It's not
clear whether this is something we want to do in the long term, since it
would simplify the code to always specify group_access at the Sequence
level. This first pass is done partially to collect better data about
places in our courses where this kind of usage is already happening.
Most of the EnrollmentTrackPartitionGroupsOutlineProcessor code and its
tests were written by @schenedx.
