In https://github.com/edx/edx-platform/pull/25955 `HiddenDescriptor`
(which was a subclass of `RawDescriptor` with a custom `student_view()`)
was converted to an XBlock. It is used as the `default_class` by the
`CachingDescriptorSystem` classes. However `RawDescriptor` is still
being used by `XMLModuleStore`. This has been replaced by
`HiddenDescriptor` as well.
= IMPORTANT WARNING =
This can be a VERY EXPENSIVE MIGRATION which may take hours or
days to run depending on the size of the courseware_studentmodule
table on your site. Depending on your database, it may also lock
this table, causing courseware to be non-functional during that
time.
If you want to run this migration manually in a more controlled
way (separate from your release pipeline), the SQL needed is:
CREATE INDEX `courseware_stats` ON `courseware_studentmodule`
(`module_id`, `grade`, `student_id`);
You can then fake the migration:
https://docs.djangoproject.com/en/2.2/ref/django-admin/#cmdoption-migrate-fake
= Motivation and Background =
TLDR: This adds an index that will speed up reports like the
Problem Grade Report. This fixes a performance regression that
was unintentionally introduced in 25da206c.
I'm capturing the entire saga below, in case Open edX operators
need to dig into it.
The tale begins in November of 2012 (yes, seriously). We had an
inline analytics feature that would display a histogram to course
staff by each problem in the LMS, detailing how students did on
that problem (e.g. 80% got 2 points, 10% got 1 point, 10% got 0
points). The courseware_studentmodule table already had an index
on the module_id (a.k.a. module_state_key), but because there
were 100K+ students that had student state for some problems,
the generation of those histograms was still extremely expensive.
During U.S. Thanksgiving weekend in late November of 2012, that
load started causing operational failures on edx.org.
As an emergency measure, I manually added a composite index for
(module_id, grade, student_id) on courseware_studentmodule in
order to stabilize the courseware on edx.org. I did _not_ follow
up properly and add it in a migration file. Later on, the inline
analytics feature was removed entirely, so the index was considered
redundant (but again, it was not properly cleaned up).
Various reports were created over the years, some of which
relied on having an index for module_id. These ran fine because
there had long been an index for that field specifically.
In 2018, the courseware_studentmodule table for edx.org ran into
the 2 TB size limit that our old RDS instance had. We had a fair
amount of monitoring for various limits that we thought we might
run into, but the per-table limit took us by surprise. The Devops/
SRE person fielding that issue needed to free up space in a hurry
in order to make the courseware functional again. Examining the
database itself, he noticed that we had a module_id index that was
technically redundant because the composite index of (module_id,
grade, student_id) would cover queries that would otherwise use it.
Again, as an emergency measure, he dropped the index on module_id
in order to free up a little space and buy enough time to do a
proper move of the database to Aurora.
Devops-of-2018 being more disciplined than me-of-2012, the index
on module_id was removed in 25da206c. The intention was to make it
so that the state of the code would match what was live on edx.org.
But because the composite index was added in an ad hoc way, what
that really meant was that now queries involving module_id were
_only_ indexed by the (module_id, grade, student_id) composite
index that existed only on edx.org and no other Open edX instances.
We didn't realize this issue until months later. @blarghmatey
created an index to re-add the index for module_id:
https://github.com/edx/edx-platform/pull/20885
The reason why we didn't accept this immediately is because
migrations for this table are very operationally risky and take
days to run. Faking this migration would have put edx.org even
more out of sync with the Open edX repo. Complicating this
somewhat was the fact that some folks still seem to be running a
variant of the inline analytics on their fork.
So in the end, we're going forward with this migration that brings
the code fully into sync with indexes on edx.org and covers the
obscure inline analytics histogram use case, while still covering
the module_id index needed for the fast generation of certain
reports that focus on a single problem.
Sorry folks.
In commit 9b37e7d0, the logic of
`streak_celebration_is_active` was accidentally
changed such that it no longer checks the
Progress Milestones waffle flag.
This commit fixes that.
Note: This also adds in a transitive check to
`courseware_mfe_is_active`,
which makes sense for Streak Celebration
and should not have any functional impact.
Centralize the logic for choosing between
MFE and Legacy-frontend courseware within
three new functions:
* courseware_mfe_is_active
* courseware_mfe_is_visible
* courseware_legacy_is_visible
This allows us to create another new function:
* get_courseware_url
which can be called anywhere in LMS/Studio
to get the canonical URL to courseware
content (whether it be MFE or Legacy).
In future commits we we begin using
get_courseware_url throughout the platform.
TNL-7796
django-not-configured is an error raised by pylint (with
the pylint-django plugin) when it's not correctly configured.
We should not be applying lint amnesty for such a violation.
[MICROBA-1024]
- Move some of the recently added logic from the instructor app to the certificates app
- Attempt to not use other certificate models directly in the code I am touching, moving this logic to certificates as well.
When this flag is enabled, users will be 50/50 bucketed into an
experiment where users get course highlights and nudges from an
external service (like sailthru or braze) rather than from
edx-platform via ACE.
AA-661
The Django setting
FEATURES['ENABLE_COURSEWARE_MICROFRONTEND']
has been an additional gate to activating
usage of the Learning MFE for an Open edX
instance.
The toggle is redundant with the
`courseware.courseware_mfe`
Waffle flag. By removing it, we simplify our config
and simplify our path towards making the Learning MFE
the default courseware experience.
TNL-7796
In preparation for switching LMS/Studio over
from serving legacy courseware URLs in certain
places (for example, resume_course_url) to serving
learning micro-frontend URLs.
TNL-7796
This commit removes several waffle toggles that have been enabled
on edx.org for years. It's time to remove the rollout gating for
these features and enable them by default.
This doesn't directly change any behavior. But it does create new
database objects by default now and allows for enabling other
schedule based features more easily.
Specifically, the following toggles were affected.
schedules.create_schedules_for_course
- Waffle flag removed as always-enabled
- We now always create a schedule when an enrollment is created
schedules.send_updates_for_course
- Waffle flag removed as always-enabled
- Course update emails are sent as long as the ScheduleConfig
allows it.
- This is not a change in default behavior, because ScheduleConfig
is off by default.
dynamic_pacing.studio_course_update
- Waffle switch removed as always-enabled
- Course teams can now always edit course updates directly in Studio
ScheduleConfig.create_schedules
ScheduleConfig.hold_back_ratio
- Model fields for rolling out the schedules feature
- Schedules are now always created
- This commit only removes references to these fields, they still
exist in the database. A future commit will remove them entirely
This commit also adds a new has_highlights field to CourseOverview.
This is used to cache whether a course has highlights, used to
decide which course update email behavior they get. Previously every
enrollment had to dig into the modulestore to determine that.
This feature uses the first_day_of_streak, last_day_of_streak and last_streak_celebration fields to determine whether the user should see a celebration.
AA-304
This release adds a new permission check when rendering the practice proctored exam view. If a learner is not in a paid enrollment track, the learner will no longer be able to use proctoring software in a practice proctored exam. As with other proctored exams, learners not in a paid enrollment track will see the exam content. This uses the 'can_take_proctored_exam' permission.
Mobile apps load HTML (and other) XBlocks individually using the
render_xblock endpoint. This is an attmept to reduce the number
of requests and JS processing needed to do so by detecting when
we have math content in HTMLBlocks and only adding the Mathjax
resources when necessary.
This is controlled by the "courseware.optimized_render_xblock"
CourseWaffleFlag. For maximum safety, we currently only optimize
in this way when directly hitting HTMLBlocks, and not for
ProblemBlock or VerticalBlock.
This was made as part of edX's Hackathon XXV.
Now that we always return an existing value from the DB rather than trusting that ID generation is deterministic and constant over time, we're free to change the generation algorithm.
Our long term goal is to switch to random IDs, but we need to first investigate the uses of save=False. In the meantime, this is a good opportunity to move away from MD5, which has a number of cryptographic weaknesses. None of the known vulnerabilities are considered exploitable in this location, given the limited ability to control the input to the hash, but we should generally be moving away from it everywhere for consistency.
This change should not be breaking even for save=False callers, since those calls are extremely rare (1 in 100,000) and should only occur after a save=True call, at which point they'll use the stored value. Even if this were not true, for a save=False/True pair of calls to result in a mismatch in output, the first of the calls would have to occur around the time of the deploy of this code.
Co-authored-by: Tim McCormack <tmccormack@edx.org>
Co-authored-by: Tim McCormack <tmccormack@edx.org>
These changes were initially made to make it easier to do SECRET_KEY rotations. Along the way, we found it made sense to refractor the code as well.
Changes made:
- changed get_to_create to create because now the code should only get to this block when a write is necessary
- added a lookup for anonymous_user_id. This is to return an existing anonymous_user_id rather than calculating. This will mitigate the results of SECRET_KEY rotation.
- Added monitoring to help us make better decisions: should we not sue SECRET_KEY, performance considerations...
- put old function behind toggle in case something goes wrong in production with new code
- refractoring function structure for better understanding
This PR lays the groundwork for a an LTI tab that can embed any LTI1.1-based
tool as an course tab. It also adds another tab based on this LTI Tab that
offers special support for embedding LTI-based discussion tools in a course
tab. If enabled this will replace the existing discussion tab.
