The progress page did a number of things that make performance terrible for
courses with large numbers of problems, particularly if those problems are
customresponse CapaModule problems that need to be executed via codejail.
The grading code takes pains to not instantiate student state and execute the
problem code. If a student has answered the question, the max score is stored
in StudentModule. However, if the student hasn't attempted the question yet, we
have to run the problem code just to call .max_score() on it. This is necessary
in grade() if the student has answered other problems in the assignment (so we
can know what to divide by). This is always necessary to know in
progress_summary() because we list out every problem there. Code execution can
be especially slow if the problems need to invoke codejail.
To address this, we create a MaxScoresCache that will cache the max raw score
possible for every problem. We select the cache keys so that it will
automatically become invalidated when a new version of the course is published.
The fundamental assumption here is that a problem cannot have two different
max score values for two unscored students. A problem *can* score two students
differently such that they have different max scores. So Carlos can have 2/3 on
a problem, while Lyla gets 3/4. But if neither Carlos nor Lyla has ever
interacted with the problem (i.e. they're just seeing it on their progress
page), they must both see 0/4 -- it cannot be the case that Carlos sees 0/3 and
Lyla sees 0/4.
We used to load all student state into two separate FieldDataCache instances,
after which we do a bunch of individual queries for scored items. Part of this
split-up was done because of locking problems, but I think we might have gotten
overzealous with our manual transaction hammer.
In this commit, we consolidate all state access in grade() and progress()
to use one shared FieldDataCache. We also use a filter so that we only pull
back StudentModule state for things that might possibly affect the grade --
items that either have scores or have children.
Because some older XModules do work in their __init__() methods (like Video),
instantiating them takes time, particularly on large courses. This commit also
changes the code that fetches the grading_context to filter out children that
can't possibly affect the grade.
Finally, we introduce a ScoresClient that also tries to fetch score
information all at once, instead of in separate queries. Technically, we are
fetching this information redundantly, but that's because the state and score
interfaces are being teased apart as we move forward. Still, this only
amounts to one extra SQL query, and has very little impact on performance
overall.
Much thanks to @adampalay -- his hackathon work in #7168 formed the basis of
this.
https://openedx.atlassian.net/browse/CSM-17
only require ccx-keys once
get_current_ccx will now expect a CourseKey instance as its argument, and will raise a value error if this expectation is not met.
document reason for local import
add special methods to pass attribute setting and deletion through to the wrapped modulestore
add __setattr__ and __delattr__ per code review, update __init__ to work with new methods
style change per code review
clean up context manager usage as recommended by code review
remove unused code and imports
convert modulestore type tests to use the `get_modulestore_type` api, remove unused imports
code quality: add docstrings
increase coverage for utils tests
fix bug found in testing.
increase test coverage on modulestore wrapper
code quality fixes
code-quality: ignore import error, but mark site for future consideration
remove references to middleware that were missed previously
use key apis rather than local implementation of key conversion. remove local implementationa
remove spurious test for attribute
fix test setUp to avoid unneeded flattening
code quality fixes
add security check ensuring that the coach is coach for *this* CCX.
prevent ccx/deprecated course id problems
1. do not allow ccx objects to be created if the course id is deprecated
2. filter out any ccx memberships that involve deprecated course ids (in case there are bad ccxs in the database)
Fix test failures and errors arising from incorrect code path execution
Create context manager to handle unwrapping and restoring ccx values for the modulestore wrapper, employ it throughout modulestore wrapper implementation
Implement the use of CCX opaque keys throughout the ccx code base
include the new custom ccx id package in the github checkouts list
update the coach dashboard wrapper to get CCX information from the incoming course_id, if possible
update function signatures for all view functions to expect CCX as passed by the dashboard wrapper (default to None), remove calls to get_ccx_for_coach as the ccx is passed in.
update reverse calls in python view code to use a CCXLocator for the URL instead of a CourseLocator
use CCXLocator where necessary
use course id to find ccx, instead of thread local
remove unused method and related tests
use course id for getting ccx
provide course id to the get_current_ccx method
ensure the course id passed in is a CourseKey instance of some type whether it starts out as a string or not
use the provided block to figure out what the course_id should be, then get the ccx for that
redirect to ccx dashboard using coach ccx if no ccx is passed in
update student dashboard listing for ccx to build an appropriate url from a CCXLocator, not from the course locator.
refactor building the ccx locator so we don't have to do it repeatedly
begin test refactoring after ccx_keys introduction
Ensure that when access checking happens, the course_locator form of a ccx locator is used. This ensures that the access check happens against the course and it is not necesarry to duplicate the entire access control structure for the course.
pick up api change in ccx-keys
create and conditionally use a wrapper for the mixed modulestore returned by xmodule.modulestore.django.modulestore
the wrapper will strip and restore ccx values from CourseKey and UsageKey objects
fix return values on a few methods
remove unused symbol
pull updated ccx-keys package
set course_id on the caching descriptor system to avoid api incompatibilities in some subsystems
use ccx.course instead of self.course
fix get method to find course keys from blocks that are not themselves keys but have a location attribute (which will be a key)
if an item coming out of the db has children, restore the ccx to them as well
if the block passed in has a CCX key, unwrap that before we try to look up the override, otherwise it will never be found.
pick up a change in the ccx keys package that allows for stripping CCX identity from a usage key
begin writing tests to cover this modulestore wrapper
remove the switch_pocs view, the url pattern for it, and the tests that covered it
remove the ccx context and the middleware responsible for setting the current CCX. These are no longer needed
all dashboard views should raise 404 if a ccx is not provided by the coach_dashboard decorator
code quality
prevent errors resulting from trying to `get` a ccx based on non-unique criteria.
remove obsolete usage of ACTIVE_CCX_KEY
fix setUp method for grading tests to properly create grades for the ccx rather than for the course.
clean up reverse calls
code quality
adding docstrings to clarify purpose of this patch
fix bug in getting ccx for coach
fix grading views to properly fetch a ccx-ified course so that grades for that version will be calculated
fix small errors in modulestore implementation
fix errant merge marker
update call to get_current_ccx after key refactoring merged with tab changes
* Updated XBlock hash - new version forces exposes force_dirty method on fields
* Includes Scope.children fields into xml_importer._update_module_location so that they are updated as well.
* Tests for mutable fields xml_importer._update_module_location with mutable fields and children
Use the lazy module to create cached attributes
Ensure tests for localized date formatting are insulated from changes in default formatting
Other minor fixes
repairs https://openedx.atlassian.net/browse/TNL-2007 and https://openedx.atlassian.net/browse/TNL-2009
Original Commit Messages:
Update ccx dashboard display to match new course display
Fix display of CCX dates on dashboard to use dates from the CCX instead of the related course.
The fix is implemented by providing date- and course-related APIs on the CCX object itself. These APIs are mimicked from the same APIs available on a course descriptor.
bring the methods for printing start and end datetimes into line with course by adding UTC under the proper circumstances
add test coverage for new APIs defined on the CCX model
fix quality violation
fix for making sure all descriptor fields are always available for course descriptors despite module_class tricks
add required fields to cache data for tests
use a different strategy to mock up the get_children method
Original Commit Messages:
use edx's own get_parent method, rather than our own.
add field to unique constraint to avoid MultipleObjectsReturned in case of multiple browser clicks on submit
fix 0011 migration, inherit from TimeStampedField and add composite index (migration only)
fix bug where adding an already registered user to a ccx would cause a crash due to an undefined variable
add assertNumQueries tests to test modules where override field providers are used
remove unnecessary teardown
implement recommended style for checking empty list
import utility methods rather than use duplicate code
added comment explaining date conversion to string for json
add logging for invalid users or emails when enrolling students
add comment about xmodule user state
avoid using get_or_create, which seems to be causing a race condition on schedule change save
relocate badly placed edvent handlers to fix multiple submit problem
Final official name is Custom Courses for EdX (CCX), rename all code to remove previous name.
Rename the FEATURE constant used to identify this feature
Rename the middleware for the CCX change
rename the constant used for storing the current poc id on the session.
rename the _PocContext threading local
rename the override provider in all places where it is referenced
`PersonalOnlineCoursesOverrideProvider` -> `CustomCoursesForEdxOverrideProvider`
generally rename symbols from overrides.py to replace `poc` with `ccx` where possible without changing model names or attributes
rename more symbols from poc to ccx
rename util functions from app utils module
general symbol renaming poc -> ccx in views.py and related url changes
Rename the coach role wherever it is used.
reword poc_coach to ccx_coach
UI rename
replace POC with CCX globally
template context variable renamed
rename poc_ to ccx_ in urls and all related locations (views, js, scss etc)
remove pocs migrations
Final massive renaming, including models. Re-built migration.
cleaning up a few tailing references
Fix reference typo in schedule template JS
undo modifications made on the fly in test setup to ensure that our tests are properly isolated from the rest of the system tests.
Fixesjazkarta/edx-platform#38
Clean up some leftover strings and comments
fixing more strings and comments in python files
fix a naming error in the schedule tab that was causing problems in deleting courseware items.
Fixesjazkarta/edx-platform#36
updating tests and utility code to match changes in infrastructure from latest rebase
