'distribute' is the ancestor of 'setuptools', and many packages require
setuptools to be installed.
Upgrade of MySQL-python from 1.2.4 to 1.2.5 is required because of an
incompatibility with setuptools>0.7.
The progress page did a number of things that make performance terrible for
courses with large numbers of problems, particularly if those problems are
customresponse CapaModule problems that need to be executed via codejail.
The grading code takes pains to not instantiate student state and execute the
problem code. If a student has answered the question, the max score is stored
in StudentModule. However, if the student hasn't attempted the question yet, we
have to run the problem code just to call .max_score() on it. This is necessary
in grade() if the student has answered other problems in the assignment (so we
can know what to divide by). This is always necessary to know in
progress_summary() because we list out every problem there. Code execution can
be especially slow if the problems need to invoke codejail.
To address this, we create a MaxScoresCache that will cache the max raw score
possible for every problem. We select the cache keys so that it will
automatically become invalidated when a new version of the course is published.
The fundamental assumption here is that a problem cannot have two different
max score values for two unscored students. A problem *can* score two students
differently such that they have different max scores. So Carlos can have 2/3 on
a problem, while Lyla gets 3/4. But if neither Carlos nor Lyla has ever
interacted with the problem (i.e. they're just seeing it on their progress
page), they must both see 0/4 -- it cannot be the case that Carlos sees 0/3 and
Lyla sees 0/4.
We used to load all student state into two separate FieldDataCache instances,
after which we do a bunch of individual queries for scored items. Part of this
split-up was done because of locking problems, but I think we might have gotten
overzealous with our manual transaction hammer.
In this commit, we consolidate all state access in grade() and progress()
to use one shared FieldDataCache. We also use a filter so that we only pull
back StudentModule state for things that might possibly affect the grade --
items that either have scores or have children.
Because some older XModules do work in their __init__() methods (like Video),
instantiating them takes time, particularly on large courses. This commit also
changes the code that fetches the grading_context to filter out children that
can't possibly affect the grade.
Finally, we introduce a ScoresClient that also tries to fetch score
information all at once, instead of in separate queries. Technically, we are
fetching this information redundantly, but that's because the state and score
interfaces are being teased apart as we move forward. Still, this only
amounts to one extra SQL query, and has very little impact on performance
overall.
Much thanks to @adampalay -- his hackathon work in #7168 formed the basis of
this.
https://openedx.atlassian.net/browse/CSM-17
fixed broken test
changes based on feedback on 6/24
fixed broken unit test after feedback changes
added more checks and updated tests
fixed broken bok choy test
Fixed pylint quality error
trying to fix pylint quality error
Updated styling for basic capa problems (multiple choice, dropdown, checkboxes, numerical input and text input) including support for mobile app experience.
This is primarily to reduce load on MongoDB, where we've lately
had performance problems that we suspect are caused by very
large course structures being evicted from MongoDB's cache. This
may potentially give us a path to better performance as well,
but that's not the goal of this commit.
Surprisingly, LZ4 seemed to actually run more slowly than zlib
for this. Possibly because of some overhead in the Python
bindings? GZip was also surprisingly slow given that it uses
zlib underneath (something like 5x slower).
Use separate cache backend for caching structures.
Abstract out course structure cache.
add datadog metrics for compressed course structure sizes
Since we're using a different cache background, we don't need to have a cache prefix
Use dummy cache backend for tests.
Fallback to default cache if course_structure_cache doesn't exist.
This is primarily to reduce load on MongoDB, where we've lately
had performance problems that we suspect are caused by very
large course structures being evicted from MongoDB's cache. This
may potentially give us a path to better performance as well,
but that's not the goal of this commit.
Surprisingly, LZ4 seemed to actually run more slowly than zlib
for this. Possibly because of some overhead in the Python
bindings? GZip was also surprisingly slow given that it uses
zlib underneath (something like 5x slower).
Use separate cache backend for caching structures.
Abstract out course structure cache.
add datadog metrics for compressed course structure sizes
Since we're using a different cache background, we don't need to have a cache prefix
Use dummy cache backend for tests.
Fallback to default cache if course_structure_cache doesn't exist.
