in the problemgradereport currently we must currently hold the entire file in memory before writing it all at once. to avoid out of memory celery issues to resolve a blocking bug for some MIT courses, add a temp waffle flag `instructor_task.use_on_disk_grade_reporting` which when activated uses this new report to (hopefully) allow the report to complete. Additional testing and consideration is required for this approach.
Adds new api to return block metadata which includes index_dictionary.
Reason for new api instead of adding it to course blocks API: data like
index_dictionary are too large for the cache used by course/blocks
transformers API.
we were failing to get the course and then going on to try course.field
which would generate a worse exception
also wraps the email send in an exception absorb because a single email
can fail to send for infinitely many boring reasons
This function is no longer needed as all XModules have been converted to XBlocks.
XBLOCK_SELECT_FUNCTION Django setting is removed too, as it could take only `prefer_xmodules` or `default_select` values.
* fix: add missing permission/auth classes to init
* fix: add missing permission class
Django was complaining about adding JWT auth without the
NotJwtRestrictedApplication class which it would add automatically. Add
the class to remove that warning.
* style: use black / isort to fix import ordering
When a student submits a problem answer, the state is stored in a
StudentModule record containing answer, score, correctness, etc. The
record, though, is updated in multiple steps within the single request
(first the grade is updated, then the state is updated separately).
Each partial save would trigger a separate StudentModuleHistory record
to be stored resulting in duplicate and inaccurate historical records.
This solution uses the RequestCache to track within a request thread
which StudentModules are updated and a single corresponding
StudentModuleHistory id. If multiple update actions occur within the
request cycle, then modify the history record that was already
generated to ensure that each submission only results in one
StudentModuleHistory record.
This issue and its solution were discussed in:
https://discuss.openedx.org/t/extra-history-record-stored-on-each-problem-submission/8081
Clearing the RequestCache was intended to address memory leaks in the
celery workers. Celery worker processes will process many tasks before
they are terminated. RequestCache cleanup typically happens in the
RequestCacheMiddleware class, and middleware never executes for celery.
To get around that issue, the CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION
setting was created to clear the RequestCache after every task was
successfully completed.
This works fine when celery is running as a separate process, as it's
set up to do in production. But during development, the
CELERY_ALWAYS_EAGER setting variable is set to True, meaning that
celery tasks are run in the same thread as the Django Request. This is
meant to make debugging easier, as task failures run as part of the
request cycle and will raise exceptions that are visible to the
browser.
However, celery tasks are triggered from many different actions. That
means that the RequestCache was being cleared many times during the
course of processing a request. This led to behavior that was
potentially slower, but also incorrect–the RequestCache was getting
flushed in a way that wouldn't happen in any deployed environment
because celery would be running in separate processes there. This came
up when trying to fix an issue around extra history records being
created during problem submissions:
https://discuss.openedx.org/t/extra-history-record-stored-on-each-problem-submission/8081
Furthermore, it's not necessary to prevent RequestCache memory leaks
when running in CELERY_AWLAYS_EAGER mode in development because the
middleware cleanup happens automatically–as everything is running as
part of the request/response cycle.
There are times in which we may want to run celery eagerly and still
clear the cache, such as testing. I have set
CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION = False in all dev and test
environments that already have CELERY_ALWAYS_EAGER = True. The unit
test that specifically tests whether the request cache is getting
cleared upon completion of a celery task then overrides
CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION = True even though
CELERY_ALWAYS_EAGER = True for the sake of that specific testing
purpose.
