The Course Info Blocks API endpoint has been known to be rather slow
to return the response. Previous investigation showed that the major
time sink was the get_course_blocks function, which is called three
times in a single request. This commit aims to improve the response
times by reducing the number of times that this function is called.
Solution Summary
The first time the function get_course_blocks is called, the result
(transformed course blocks) is stored in the current WSGI request
object. Later in the same request, before the second get_course_blocks
call is triggered, the already transformed course blocks are taken
from the request object, and if they are available, get_course_blocks
is not called (if not, it is called as a fallback). Later in the
request, the function is called again as before (see Optimization
Strategy and Difficulties).
Optimization Strategy and Difficulties
The original idea was to fetch and transform the course blocks once
and reuse them in all three cases, which would reduce get_course_blocks
call count to 1. However, this did not turn out to be a viable solution
because of the arguments passed to get_course_blocks. Notably, the
allow_start_dates_in_future boolean flag affects the behavior of
StartDateTransformer, which is a filtering transformer modifying the
block structure returned.
The first two times allow_start_dates_in_future is False, the third
time it is True. Setting it to True in all three cases would mean that
some blocks would be incorrectly included in the response.
This left us with one option - optimize the first two calls. The
difference between the first two calls is the non-filtering
transformers, however the second call applies a subset of transformers
from the first call, so it was safe to apply the superset of
transformers in both cases. This allowed to reduce the number of
function calls to 2. However, the cached structure may be further
mutated by filters downstream, which means we need to cache a copy of
the course structure (not the structure itself). The copy method itself
is quite heavy (it calls deepcopy three times), making the benefits of
this solution much less tangible. In fact, another potential
optimization that was considered was to reuse the collected block
structure (pre-transformation), but since calling copy on a collected
structure proved to be more time-consuming than calling get_collected,
this change was discarded, considering that the goal is to improve
performance.
Revised Solution
To achieve a more tangible performance improvement, it was decided to
modify the previous strategy as follows:
* Pass a for_blocks_view parameter to the get_blocks function to make
sure the new caching logic only affects the blocks view.
* Collect and cache course blocks with future dates included.
* Include start key in requested fields.
* Reuse the cached blocks in the third call, which is in
get_course_assignments
* Before returning the response, filter out any blocks with a future
start date, and also remove the start key if it was not in requested
fields
There was problem in filter_discussion_xblocks_from_response(). This
function was breaking the list response for the BlocksInCourseView by
returning a dict instead of list.
Adds new api to return block metadata which includes index_dictionary.
Reason for new api instead of adding it to course blocks API: data like
index_dictionary are too large for the cache used by course/blocks
transformers API.
The /jump_to/ LMS endpoint is used in a number of places
to direct users to courseware. It currently only redirects to
Legacy courseware URLs, which then conditionally may
redirect to the Learning MFE.
Two issues with this:
1. Performance Impact: In most cases, going to Legacy first
is just an extra redirect.
2. Confusion for Privileged Users: Neither course nor global
staff are auto-redirected from the Legacy experience to the
MFE. Thus, these priviliged users confusingly never see the
MFE by default; they must always manually click into it.
This commit makes it so that /jump_to/ directs
users to whatever the default courseware experience is
for them. For staff of courses active in the new experience,
this will impact (at a minimum) the "View Live"
links in Studio, all links on the old and new LMS
course outline, and the "Resume" links on the course
dashboard. Learners should see no difference other than
a performance improvement when following courseware links
from the LMS.
This also adds an optional 'experience=[legacy|new]'
query param to /jump_to/, allowing us to specifically
generate Legacy courseware URLs for the
"View in Legacy Experience" tool.
TNL-7796
* LEARNER-8158
Fixed completion param issue
- There is a case where we are sending requested_fields in params as comma separated list
e.g. requested_fields=children,show_gated_sections,graded,special_exam_info,completion.
- We didn't test for this case in first place and test cases were sending requested_fields as list.
- Now we can also handle this comma separated completion field which was getting ignored before.
Opens the course blocks API to public access, and allows anonymous users to
use the API to fetch data about public courses. Anonymous users need to
explicitly pass an empty username parameter to get the block data that is
visible to the public.
These are expensive, read-only web requests. Unfortunately,
middleware adds writes, and we currently run with view-level
transactions enabled by default. Holding those long transactions
open has caused extra load on the database and been our largest
sources of django.db.utils:OperationError exceptions.
This has been particularly noticeable as we start deploying the
new Courseware MFE, which uses the BlocksInCourseView more
frequently.
