5ef1e08050
When crawlers like edX-downloader make requests on courseware, they are often concurrently loading many units in the same sequence. This causes contention for the rows in courseware_studentmodule that store the student's state for various XBlocks/XModules, most notably for the sequence, chapter, and course -- all of which record and update user position information when loaded. It would be nice if we could actually remove these writes altogether and come up with a cleaner way of keeping track of the user's position. In general, GETs should be side-effect free. However, any such change would break backwards compatibility, and would require close coordination with research teams to make sure they weren't negatively affected. This commit identifies crawlers by user agent (CrawlersConfig model), and blocks student state writes if a crawler is detected. FieldDataCache writes simply become no-ops. It doesn't actually alter the rendering of the courseware in any way -- the main impact is that the blocks won't record your most recent position, which is meaningless for crawlers anyway. This can also be used as a building block for other policy we want to define around crawlers. We just have to be mindful that this only works with "nice" crawlers who are honest in their user agents, and that significantly more sophisticated (and costly) measures would be necessary to prevent crawlers that try to be even trivially sneaky. [PERF-403]
Open edX -------- This is the root package for Open edX. The intent is that all importable code from Open edX will eventually live here, including the code in the lms, cms, and common directories. If you're adding a new Django app, place it in core/djangoapps. If you're adding utilities that require Django, place them in core/djangolib. If you're adding code that defines no Django models or views of its own but is widely useful, put it in core/lib. Note: All new code should be created in this package, and the legacy code will be moved here gradually. For now the code is not structured like this, and hence legacy code will continue to live in a number of different packages.