The cms/startup.py and lms/startup.py files were created to
allow us to do a lot of custom initialization around things
like the ModuleStore, monkey-patching, adding MIME types to
our process, etc. As far back as 2017, we recognized that
this was a bad thing, marked these modules as "deprecated",
and started removing things or putting them in the standard
Django locations for them (0279181).
In its current state, these startup modules no longer do any
custom work, and just invoke django.startup(). But this is
meant for running Django code in "standalone" usage, e.g. if
you have a script that isn't a management command but needs
some Django functionality.
The "runserver" command used during development normally
launches a child process to serve requests and knows how to
kill and respawn that process when files are modified, so
that changes are reflected. It can also normally handle the
case where there's a SyntaxError in the child process, and
fixing that error will reload the code again.
Something about running django.startup() manually interferes
with this functionality in "runserver". It still reloads the
code in response to changes, but if the code gets into a
broken state for any reason (like a syntax error), the master
process itself dies. That causes the container to restart,
only to die again shortly afterwards in a loop until the
error is fixed. The container restarts will break any shell
you had opened into the container, as well as any IDE
integrations that connected to that container to access the
files and Python instance.
Getting rid of the custom startup code fixes this and moves
us one small step closer to being a more normal Django
project.
Prior to this commit, the LMS would log the following error in tutor
production:
pymongo/topology.py:175: UserWarning: MongoClient opened before fork.
Create MongoClient only after forking. See PyMongo's documentation for
details:
https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe
Quoting from that page:
> PyMongo is not fork-safe. Care must be taken when using instances of
> MongoClient with fork(). Specifically, instances of MongoClient must
> not be copied from a parent process to a child process. Instead, the
> parent process and each child process must create their own instances
> of MongoClient. Instances of MongoClient copied from the parent
> process have a high probability of deadlock in the child process due
> to the inherent incompatibilities between fork(), threads, and locks
> described below. PyMongo will attempt to issue a warning if there is a
> chance of this deadlock occurring.
For edx-platform, the MongoClient connection is initalized with the
modulestore() invocation. That call creates and caches a global variable
that Studio or the LMS will reuse across the life of the worker process.
That initialization was put into lms/wsgi.py in 7c758ec9, but originated
in lms/startup.py with 51d0dd1. The original reason for it is because at
that time (2013), we still supported the XML Modulestore, which stored
courses on disk as directories of OLX files and static assets. The XML
Modulestore would then read the entirety of those courses into memory at
startup. This meant that the startup process was *extremely* expensive,
so we needed to have it happen before the workers started serving
requests to users, instead of having the system lazily read them in when
the first user request arrived.
Loading course content in this form hasn't been supported since 2016,
meaning that modulestore initialization is no longer the performance
time bomb that it once was. The fact that this code remained here is
likely an oversight, which was considered harmless until @ztraboo
reported these pymongo log messages during the course of investigating
performance issues:
https://discuss.openedx.org/t/atlas-mongodb-performance-issues-un-indexed-queries/12803/16
Two potential followups that should be explored after this:
1. Tutor should probably be forking earlier than this, before we load
Django settings and initialize database and cache connections.
2. It's possible that the caching mechanism for modulestore should be
revisited to operate at the request cache level. The performance
benefit of keeping it around may not be worth the potential memory
leaks. Anything we do here would have to be very carefully monitored
though, since connection costs may add up.
django-not-configured is an error raised by pylint (with
the pylint-django plugin) when it's not correctly configured.
We should not be applying lint amnesty for such a violation.
The upgrade to Django 2.2 has caused a spike in production logging due to RemovedInDjango30Warning deprecation warnings. For now, turn off logging of DeprecationWarning, ImportWarning, and PendingDeprecationWarning whenever we're using WSGI (they'll continue to be logged in devstack). We may eventually want this behind an easily flipped toggle, but for now just remove it to get this resolved quickly.
