d8aab3f72c
Prior to this commit, the LMS would log the following error in tutor production: pymongo/topology.py:175: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe Quoting from that page: > PyMongo is not fork-safe. Care must be taken when using instances of > MongoClient with fork(). Specifically, instances of MongoClient must > not be copied from a parent process to a child process. Instead, the > parent process and each child process must create their own instances > of MongoClient. Instances of MongoClient copied from the parent > process have a high probability of deadlock in the child process due > to the inherent incompatibilities between fork(), threads, and locks > described below. PyMongo will attempt to issue a warning if there is a > chance of this deadlock occurring. For edx-platform, the MongoClient connection is initalized with the modulestore() invocation. That call creates and caches a global variable that Studio or the LMS will reuse across the life of the worker process. That initialization was put into lms/wsgi.py in7c758ec9, but originated in lms/startup.py with51d0dd1. The original reason for it is because at that time (2013), we still supported the XML Modulestore, which stored courses on disk as directories of OLX files and static assets. The XML Modulestore would then read the entirety of those courses into memory at startup. This meant that the startup process was *extremely* expensive, so we needed to have it happen before the workers started serving requests to users, instead of having the system lazily read them in when the first user request arrived. Loading course content in this form hasn't been supported since 2016, meaning that modulestore initialization is no longer the performance time bomb that it once was. The fact that this code remained here is likely an oversight, which was considered harmless until @ztraboo reported these pymongo log messages during the course of investigating performance issues: https://discuss.openedx.org/t/atlas-mongodb-performance-issues-un-indexed-queries/12803/16 Two potential followups that should be explored after this: 1. Tutor should probably be forking earlier than this, before we load Django settings and initialize database and cache connections. 2. It's possible that the caching mechanism for modulestore should be revisited to operate at the request cache level. The performance benefit of keeping it around may not be worth the potential memory leaks. Anything we do here would have to be very carefully monitored though, since connection costs may add up.
Open edX Platform
#################
| |License: AGPL v3| |Status| |Python CI|
.. |License: AGPL v3| image:: https://img.shields.io/badge/License-AGPL_v3-blue.svg
:target: https://www.gnu.org/licenses/agpl-3.0
.. |Python CI| image:: https://github.com/openedx/edx-platform/actions/workflows/unit-tests.yml/badge.svg
:target: https://github.com/openedx/edx-platform/actions/workflows/unit-tests.yml
.. |Status| image:: https://img.shields.io/badge/status-maintained-31c653
Purpose
*******
The `Open edX Platform <https://openedx.org>`_ is a service-oriented platform for authoring and
delivering online learning at any scale. The platform is written in
Python and JavaScript and makes extensive use of the Django
framework. At the highest level, the platform is composed of a
monolith, some independently deployable applications (IDAs), and
micro-frontends (MFEs) based on the ReactJS.
This repository hosts the monolith at the center of the Open edX
platform. Functionally, the edx-platform repository provides two services:
* CMS (Content Management Service), which powers Open edX Studio, the platform's learning content authoring environment; and
* LMS (Learning Management Service), which delivers learning content.
Documentation
*************
Documentation can be found at https://docs.openedx.org/projects/edx-platform.
Getting Started
***************
For Production
==============
Installing and running an Open edX instance is not simple. We strongly
recommend that you use a service provider to run the software for you. They
have free trials that make it easy to get started:
https://openedx.org/get-started/
However, if you have the time and expertise, then it is is possible to
self-manage a production Open edX instance. To help you build, customize,
upgrade, and scale your instance, we recommend using `Tutor`_, the
community-supported, Docker-based Open edX distribution.
You can read more about getting up and running with a Tutor deployment
at the `Site Ops home on docs.openedx.org`_.
For Development
===============
Tutor also features a `development mode`_ which will also help you modify,
test, and extend edx-platform. We recommend this method for all Open edX
developers.
Bare Metal (Advanced)
=====================
It is also possible to spin up an Open edX platform directly on a Linux host.
This method is less common and mostly undocumented. The Open edX community will
only be able to provided limited support for it.
Running "bare metal" is only advisable for (a) developers seeking an
adventure and (b) experienced system administrators who are willing to take the
complexity of Open edX configuration and deployment into their own hands.
System Dependencies
-------------------
Interperters/Tools:
* Python 3.11 (preferred) or 3.8 (compatible, for now)
* Node 18
Services:
* MySQL 8.0
* Mongo 7.x
* Memcached
Language Packages:
* Frontend:
- ``npm clean-install`` (production)
- ``npm clean-install --dev`` (development)
* Backend build:
- ``pip install -r requirements/edx/assets.txt``
* Backend application:
- ``pip install -r requirements/edx/base.txt`` (production)
- ``pip install -r requirements/edx/dev.txt`` (development)
Build Steps
-----------
Create a MySQL database and a MySQL user with write permissions, and configure
Django to use them. Then, run migrations::
./manage.py lms migrate
./manage.py cms migrate
Build static assets (for more details, see `building static
assets`_)::
npm run build # or, 'build-dev'
Download locales and collect static assets (can be skipped for development
sites)::
make pull_translations
./manage.py lms collectstatic
./manage.py cms collectstatic
Run the Platform
----------------
First, ensure MySQL, Mongo, and Memcached are running.
Start the LMS::
./manage.py lms runserver
Start the CMS::
./manage.py cms runserver
This will give you a mostly-headless Open edX platform. Most frontends have
been migrated to "Micro-Frontends (MFEs)" which need to be installed and run
separately. At a bare minimum, you will need to run the `Authentication MFE`_,
`Learner Home MFE`_, and `Learning MFE`_ in order meaningfully navigate the UI.
.. _Tutor: https://github.com/overhangio/tutor
.. _Site Ops home on docs.openedx.org: https://docs.openedx.org/en/latest/site_ops/index.html
.. _development mode: https://docs.tutor.edly.io/dev.html
.. _building static assets: ./docs/references/static-assets.rst
.. _Authentication MFE: https://github.com/openedx/frontend-app-authn/
.. _Learner Home MFE: https://github.com/openedx/frontend-app-learner-dashboard
.. _Learning MFE: https://github.com/openedx/frontend-app-learning/
License
*******
The code in this repository is licensed under version 3 of the AGPL
unless otherwise noted. Please see the `LICENSE`_ file for details.
.. _LICENSE: https://github.com/openedx/edx-platform/blob/master/LICENSE
More about Open edX
*******************
See the `Open edX site`_ to learn more about the Open edX world. You can find
information about hosting, extending, and contributing to Open edX software. In
addition, the Open edX site provides product announcements, the Open edX blog,
and other rich community resources.
.. _Open edX site: https://openedx.org
Getting Help
************
If you're having trouble, we have discussion forums at
https://discuss.openedx.org where you can connect with others in the community.
Our real-time conversations are on Slack. You can request a `Slack
invitation`_, then join our `community Slack team`_.
For more information about these options, see the `Getting Help`_ page.
.. _Slack invitation: https://openedx.org/slack
.. _community Slack team: http://openedx.slack.com/
.. _Getting Help: https://openedx.org/getting-help
Issue Tracker
*************
We use Github Issues for our issue tracker. You can search
`previously reported issues`_. If you need to report a bug, or want to discuss
a new feature before you implement it, please `create a new issue`_.
.. _previously reported issues: https://github.com/openedx/edx-platform/issues
.. _create a new issue: https://github.com/openedx/edx-platform/issues/new/choose
How to Contribute
*****************
Contributions are welcome! The first step is to submit a signed
`individual contributor agreement`_. See our `CONTRIBUTING`_ file for more
information – it also contains guidelines for how to maintain high code
quality, which will make your contribution more likely to be accepted.
New features are accepted. Discussing your new ideas with the maintainers
before you write code will also increase the chances that your work is accepted.
Code of Conduct
***************
Please read the `Community Code of Conduct`_ for interacting with this repository.
Reporting Security Issues
*************************
Please do not report security issues in public. Please email
security@openedx.org.
.. _individual contributor agreement: https://openedx.org/cla
.. _CONTRIBUTING: https://github.com/openedx/.github/blob/master/CONTRIBUTING.md
.. _Community Code of Conduct: https://openedx.org/code-of-conduct/
People
******
The current maintainers of this repository can be found on `Backstage`_.
.. _Backstage: https://backstage.openedx.org/catalog/default/component/edx-platform