Clearing the RequestCache was intended to address memory leaks in the
celery workers. Celery worker processes will process many tasks before
they are terminated. RequestCache cleanup typically happens in the
RequestCacheMiddleware class, and middleware never executes for celery.
To get around that issue, the CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION
setting was created to clear the RequestCache after every task was
successfully completed.
This works fine when celery is running as a separate process, as it's
set up to do in production. But during development, the
CELERY_ALWAYS_EAGER setting variable is set to True, meaning that
celery tasks are run in the same thread as the Django Request. This is
meant to make debugging easier, as task failures run as part of the
request cycle and will raise exceptions that are visible to the
browser.
However, celery tasks are triggered from many different actions. That
means that the RequestCache was being cleared many times during the
course of processing a request. This led to behavior that was
potentially slower, but also incorrect–the RequestCache was getting
flushed in a way that wouldn't happen in any deployed environment
because celery would be running in separate processes there. This came
up when trying to fix an issue around extra history records being
created during problem submissions:
https://discuss.openedx.org/t/extra-history-record-stored-on-each-problem-submission/8081
Furthermore, it's not necessary to prevent RequestCache memory leaks
when running in CELERY_AWLAYS_EAGER mode in development because the
middleware cleanup happens automatically–as everything is running as
part of the request/response cycle.
There are times in which we may want to run celery eagerly and still
clear the cache, such as testing. I have set
CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION = False in all dev and test
environments that already have CELERY_ALWAYS_EAGER = True. The unit
test that specifically tests whether the request cache is getting
cleared upon completion of a celery task then overrides
CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION = True even though
CELERY_ALWAYS_EAGER = True for the sake of that specific testing
purpose.
Call into the exam service instead of the edx-proctoring plugin on course publish if the course_apps.exams_ida course waffle flag is enabled. This is an early step in moving away from edx-proctoring
[APER-1922]
We are converting the legacy UI of the `records` app in the Credentials IDA (credentials.edx.org/records/, credentials.edx.org/records/programs/{uuid}, etc.) to a new MFE. Today, the Program Dashboard and the legacy (non-MFE) profile page have buttons that route learners to the Credentials IDA pages. We need to (optionally) introduce a way to route learner's to the new MFE instead.
- Introduces a new configuration setting called `LEARNER_RECORD_MICROFRONTEND_URL` (defaulting to None). This will be used by the LMS to store the base URL of the new MFE (e.g. records.stage.edx.org).
- Introduces a new waffle switch named `USE_LEARNER_RECORD_MFE`. This will be used to control whether routing learner's to the new MFE is enabled from the LMS's side.
- Updates the existing `get_credentials_records_url` function to add additional logic that will determine if we need to build a link to the legacy FE or the new MFE
- Adds tests for new and existing behavior. There were no existing unit tests for the utility function that I updated.
* fix: update learner home api
* refactor: update learner home API pathing
* fix: remove unused logger
* fix: remove unused import
* style: make pylint happy
* style: remove extra space
* style: remove extra space
Co-authored-by: nsprenkle <nsprenkle@2u.com>
Co-authored-by: Nathan Sprenkle <nsprenkle@users.noreply.github.com>
* refactor: enterprise dependencies for EdxRestAPIClient replacement
This is a part of https://github.com/openedx/public-engineering/issues/42
- add settings for enterprise-backend-service DOT application
- update utils used by enterprise to get rid of EdxRestAPIClient
- original utils stays in the code (to keep edx-platform api
clients working) till the
https://github.com/openedx/public-engineering/issues/39 deprecation
work will be done
* fix: fix typo in the docstring
Previously, our rate-limiting code trusted the entire `X-Forwarded-For`
header, allowing a malicious client to spoof that header and evade
rate-limiting. This commit introduces a new module and setting
allowing us to make a more conservative choice of IPs.
- Create new `openedx.core.djangoapps.util.ip` module for producing
the IP "external chain" for requests based on the XFF header and the
REMOTE_ADDR.
- Include a function that gives the safest choice of IPs.
- Add new setting `CLOSEST_CLIENT_IP_FROM_HEADERS` for configuring how
the external chain is derived (i.e. setting the trust
boundary). Currently has a default, but we may want to make it
mandatory in the future.
- Change `django-ratelimit` code to use the proximate IP in the external
chain -- the one just outside the trust boundary.
Also:
- Change `XForwardedForMiddleware` to use more conservative choice for
its `REMOTE_ADDR` override
- Other adjustments to `XForwardedForMiddleware` as needed in order to
initialize new module and support code that needs the real
`REMOTE_ADDR` value
- Metrics for observability into the change (and XFF composition)
- Feature switch to restore legacy mode if needed
This also gives us a path forward to removing use of the django-ipware
package, which is no longer maintained and has a handful of bugs that make it
difficult to use safely.
Internal ticket: ARCHBOM-2056
Creates a link to the new communication mfe's bulk email tool in the
instructor dashboard version. Staff can now use either experience. In
the future, we plan on turning off the old experience, like analytics.
- Adds Enhanced Staff Grader (ESG) backend-for-frontend (BFF) in `lms/djangoapps/ora_staff_grader`
- Adds routing to ESG BFF at `{lms_url}/api/ora_staff_grader/*`
- Adds mock implementation routing at `{lms_url}/api/ora_staff_grader/mock/*`
- Adds `ORA_GRADING_MICROFRONTEND_URL` setting for routing to ESG microfrontend (MFE)
- Updates to the teams app:
- Add`get_teams_in_teamset` to the teams API.
- Add `get_team_names` to teams service.
- Adds `openassessment.staffgrader` app for appropriate ORA migrations.
- Modifies management commands for creation of users.
- Updates test factory to return display org with course overview.
Co-authored-by: jansenk <jkantor@edx.org>
Co-authored-by: Leangseu Kim <lkim@edx.org>
Co-authored-by: Ben Warzeski <bwarzeski@edx.org>
* feat: Add support for using the discussions MFE UI instead of existing UI
Adds a new course waffle flag that when set along with the discussions MFE URL shows the discussions MFE UI instead of the regular UI.
* test: add tests
* squash!: more consistent url name
* chore: update deprecated import from collections
* chore: remove outdated imports from markdown library
as it hasn't been supported since 2.0.3 and we're on 3.x.
This was deprecated at least as early as 2012!
* docs: add docstring and remove lint-amnesty to markdown plugin
* chore: remove deprecated etree import
* style: remove unnecessary-comprehension for sets
* style: resolve a number of amnestied pylint complaints
Co-authored-by: stvn <stvn@mit.edu>
[MICROBA-1569]
- filter bulk course email recipients based on the last_login date of a learner's user account
- introduces a new setting named `BULK_COURSE_EMAIL_LAST_LOGIN_ELIGIBILITY_PERIOD` that sets the login threshold to be included (in months) to a bulk course email message(if set)
This also requires using different session cookie names in devstack for
LMS and Studio so that we can properly use the OAuth flow locally, rather
than just sharing a session cookie on the localhost domain. (Note that
ports are not used for cookie segmentation in browsers.)
This depends on the following provisioning PR:
https://github.com/edx/devstack/pull/825
LOGIN_REDIRECT_WHITELIST has been used to filter redirect-url while processing logout requests but its configurations were not picked through environment files like lms.yml or studio.yml. This PR fixes that bug.
The Django setting
FEATURES['ENABLE_COURSEWARE_MICROFRONTEND']
has been an additional gate to activating
usage of the Learning MFE for an Open edX
instance.
The toggle is redundant with the
`courseware.courseware_mfe`
Waffle flag. By removing it, we simplify our config
and simplify our path towards making the Learning MFE
the default courseware experience.
TNL-7796
MST-637: edx-proctoring 3.0.0 now requires a new key for use in ID
generation, starting from same value as SECRET_KEY (pre-rotation, but
rotation never happened for devstack.)
Remote config changes already made for relevant environments.
Deprecated edx-platform import paths
(for example, `student` instead of
`common.djangoapps.student`) currently raise
warnings when used. We want to fully remove
support for those paths.
As an easily reversible way to initially remove
support, we add a new setting to LMS and Studio
called `ERROR_ON_DEPRECATED_EDX_PLATFORM_IMPORTS`,
defaulting to False. We set it to True for devstack
and will set it to True in Stage and Production
soon. If critical errors occur, we can easily
flip the setting back to False.
* Install `organizations` app into LMS and Studio non-optionally.
* Add toggle `ORGANIZATIONS_AUTOCREATE` to Studio.
* Remove the `FEATURES["ORGANIZATIONS_APP"]` toggle.
* Use the new `organizations.api.ensure_organization` function to
either validate or get-or-create organizations, depending
on the value of `ORGANIZATIONS_AUTOCREATE`,
when creating course runs and V2 content libraries.
We'll soon use it for V1 content libraries as well.
* Remove the `util.organizations_helpers` wrapper layer
that had to exist because `organizations` was an optional app.
* Add `.get_library_keys()` method to the Split modulestore.
* Add Studio management command for backfilling organizations tables
(`backfill_orgs_and_org_courses`).
For full details, see
https://github.com/edx/edx-organizations/blob/master/docs/decisions/0001-phase-in-db-backed-organizations-to-all.rst
TNL-7646
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
This patch would enable routing learner to logistration MFE
via forgot password url than on platform when
ENABLE_LOGISTRATION_MICROFRONTEND feature flag is set.
VAN-98
Removed most of the deprecated shoppingcart app, leaving just enough to allow us to cleanly remove the related database tables later. Also removed the relevant Django settings that weren't in use elsewhere.
It is already enabled in devstack LMS, stage LMS/Studio,
and prod LMS/Studio.
However, it is currently disabled in edge LMS/Studio,
and as far as I know, sandbox LMS/Studio as well as the
default Open edX LMS/Studio.
We would like to move towards enabling it globally by
default, and enabling it in devstack Studio would be
a first step towards that.
* Moving plugins infrastructure to edx-django-utils
This PR extracts the code that enables plugins in edx-platform and puts it in edx-django-utils. This is done to allow other IDAS to add plugin functionality.
Set `SESSION_COOKIE_SAMESITE=Lax` for `devstack_docker` environment by default to allow login to LMS service. This is a fix for `devstack_docker` default value set to `Lax` for `DCS_SESSION_COOKIE_SAMESITE`. It was defaulting to `SameSite=None` which requires a secure site which `localhost` site does not by default. Setting this `SameSite` cookie attribute to something other than `None` will continue to allow login to the LMS for `devstack_docker` environment. Regards to #23671 and https://discuss.openedx.org/t/lti-xblock-and-samesite/759/16
* using new welcome template when redirected from enterprise proxy login view
* enabling safe redirects to enterprise learner portal from login in devstack
* ading admin portal to login redirect whitelist
* running make upgrade to version bump edx-enterprise
