We were using an old version of the python3-saml library,
which was causing issues with newer versions of social-core.
The reason it was pinned was because our etree implementation
didn't support several fields that the saml library did, so
we are now importing those entities as well.
* test: warn about dependencies from cms->openedx->lms and vice versa
* test: warn about importing from package's internal implementation code
* chore: Update some imports to use public APIs only
* chore: Update 'bookmarks' app to have stricter public API
* fix: we are sharing 'adapters' from olx_rest_api to content_staging
The IP chain code has moved to edx-django-utils—except for the legacy-IP code,
which is now in a new module. This will allow other IDAs to use the IP code.
Commit includes some adjustments to the toggle annotation.
Part of <https://github.com/openedx/edx-django-utils/issues/241>.
We already add the user id (imperfectly) to many requests.
However, when a user starts off unauthenticated, it is not
possible to correlate to those requests. Adding the raw
IP chain provides that possibility. See new custom attribute
ip_chain.raw.
JIRA: https://openedx.atlassian.net/browse/BOM-2580
This PR aims at refactoring the discussion xblock sub project and moving it within the xmodule directory effectively removing its position as a sub project within edx-platform
* refactor: enterprise dependencies for EdxRestAPIClient replacement
This is a part of https://github.com/openedx/public-engineering/issues/42
- add settings for enterprise-backend-service DOT application
- update utils used by enterprise to get rid of EdxRestAPIClient
- original utils stays in the code (to keep edx-platform api
clients working) till the
https://github.com/openedx/public-engineering/issues/39 deprecation
work will be done
* fix: fix typo in the docstring
Tests which @requires_blockstore (i.e. the Blockstore service) have
been made to run as a unit test using the installed Blockstore app, and
will be run by the platform CI.
The Blockstore service tests can still be run manually by setting
EDXAPP_RUN_BLOCKSTORE_TESTS=1
Related fixes:
* adds blockstore bundle storage settings
* let the studio devstack and test servers serve static files from
the /media URL This allows the blockstore/content libraries API to
serve blockstore assets in dev.
* Wrap ContentLibrary creation in an atomic transaction, so that if it
fails, the related bundle can be deleted directly from the database
during the exception handler. (Previously, we called a REST API which
deleted it as part of a separate service.)
* adds blockstore as a requirement and an installed app, with
configurable bundle storage settings.
* adds waffle switch and setting to allow use of blockstore's python API
instead of REST API in live testservers and in production.
* adds database router which, when a `blockstore` DATABASE connection is
configured, allows the platform to use the blockstore service's
database instead of the default edxapp database.
* replaces blockstore_api exceptions and models with blockstore.app.api classes
* minor fixes to the blockstore_api to make the Blockstore REST API
return data packaged the same as the Blockstore Python API.
Previously, our rate-limiting code trusted the entire `X-Forwarded-For`
header, allowing a malicious client to spoof that header and evade
rate-limiting. This commit introduces a new module and setting
allowing us to make a more conservative choice of IPs.
- Create new `openedx.core.djangoapps.util.ip` module for producing
the IP "external chain" for requests based on the XFF header and the
REMOTE_ADDR.
- Include a function that gives the safest choice of IPs.
- Add new setting `CLOSEST_CLIENT_IP_FROM_HEADERS` for configuring how
the external chain is derived (i.e. setting the trust
boundary). Currently has a default, but we may want to make it
mandatory in the future.
- Change `django-ratelimit` code to use the proximate IP in the external
chain -- the one just outside the trust boundary.
Also:
- Change `XForwardedForMiddleware` to use more conservative choice for
its `REMOTE_ADDR` override
- Other adjustments to `XForwardedForMiddleware` as needed in order to
initialize new module and support code that needs the real
`REMOTE_ADDR` value
- Metrics for observability into the change (and XFF composition)
- Feature switch to restore legacy mode if needed
This also gives us a path forward to removing use of the django-ipware
package, which is no longer maintained and has a handful of bugs that make it
difficult to use safely.
Internal ticket: ARCHBOM-2056
This:
1. Removes the `filestore` property from the `ModuleSystem` in favor of
the `runtime.resources_fs` property.
In the original code, `filestore` is equal to
`DescriptorSystem.runtime.resources_fs`. It's safe to replace it with
`ModuleSystem.runtime.resources_fs` because both runtimes are combined
using the `CachingDescriptorSystem`. It provides the `resources_fs` property
that uses the same file storage.
2. Renames `filestore` argument to `resources_fs` in the `LoncapaSystem`
constructor.
3. Adds the deprecated `filestore` property to the `ModuleSystemShim`
and `RuntimeShim`.
XFF is just part of the chain; record the length of the whole chain instead
(which is always one larger).
Also include junk in one of the test values for realism.
Processing cookies at response time included cookies
that were temporary, like the JWT cookie that is
created by the server by combining the JWT header-payload
and JWT signature cookies. Since we are trying to monitor
the cookie header, we do not want to process this cookie.
However, since we want to include the user id in the logging
message, we delay the logging until response time.
Also, fixed docstring which mislabeled a custom attribute.
ARCHBOM-2055
Deprecates the following attributes from ModuleSystem:
* replace_urls
* replace_course_urls
* replace_jump_to_id_urls
A new ReplaceURLService is created as replacement with a unified replace_urls method
In case of unusual cookie headers containing "Cookie ",
add custom attributes for monitoring:
- cookies.header.corrupt_count
- cookies.header.corrupt_key_count
See annotation documentation for more details.
Separately, updated to skip cookie log sampling for
0 size cookie header.
ARCHBOM-2055
Contains a number of cookie monitoring changes.
Enhancements:
- Add sampling capability for cookie logging on headers
smaller than the threshold. For details, see
COOKIE_SAMPLING_REQUEST_COUNT.
- Add cookie header size to log message.
- Sort logged cookies starting with largest cookie.
- Move logging from Middleware request processing
to response processing to ensure the user id is
available for logging for authenticated calls.
- Added cookies.header.size.computed to check
if there are any large hidden duplicate cookies.
Can be compared against the cookies.header.size
custom attribute.
- Add delimiters into logs to make it simpler to parse
when the logging tools accidentally exports multiple
log lines together.
Removed:
- Legacy cookie capture code. This code was dangerous to
to enable and provided more limited insight than the
newer logging, so this was removed to simplify the code.
Other refactors:
- Switched Middleware to use new Django format, rather
than the Mixin.
- Moved tests to its own test class. Note: this
middleware is likely to move to a separate
library.
ARCHBOM-2055
This should really be all we need for most cases, and we don't want to
emit sensitive data more than necessary, even encrypted. If we need to
inspect one cookie in particular, we can add special logging for that.
Also, change to greater-than-or-equal for threshold to match setting docs.
ref: ARCHBOM-2042
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
* Deprecates ModuleSystem can_execute_unsafe_code, get_python_lib_zip and cache properties
* Adds a new CacheService and SandboxService to provide the deprecated property
* Adds tests for the added CacheService and SandboxService
* Updates the ModuleSystemShim tests in Lms and Studio
Suppress them both in tests (via setup.py and pytest.ini)
and in management command & application runs
(via logsettings.py).
Developers aren't looking at these warnings; they'll be dealt with in a
formal process for upgrading Django. Suppress them for now so that
important information isn't lost in the noise.
