* Consolidates and renames the runtime used as a base for all the others:
* Before: `xmodule.x_module:DescriptorSystem` and
`xmodule.mako_block:MakoDescriptorSystem`.
* After: `xmodule.x_module:ModuleStoreRuntime`.
* Co-locates and renames the runtimes for importing course OLX:
* Before: `xmodule.x_module:XMLParsingSystem` and
`xmodule.modulestore.xml:ImportSystem`.
* After: `xmodule.modulestore.xml:XMLParsingModuleStoreRuntime` and
`xmodule.modulestore.xml:XMLImportingModuleStoreRuntime`.
* Note: I would have liked to consolidate these, but it would have
involved nontrivial test refactoring.
* Renames the stub Old Mongo runtime:
* Before: `xmodule.modulestore.mongo.base:CachingDescriptorSystem`.
* After: `xmodule.modulestore.mongo.base:OldModuleStoreRuntime`.
* Renames the Split Mongo runtime, the which is what runs courses in LMS and CMS:
* Before: `xmodule.modulestore.split_mongo.caching_descriptor_system:CachingDescriptorSystem`.
* After: `xmodule.modulestore.split_mongo.runtime:SplitModuleStoreRuntime`.
* Renames some of the dummy runtimes used only in unit tests.
The library consisted of this set of utilities and a cli and was only
being used in the edx-platform repo. The CLI will be DEPRed along with
the repo but the code that is being used for retirement will be moved
here.
The Python API for declaring derived settings was confusing to the uninitiated
reader, and also prone to spelling mistakes. This replaces the API with one
that is more readable and more concise, and updates the implementation of
`derive_settings` to properly derive settings declared using the new API.
BREAKING CHANGE: The `derived` and `derived_collection_entry` function are
replaced with the `Derived` class. We do not expect those functions to have
been used outside of edx-platform, but if they are, this commit will cause them
to loudly ImportError.
Note that there should be NO change in behavior to the `derive_settings`
function, which we DO know to be used by some external edx-platform plugins.
Part of: https://github.com/openedx/edx-platform/issues/36215
Implements the connection from the teams feature to the content groups feature. This implementation uses the dynamic partition generator extension point to associate content groups with the users that belong to a Team.
This implementation was heavily inspired by the enrollment tracks dynamic partitions.
To determine whether or not we need to revoke any program certificates, we first run get_revokable_program_uuids. This calls get_certified_programs, which calls get_credentials, which uses the OpenEdx Core utility method get_api_data. get_api_data makes the API call inside a try block, does raise_for_status also inside the try block, and then catches the exception, logs it, and returns an empty result to the caller.
This means that on a failure to call the credentials API, get_credentials can’t tell the difference between a failure to hit the API (because credentials is, as it sometimes is during a notify_programs run, overloaded), or a learner with no program certificates. In this particular case, this is absolute failure, incorrect behavior.
* Adds a new flag, `raise_on_error` which will make `get_api_data` log the exception and then re-raise the HTTPError, defaulting to false in order to avoid changing the behavior on any other callers
* Also: my editor reformatted all of the touched files to our modern code standards, and it seemed appropriate to let it do that.
* Also: added type hints in some cases, because they helped me write the code and debug. Our test suite definitely reports mypy results on type errors so we are verifying that hints are correct.
FIXES: APER-3146
Removes expected part of EXPECTED_ERRORS with a variety of changes.
- In many placed in the code, "expected" was used to mean
"ignored and expected", and all such instances are renamed to "ignored".
- The setting ``EXPECTED_ERRORS`` is renamed to ``IGNORED_ERRORS``,
which better matches how it was being used in the first place.
- The setting ``EXPECTED_ERRORS[REASON_EXPECTED]`` is renamed to ``IGNORED_ERRORS[REASON_IGNORED]``.
- The setting toggle ``EXPECTED_ERRORS[IS_IGNORED]`` is removed,
because it will now always be True.
- The how-to will is renamed to how_tos/logging-and-monitoring-ignored-errors.rst.
See 0002-logging-and-monitoring-expected-errors-removed.rst for more details.
Implements DEPR: https://github.com/openedx/edx-platform/issues/32405
**BREAKING CHANGE:** The rename of the setting ``EXPECTED_ERRORS`` to
``IGNORED_ERRORS``, and ``REASON_EXPECTED`` to ``REASON_IGNORED``,
was implemented without backward compatibility. Simply copy the old settings
with the new name as an expand phase before deleting the old names in the
contract phase.
JIRA: https://openedx.atlassian.net/browse/BOM-2580
This PR aims at refactoring the discussion xblock sub project and moving it within the xmodule directory effectively removing its position as a sub project within edx-platform
Deprecates the following attributes from ModuleSystem:
* replace_urls
* replace_course_urls
* replace_jump_to_id_urls
A new ReplaceURLService is created as replacement with a unified replace_urls method
In case of unusual cookie headers containing "Cookie ",
add custom attributes for monitoring:
- cookies.header.corrupt_count
- cookies.header.corrupt_key_count
See annotation documentation for more details.
Separately, updated to skip cookie log sampling for
0 size cookie header.
ARCHBOM-2055
Contains a number of cookie monitoring changes.
Enhancements:
- Add sampling capability for cookie logging on headers
smaller than the threshold. For details, see
COOKIE_SAMPLING_REQUEST_COUNT.
- Add cookie header size to log message.
- Sort logged cookies starting with largest cookie.
- Move logging from Middleware request processing
to response processing to ensure the user id is
available for logging for authenticated calls.
- Added cookies.header.size.computed to check
if there are any large hidden duplicate cookies.
Can be compared against the cookies.header.size
custom attribute.
- Add delimiters into logs to make it simpler to parse
when the logging tools accidentally exports multiple
log lines together.
Removed:
- Legacy cookie capture code. This code was dangerous to
to enable and provided more limited insight than the
newer logging, so this was removed to simplify the code.
Other refactors:
- Switched Middleware to use new Django format, rather
than the Mixin.
- Moved tests to its own test class. Note: this
middleware is likely to move to a separate
library.
ARCHBOM-2055
This should really be all we need for most cases, and we don't want to
emit sensitive data more than necessary, even encrypted. If we need to
inspect one cookie in particular, we can add special logging for that.
Also, change to greater-than-or-equal for threshold to match setting docs.
ref: ARCHBOM-2042
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
* Deprecates ModuleSystem can_execute_unsafe_code, get_python_lib_zip and cache properties
* Adds a new CacheService and SandboxService to provide the deprecated property
* Adds tests for the added CacheService and SandboxService
* Updates the ModuleSystemShim tests in Lms and Studio
* feat: Add a new way to enable/disable teams
Adds a new mechanism for enabling/disabling the team feature in a course using an 'enabled' field to the teams config.
If this field is set to true, teams is enabled (team sets/groups) still need to be defined. If this is set to false then teams is disabled whether or not team sets are defined.
* fix: review feedback
The learning MFE paths include /course/{course_id} which doesn't match /courses/{course_id} which is what the regex expects. This causes issues with the Wiki when when accessed from the learning MFE doesn't detect that course it's related to in the middleware.
* fix: adding more parameters for cookie monitoring
Added: cookies_total_num, cookies_unaccounted_size.
-Both are to help us gauge how many cookies we are not collecting data for.
Increased: # of cookies data collected
The `error_expected` custom attribute used to contain
both the class name and the error message. This had
the following issues:
* Combining data in the same custom attribute limits
the ability to query.
* The additional error class and message data is only
needed for ignored errors, since this data isn't
available elsewhere.
The following changes were made:
* `error_expected` will always have the value True
if present.
* `error_ignored` no longer exists.
* `error_ignored_class` will contain the error module
and class for ignored errors.
* `error_ignored_message` will contain the error message
for ignored errors.
ARCHBOM-1708
Adds logging and monitoring capabilities for expected
errors. See the ADR and how-to documentation for
details of how to configure and use the EXPECTED_ERRORS
setting and new monitoring and logging.
ARCHBOM-1708
Co-authored-by: Tim McCormack <tmccormack@edx.org>
Instead of adding new attributes for each cookie name we create
consistent attribute names. This should prevent any issues where we
have too many different unique attribute names because the cookie names
are unique to the user.
We added two new settings to make the number of cookies and groups
capture configurable:
- TOP_N_COOKIES_CAPTURED
- TOP_N_COOKIE_GROUPS_CAPTURED
Setting a new metric per cookie name resulted in a lot of metrics
getting added to New Relic. In some cases, this was causing other
more important metrics to not get registered.
We want to be able to easily figure out what our biggest cookies are and we
want to also group cookies by prefix because certain services create multiple
cookies and then put unique identifiers in the cookie name.
For example braze cookie names use the following pattern:
ab.storage.<userId>
ab.storage.<deviceId>
ab.storage.<sessionId>
In this case we want to group all the `ab` cookies together so we can see
their total size.
New attributes:
cookies.<group_prefix>.group.size: The size of a group of cookies. For example
the sum of the size of all braze cookies would be the value of the
`cookies.ab.group.size` attribute.
cookies.max.name: The name of the largest cookie sent by the user.
cookies.max.size: The size of the largest cookie sent by the user.
cookies.max.group.name: The name of the largest group of cookies. A single cookie
counts as a group of one for this calculation.
cookies.max.group.size: The sum total size of all the cookies in the largest group.
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
* Moving plugins infrastructure to edx-django-utils
This PR extracts the code that enables plugins in edx-platform and puts it in edx-django-utils. This is done to allow other IDAS to add plugin functionality.
* Replace track calls with eventtracking in task_track and server_track
* Making events compatible with events emitted via track app
* Fixed broken tests and quality violation
* Removed useless changes in shim. using in to have and keys in event context
Co-authored-by: zia.fazal@arbisoft.com <zia.fazal@arbisoft.com>
