As part of dissolving our sub-projects in edx-platform, we are moving this package under the xmodule directory.
We have fixed all the occurences of import of this package and also fixed all documents related references.
This might break your platform if you have any reference of `import capa` or `from capa import` in your codebase or in any Xblock.
Ref: https://openedx.atlassian.net/browse/BOM-2582
This removes the `rebind_noauth_module_to_user` argument from the
ModuleSystem constructor and moves it to a separate service called
"rebinder" in the class `RebindModuleService`. This is used in the
LTI module to bind calls received by its noauth endpoint to bind
the module the real_user.
Deprecates the following attributes from ModuleSystem:
* replace_urls
* replace_course_urls
* replace_jump_to_id_urls
A new ReplaceURLService is created as replacement with a unified replace_urls method
Description
This is a follow up to #29058 and #29413. This is the next step in moving part of the modulestore data (the course indexes / "active versions" table) from MongoDB to MySQL.
There are four steps planned in moving course index data to MySQL:
Step 1: create the tables in MySQL, start writing to MySQL + MongoDB ✅ done
Step 2: migrate all remaining courses to MySQL ✅ done
Step 3: switch reads from MongoDB to MySQL (this PR)
Step 4 (much later, once we know this is working well): stop writing to MongoDB altogether.
Supporting information
OpenCraft Jira ticket: MNG-2557
Status
✅ Tested with a large Open edX instance is in progress.
Testing instructions
Try making changes in Studio and verify that they work fine.
Deadline
None
`videosequence` and `problemset` have been replaced with `sequential`.
`problemset` and `videosequence` are old-but-not-entirely-unused aliases to the `sequential` block type (in Studio-speak, "Subsection").
Since [these block types have been removed from the 6 courses that used them](https://openedx.atlassian.net/browse/DEPR-151?focusedCommentId=588197), this ticket removes the support for the `problemset` and `videosequence` block-types.
For more information, see ticket: [DEPR-151](https://openedx.atlassian.net/browse/DEPR-151)
Convert more tests from MONGO_AMNESTY to SPLIT modulestores.
This is in preparation for just wholesale denying access to Old
Mongo, so I either converted tests to split or just deleted some
test variants that were Old Mongo specific. (e.g. ddt lines)
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
* Deprecates ModuleSystem can_execute_unsafe_code, get_python_lib_zip and cache properties
* Adds a new CacheService and SandboxService to provide the deprecated property
* Adds tests for the added CacheService and SandboxService
* Updates the ModuleSystemShim tests in Lms and Studio
Previously, it had some basic manual masquerading by calling the
endpoint with ?user=mytestuser. But this adds standard session
masquerading support to the endpoint as well.
This support is limited by LS's own partition group support. It
only looks at the enrollment track partition currently. Further
FBE and cohort partition support will come later.
But this commit opens up normal session masquerading for:
- Generic student
- Specific student
- Enrollment track
AA-1151
from the lms get_module_for_descriptor and related methods.
This prefix was previously inferred from the request's base URL and
passed through several methods to reach the XQueueService creation.
But the request base URL always matches the LMS_ROOT_URL (or the preview
URL), so this change simply uses that setting instead.
* Deprecates ModuleSystem.xqueue property
* Adds new XQueueService to provide the deprecated property values to the LMS runtime
(Studio does not need the XQueueService.)
* Adds tests for new service and updates the ModuleSystemShim tests in LMS and Studio
* Fixes existing tests.
Deprecates these ModuleSystem attributes in favor of the user service:
* user_location
* get_real_user
* get_user_role
Related changes:
* Stores the user location into DjangoXBlockUserService's optional attribute as request_country_code
* Uses the student model's user_by_anonymous_it to fetch the (cached) real user
* Updates affected tests
* refactor: deprecates ModuleSystem.render_template
in favor of the added MakoSystem render_template method.
Related changes:
* Adds the MakoService to the StudioEditModuleRuntime,
PreviewModuleSystem, LmsModuleSystem, and XBlockRuntime
* MakoService constructor takes a `namespace_prefix` string, so that the
CMS PreviewModuleSystem can render to LMS templates, without needing
the special render_from_lms helper method.
* ModuleSystem.render_template becomes a read-only property, so the
constructor calls and test module systems are updated accordingly.
* Adds tests for the MakoService and module system shims.
(cherry picked from commit 457f959356)
* refactor: use MakoService.render_template to remove deprecation warnings
from block code.
(cherry picked from commit 8d62d337f5)
* refactor: use MakoService.render_template to remove deprecation warnings
from test code.
(cherry picked from commit 26b43465a4)
* test: Adds a test to verify the bug introduced by the previous changes
The AuthoringMixin is automatically added to all XBlocks (see
settings.XBLOCK_MIXINS), and AuthoringMixin.visibility_view expects the
"mako" service.
This test verifies the bug by testing the PureXBlock, which does not
require the "mako" service, and so fails when the visibility_view is
rendered.
* fix: AuthoringMixin needs mako service
which fixes the visibility_view for XBlocks which don't explicitly
require the mako service.
Also removes the unneeded class property _services_requested from
AuthoringMixin and StudioEditableBlock. This property is better provided
by the XBlockMixin class.
in favor of the added MakoSystem render_template method.
Related changes:
* Adds the MakoService to the StudioEditModuleRuntime,
PreviewModuleSystem, LmsModuleSystem, and XBlockRuntime
* MakoService constructor takes a `namespace_prefix` string, so that the
CMS PreviewModuleSystem can render to LMS templates, without needing
the special render_from_lms helper method.
* ModuleSystem.render_template becomes a read-only property, so the
constructor calls and test module systems are updated accordingly.
* Adds tests for the MakoService and module system shims.
This bug occurs when get_module_system_for_user is called more than once
per request, for blocks with different values for the
requires_per_student_anonymous_id property.
Removes references to these deprecated attributes from the platform code:
* runtime.anonymous_student_id
* runtime.seed
* runtime.user_id
* runtime.user_is_staff
Related changes:
* Ensure that all platform XBlocks which use these attributes "need" the user service.
* ProblemBlock: Removes check for existence of runtime.seed attribute in preparation for removal of this attribute from ModuleSystem.
* edxnotes: Catches NoSuchServiceError just in case some XBlocks using notes don't have the user service.
* UserTagsService refactor: pass user and course_id on creation
(cherry picked from commit 753839276f)
The following ModuleSystem attributes are deprecated by this change, and should be pulled directly from the user service instead:
* anonymous_student_id
* seed
* user_id
* user_is_staff
Related changes:
* Removes the `user` and `anonymous_student_id` parameters from the ModuleService constructor.
* Stores anonymous_user_id in XBlockDjangoUserService's opt_attr
* Pulls out constants used by DjangoXBlockUserService opt_attr so they can be used in the platform code.
* LmsModuleSystem uses the user service created in wrapper function for runtime.publish to avoid requiring the user
service to be "needed" by all XBlocks.
* LmsModuleSystem no longer checks for instances of XModuleDescriptor when deciding what kind of anonymous_user_id to
provide: all XModules are XBlocks, so this check is unnecessary.
* XBlockRuntime returns a user service when requested
* Adds tests for deprecated ModuleSystem attributes and changes to XBlockDjangoUserService.
(cherry picked from commit c41e7fb93a)
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
This commit starts writing course indexes (active_versions) to both MySQL and Mongo, but continues to read from MongoDB only.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.
By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
Removes references to these deprecated attributes from the platform code:
* runtime.anonymous_student_id
* runtime.seed
* runtime.user_id
* runtime.user_is_staff
Related changes:
* Ensure that all platform XBlocks which use these attributes "need" the user service.
* ProblemBlock: Removes check for existence of runtime.seed attribute in preparation for removal of this attribute from ModuleSystem.
* edxnotes: Catches NoSuchServiceError just in case some XBlocks using notes don't have the user service.
* UserTagsService refactor: pass user and course_id on creation
The following ModuleSystem attributes are deprecated by this change, and should be pulled directly from the user service instead:
* anonymous_student_id
* seed
* user_id
* user_is_staff
Related changes:
* Removes the `user` and `anonymous_student_id` parameters from the ModuleService constructor.
* Stores anonymous_user_id in XBlockDjangoUserService's opt_attr
* Pulls out constants used by DjangoXBlockUserService opt_attr so they can be used in the platform code.
* LmsModuleSystem uses the user service created in wrapper function for runtime.publish to avoid requiring the user
service to be "needed" by all XBlocks.
* LmsModuleSystem no longer checks for instances of XModuleDescriptor when deciding what kind of anonymous_user_id to
provide: all XModules are XBlocks, so this check is unnecessary.
* XBlockRuntime returns a user service when requested
* Adds tests for deprecated ModuleSystem attributes and changes to XBlockDjangoUserService.
Because xblock handlers normally get a block tree that already has
inaccessible blocks stripped, they don't see FBE gated blocks at
all. So the get_completion handler would return True for FBE units
incorrectly. Leading to a visual bug as an audit user went through
their units in the courseware.
In order to let the handler know about the full tree, I've added a
new attribute you can set on your xblock handlers:
my_handler.will_recheck_access = True
This will tell the top-level handler code get the full tree for you.
As part of this, I've also changed the sequence xblock handler's
into proper xblock handlers (not old-style xmodule handlers).
This changes their URLs slightly. I've kept the old URLs for now
as well, but they'll be removed after Maple.
AA-409
In https://github.com/edx/edx-platform/pull/25955 `HiddenDescriptor`
(which was a subclass of `RawDescriptor` with a custom `student_view()`)
was converted to an XBlock. It is used as the `default_class` by the
`CachingDescriptorSystem` classes. However `RawDescriptor` is still
being used by `XMLModuleStore`. This has been replaced by
`HiddenDescriptor` as well.
This release adds a new permission check when rendering the practice proctored exam view. If a learner is not in a paid enrollment track, the learner will no longer be able to use proctoring software in a practice proctored exam. As with other proctored exams, learners not in a paid enrollment track will see the exam content. This uses the 'can_take_proctored_exam' permission.
Now that we always return an existing value from the DB rather than trusting that ID generation is deterministic and constant over time, we're free to change the generation algorithm.
Our long term goal is to switch to random IDs, but we need to first investigate the uses of save=False. In the meantime, this is a good opportunity to move away from MD5, which has a number of cryptographic weaknesses. None of the known vulnerabilities are considered exploitable in this location, given the limited ability to control the input to the hash, but we should generally be moving away from it everywhere for consistency.
This change should not be breaking even for save=False callers, since those calls are extremely rare (1 in 100,000) and should only occur after a save=True call, at which point they'll use the stored value. Even if this were not true, for a save=False/True pair of calls to result in a mismatch in output, the first of the calls would have to occur around the time of the deploy of this code.
Co-authored-by: Tim McCormack <tmccormack@edx.org>
Co-authored-by: Tim McCormack <tmccormack@edx.org>
