We'd like to finish the removal of MD4 key hashing, but want to know if
cutting over to BLAKE2b will cause too large a cache turnover. Hopefully
this monitoring will give us some insight into the rate of large keys.
PII Annotations are very out of date, this commit adds most that were
missing in edx-platform, and some additional annotations to the
safelist. It is not comprehensive, several other upstream Open edX
packages also need to be updated. It also does not include removing
annotations that have been moved upstream, or been removed entirely.
Those are separate follow-on tasks.
* feat!: removes deprecated v1 certificate behavior
this removes the long-deprecated v1 certificate behavior. This removes
the old-style date selection behavior (ie., not a choice between
*Immediately upon passing*, *End date of course*, *A date after the course
end date*), which is no longer reliably maintained or supported in
Studio or Credentials.
FIXES: #35399
1. Upgraded Python dependency edx-enterprise
- Removed unencrypted credentials from SAP configuration model
2. Test updates
- Skipped `test_migrations_are_in_sync` for unencrypted credentials removal
- Updated related tests and requirements.
Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master`
- Upgraded Python dependency for edx-enterprise
- Removed references to the char field `decrypted_secret`
- Updated the skip reason message for `test_migrations_are_in_sync`
Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master`
Some models in third_party_auth used settings.SITE_ID as a field
default, which caused Django to say migrations were out of sync whenever
settings.SITE_ID happened to be anything other than 1 for any developer:
Your models in app(s): 'third_party_auth' have changes that are not
yet reflected in a migration, and so won't be applied. Run
'manage.py makemigrations' to make new migrations, and then re-run
'manage.py migrate' to apply them.
This could easily happen if a developer is testing out site
configuration or site-specific theming and ends up with a SITE_ID other
than 1.
The fix, inspired by a StackOverflow answer [1], is to simply create
a wrapper function for the dynamic default value. The wrapper function,
rather than the current value of SITE_ID, will be serialized to the
migraiton file.
This commit includes a migration file, but from a database perspective,
the migration is a no-op.
[1] https://stackoverflow.com/a/12654998
This is an attempt to fix a performance problem on the libraries home page. When you go to studio home and click on the libraries tab, on prod it will be quick for admins but extremely slow for course instructors (> 12 seconds) and leads to timeouts. It grows with the number of libraries that are assigned to the instructor.
The Python code for the request to load libraries for a particular user goes through all existing libraries and then checks all of the user's roles for each library, which results in a complexity of O(l*r), l=libraries, r=roles. This PR improves the complexity to O(l).
The BulkRoleCache and RoleCache classes were using a python set to store all roles for a particular user. A user can have a large number of roles, and lookup speed of iterating through a set is slow (O(n)). Most roles don't have the same course id, however. So if you have the course id of the role you're looking for, we can use a dict of course ids that contain related roles. The number of roles per course id is negligible, so we arrive at a lookup speed of O(1) when looking up a user's roles that belong to a specific course id.
The BulkRoleCache now caches and stores user roles in a data structure like this:
{
user_id_1: {
course_id_1: {role1, role2, role3}, # Set of roles associated with course_id_1
course_id_2: {role4, role5, role6}, # Set of roles associated with course_id_2
[ROLE_CACHE_UNGROUPED_ROLES_KEY]: {role7, role8} # Set of roles not tied to any specific course or library. For example, Global Staff roles.
},
user_id_2: { ... } # Similar structure for another user
}
While this changes the data structure used to store roles under the hood and adds the new property `roles_by_course_id` to the RoleCache,
when initializing the RoleCache will store roles additionally in the previous data structure - as a flat set - in the `_roles` property accessible via `all_roles_set`. This establishes
backwards compatibility.
We are now storing roles twice in the RoleCache (in each of the two data structures), which means this takes twice as much memory, but only in the scope of a request.
Blockstore and all of its (experimental) functionality has been replaced with
openedx-learning, aka "Learning Core". This commit uninstalls the now-unused
openedx-blockstore package and removes all dangling references to it.
Note: This also removes the `copy_library_from_v1_to_v2` management command,
which has been broken ever since we switched from Blockstore to Learning Core.
Part of this DEPR: https://github.com/openedx/public-engineering/issues/238
