**Previously**
When a course administrator changed the `certificates_display_behavior` (presumably to `end_with_date`) AND set the `certificate_available_date` in Studio, the `certificate_available_date` was not syncing to Credentials.
This was because we chose to send the `certificate_available_date` only if the course is self-paced and the `certificate_display_behavior` is set to `end_with_date`. [See PR #28275](https://github.com/openedx/edx-platform/pull/28275). However, we were checking those two conditions by looking at the relevant `CourseOverview`, which was not yet truly saved to reflect the updated display behavior at the time of the check due to atomic requests. [Read more about atomic requests and transactions here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#tying-transactions-to-http-requests-1); we have `ATOMIC_REQUESTS` set to `TRUE` in our codebase. Because the `certificate_display_behavior` was not (yet) `end_with_date`, the post to Credentials was not being fired.
**Solution**
To fix, this commit sends the `COURSE_CERT_DATE_CHANGE` signal `on_commit` instead, which waits until the transaction has completed and the update to the `CourseOverview` has been truly applied to the database. [Read more about `on_commit` here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#django.db.transaction.on_commit). Now, when the relevant `CourseOverview` is read, it will have the updated `certificate_display_behavior`.
See the [Django docs for how to test on_commit callbacks here](https://docs.djangoproject.com/en/3.2/topics/testing/tools/#django.test.TestCase.captureOnCommitCallbacks); this seems to be our first time using the built-in method.
This commit also cleans up some previous code that was meant to get around the problem caused by atomic requests, that is now unneccessary with this fix. It essentially reverses the work done in [PR #26991](https://github.com/openedx/edx-platform/pull/26991): we no longer need to explicitly pass the `certificate_available_date` since we can trust the `CourseOverview` to be properly updated.
**Rejected Solutions**
A. Simply publish the `COURSE_CERT_DATE_CHANGE` signal `on_commit`; no other changes. Rejected because: This would fix the problem, but leaves a lot of unnecessary code and some puzzling inconsistencies. I prefer the solution above because we are cleaning up behind ourselves.
B. Pass the new `certificate_display_behavior` along with the `certificate_available_date`; read those direclty instead of checking the (not-yet-properly-updated) `CourseOverview`. Rejected because: The pattern of passing the new `certificate_available_date` down through all these methods was put in place to get around the atomic requests problem. I believe `on_commit` to be a better solution to getting around that problem. I’d like to move away from passing data down through several functions / methods.
C. Start the celery task `on_commit` (rather than send the signal `on_commit`). Rejected because: The signal receiver basically only starts the celery task, and I find the break to be a bit more readable when sending the signal. No need to split hairs here.
D. Remove the check for pacing and display behavior; send the updated `certificate_available_date` every time there is a change, no matter what the current display behavior is. Rejected because: We intentionally added this check in [PR #28275](https://github.com/openedx/edx-platform/pull/28275) because the task was not behaving as expected without it (specifically around self-paced courses). I assume this is still necessary.
**Relevant Prior Work**
The following PRs--in order--show how this section (and other relevant sections) of the code have been changed over time:
1. [Move cert date signals to avoid race conditions #26841](https://github.com/openedx/edx-platform/pull/26841)
2. [feat: Pass date in cert date update signal #26991](https://github.com/openedx/edx-platform/pull/26991)
3. [Fix certificate available date sync #28275](https://github.com/openedx/edx-platform/pull/28275)
4. [fix: Correct an issue where cert available date was not sent to Crede… #28524](https://github.com/openedx/edx-platform/pull/28524)
MICROBA-1818
Fix for an error related to endless recursion
Dev Notes:
The student gets the passing grade
-> CourseGradeFactory sends the `COURSE_GRADE_NOW_PASSED` signal after the grade calculation
-> `listen_for_passing_grade` receives the signal and calls `generate_certificate_task`
-> `generate_certificate_task` calls `_generate_regular_certificate_task`
-> `_generate_regular_certificate_task` trying to get grade by calling the `_get_course_grade`
-> `_get_course_grade` calls `CourseGradeFactory().read()` but I have persistent grades off, so actually that ends with `CourseGradeFactory().update()`
-> `CourseGradeFactory().update()` sends the `COURSE_GRADE_NOW_PASSED`
And so on
Contains a number of cookie monitoring changes.
Enhancements:
- Add sampling capability for cookie logging on headers
smaller than the threshold. For details, see
COOKIE_SAMPLING_REQUEST_COUNT.
- Add cookie header size to log message.
- Sort logged cookies starting with largest cookie.
- Move logging from Middleware request processing
to response processing to ensure the user id is
available for logging for authenticated calls.
- Added cookies.header.size.computed to check
if there are any large hidden duplicate cookies.
Can be compared against the cookies.header.size
custom attribute.
- Add delimiters into logs to make it simpler to parse
when the logging tools accidentally exports multiple
log lines together.
Removed:
- Legacy cookie capture code. This code was dangerous to
to enable and provided more limited insight than the
newer logging, so this was removed to simplify the code.
Other refactors:
- Switched Middleware to use new Django format, rather
than the Mixin.
- Moved tests to its own test class. Note: this
middleware is likely to move to a separate
library.
ARCHBOM-2055
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
MST-1161. The regeneration command for unverified certs has to be used to regenerate all unverified certs after the release of the honor code feature on January 19th. Because of this, the command is now updated to regenerate all unverified certificates if no specific course keys are provided. I've also included an option to include excluded keys, so that all unverified certs, except for those in an excluded course key, can be regenerated.
MST-1190. If the integrity signature flag has been enabled for a course, update the verified certificate description. The flag value should also be accessible to cert templates, so it has been added to the context.
[MICROBA-1594]
* Update course certificate generation logic when ID verification fails to check if the enrollment mode requires IDV.
This change fixes an issue where Honor certificates could not be generated in an Open edX installation unless a manual IDV override was added for the student.
The VERIFIED_NAME_FLAG, the VerifiedNameEnabledView, and the verified_name_enabled key removed from responses for both VerifiedNameView view and VerifiedNameHistoryView
were removed as part https://github.com/edx/edx-name-affirmation/pull/12. This was released in version 2.0.0 of the edx-name-affirmation PyPI package. Please see below for additional context for the removal, copied from the name-affirmation commit message.
The VERIFIED_NAME_FLAG was added as part https://github.com/edx/edx-name-affirmation/pull/12, [MST-801](https://openedx.atlassian.net/browse/MST-801) in order to control the release of the Verified Name project. It was used for a phased roll out by percentage of users.
The release reached a percentage of 50% before it was observed that, due to the way percentage roll out works in django-waffle, the code to create or update VerifiedName records was not working properly. The code was written such that any change to a SoftwareSecurePhotoVerification model instance sent a signal, which was received and handled by the Name Affirmation application. If the VERIFIED_NAME_FLAG was on for the requesting user, a Celery task was launched from the Name Affirmation application to perform the creation of or update to the appropriate VerifiedName model instances based on the verify_student application signal. However, we observed that when SoftwareSecurePhotoVerification records were moved into the "created" or "ready" status, a Celery task in Name Affirmation was created, but when SoftwareSecurePhotoVerification records were moved into the "submitted" status, the corresponding Celery task in Name Affirmation was not created. This caused VerifiedName records to stay in the "pending" state.
The django-waffle waffle flag used by the edx-toggle library implements percentage rollout by setting a cookie in a learner's browser session to assign them to the enabled or disabled group.
It turns out that the code that submits a SoftwareSecurePhotoVerification record, which moves it into the "submitted" state, happens as part of a Celery task in the verify_student application in the edx-platform. Therefore, we believe that because there is no request object in a Celery task, the edx-toggle code is defaulting to the case where there is no request object. In this case, the code checks whether the flag is enabled for everyone when determining whether the flag is enabled. Because of the percentage rollout (i.e. waffle flag not enabled for everyone), the Celery task in Name Affirmation is not created. This behavior was confirmed by logging added as part of https://github.com/edx/edx-name-affirmation/pull/62.
We have determined that we do not need the waffle flag, as we are comfortable that enabling the waffle flag for everyone will fix the issue and are comfortable releasing the feature to all users. For this reason, we are removing references to the flag.
[MST-1130](https://openedx.atlassian.net/browse/MST-1130)
Only courses which actually have honor code on and only unverified
certs will be regenerated. All other certs wouldn't change so
don't cause trouble.
MST-855
this appears to be a throttling method on first glance but is instead
an async task handling hack, explain it
and then provide a way to override it for tight control of timing
while regenerating multiple certs
IDV is on its way to retirement, so it's not going to be necessary for cert
generation forever.
Introduces a function to combine the honor code flag with IDV to tell
cert generation if it should care about a missing verification.
Various tests expanded to cover the retired case. The additional calls
in test_task_helper.py are caused by one call to fetch course
overrides which finds none, and that forces one check of the
background flag per student, 71 + 1 + 5 = 77.
MST-854
Includes:
- general documentation
- links to individual events definitions and location
- adding examples to events docs
- adding annotations at the trigger location
rest-condition has not been updated for a while, and with DRF 3.9 it is no longer needed since the ability to combine Permission classes using boolean operators is now provided by DRF directly.
- Fixed LANGUAGE_COOKIE settings name to LANGUAGE_COOKIE_NAME beacuse later is recognised by django
- Added test to verify cookies use in dark lang middleware
- Fixing Django 3.0 tests
* Add COURSE_ENROLLMENT_CHANGED: sent after the enrollment update
* Add COURSE_UNENROLLMENT_COMPLETED: sent after the user's unenrollment
* Add CERTIFICATE_CREATED after the user's certificate generation has
been completed
* Add CERTIFICATE_CHANGED: after the certification update has been
completed
* Add CERTIFICATE_REVOKED: after the certificate revocation has been
completed
* Add COHORT_MEMBERSHIP_CHANGED: when a cohort membership update ends
When first building the Certificate Date Override feature, I set up the
CertificateDateOverride model to store the override dates as Dates
instead of DateTimes.
Turns out this is not how edX typically handles dates, and it’s causing
some minor headaches around needing to convert values. Also, using just
Dates causes timezone issues.
MICROBA-1488
We use the `notify_credentials` management command to keep certificate-
related data in the LMS and Credentials service in sync. We can run it
with specific arguments (user_ids, course_keys, etc.) when we notice a
data discrepancy; and it is run regularly by a Jenkins job with the
`--auto` flag every ~4 hours to keep things up-to-date.
Because we probably never want to notify credentials of of ALL the
GeneratedCertificates, the celery task must be given some arguments
to filter down to the relevant certificates. Running the management
command with the `--auto` flag (as the Jenkins job does) adds
`start_date` and `end_date` arguments of 4 hours ago and now,
respectively.
The handle_notify_credentials celery task then takes those arguments and
looks for any GeneratedCertificates that have been modified within the
given time range by checking the GeneratedCertificate modified_date.
It will send the current data for those certificates to credentials.
However, we also want to notify credentials about certificates that have
an associated CertificateDateOverride that has changed within that time
range: added, updated, or deleted. But changes to a
CertificateDateOverride won’t affect the GeneratedCertificate’s modified
date, and therefore wouldn’t be included in the list of certs cent to
credentials.
This commit adds a check for changed CertificateDateOverrides and
includes their associated GeneratedCertificates in the list of certs. We
use the CertificateDateOverride’s history model for this check so that
we can include certificates whose override was deleted.
MICROBA-1489
Previously, using the bulk delete tool in the Django admin for
`CertificateDateOverride`s would not trigger the post to the credentials
service to delete the corresponding date override stored there. This was
because we were sending the COURSE_CERT_CHANGED signal from an
overridden version of the delete() method. But the Django docs warn:
"Note that the delete() method for an object is not necessarily called
when deleting objects in bulk using a QuerySet or as a result of a
cascading delete. To ensure customized delete logic gets executed, you
can use pre_delete and/or post_delete signals."
Now we use the post_delete signal instead of overriding the delete()
method to make sure the COURSE_CERT_CHANGED signal is sent for both
individual and bulk deletes.
MICROBA-1487
Because we have ATOMIC_REQUESTS turned on, the COURSE_CERT_CHANGED
signal was being sent *before* the date override was committed to the
database. This means that the date override data sent to credentials on
CertificateDateOverride save() was always one commit behind.
Django provides an on_commit() function to allow us to perform actions
only after a transaction is completed successfully. I believe this
is when we want this signal to be sent.
This is an alternative to 1) disabling atomic transactions for a view
(honestly, I’m not sure which view we would target anyway), or 2)
passing more data down through the signal.
Toward MICROBA-1423
Certificate tests required an update to switch from boolean verified
flag to status. This feature is not deployed yet so only test uses
needed attention.
MST-969 and friends
When sending a GeneratedCertificate to Credentials, send the associated
CertificateDateOverride (if there is one), or else None. This
will be triggered after any save of a GeneratedCertificate, and after
any save or deletion of a single CertificateDateOverride.
Credentials will eventually store its own copy of this date override, or
edit or remove exiting date overrides.
