[APER-1941]
We are aware of a product issue that causes a `certificate_available_date` (CAD) to be set for self-paced courses (and thus copied to the course-run's (course) certificate configuration) that causes an issue with learners' Program Records to be inaccurate. The stored CAD in Credentials is causing these certificates to be marked as "unearned" on the Program Record in Credentials, as the IDA believes the learner should *not* have access to them yet (but these certificates are available in the LMS).
A management command was recently introduced in Studio that can be used to clean/remove the `certificate_available_date` data from a course-run in Mongo. These updates aren't making it to the Credentials IDA because of an issue with our logic in the `update_certificate_visible_date_on_course_update` Celery task. This task assumes that we only want to send updates for *Instructor-Paced* courses that have a Certificate Display Behavior set to `end_with_date`. In reality, we need updates to pass to Credentials for _some_ self-paced courses with bad data.
This PR hopes to update our infrastructure to allow these updates to flow to Credentials.
* Improve logging for failed requests to the Credentials IDA's `course_certificates` endpoint when updating a course certificate configuration.
* Update docstrings and comments where appropriate
* Split the logic of the update_certificate_visible_date_on_course_update task into two tasks. The former task will continue to _just_ handle visible_date attribute updates. The latter (new) task will be dedicated to making the REST API call that updates the `certificate_available_date` data in Credentials.
* Update the `handle_course_cert_date_change` function wqhen the COURSE_CERT_DATE_CHANGE signal is received to queue both the "visible_date" and "certificate available date" Celery tasks.
* Update existing tests for the task changes.
**Previously**
When a course administrator changed the `certificates_display_behavior` (presumably to `end_with_date`) AND set the `certificate_available_date` in Studio, the `certificate_available_date` was not syncing to Credentials.
This was because we chose to send the `certificate_available_date` only if the course is self-paced and the `certificate_display_behavior` is set to `end_with_date`. [See PR #28275](https://github.com/openedx/edx-platform/pull/28275). However, we were checking those two conditions by looking at the relevant `CourseOverview`, which was not yet truly saved to reflect the updated display behavior at the time of the check due to atomic requests. [Read more about atomic requests and transactions here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#tying-transactions-to-http-requests-1); we have `ATOMIC_REQUESTS` set to `TRUE` in our codebase. Because the `certificate_display_behavior` was not (yet) `end_with_date`, the post to Credentials was not being fired.
**Solution**
To fix, this commit sends the `COURSE_CERT_DATE_CHANGE` signal `on_commit` instead, which waits until the transaction has completed and the update to the `CourseOverview` has been truly applied to the database. [Read more about `on_commit` here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#django.db.transaction.on_commit). Now, when the relevant `CourseOverview` is read, it will have the updated `certificate_display_behavior`.
See the [Django docs for how to test on_commit callbacks here](https://docs.djangoproject.com/en/3.2/topics/testing/tools/#django.test.TestCase.captureOnCommitCallbacks); this seems to be our first time using the built-in method.
This commit also cleans up some previous code that was meant to get around the problem caused by atomic requests, that is now unneccessary with this fix. It essentially reverses the work done in [PR #26991](https://github.com/openedx/edx-platform/pull/26991): we no longer need to explicitly pass the `certificate_available_date` since we can trust the `CourseOverview` to be properly updated.
**Rejected Solutions**
A. Simply publish the `COURSE_CERT_DATE_CHANGE` signal `on_commit`; no other changes. Rejected because: This would fix the problem, but leaves a lot of unnecessary code and some puzzling inconsistencies. I prefer the solution above because we are cleaning up behind ourselves.
B. Pass the new `certificate_display_behavior` along with the `certificate_available_date`; read those direclty instead of checking the (not-yet-properly-updated) `CourseOverview`. Rejected because: The pattern of passing the new `certificate_available_date` down through all these methods was put in place to get around the atomic requests problem. I believe `on_commit` to be a better solution to getting around that problem. I’d like to move away from passing data down through several functions / methods.
C. Start the celery task `on_commit` (rather than send the signal `on_commit`). Rejected because: The signal receiver basically only starts the celery task, and I find the break to be a bit more readable when sending the signal. No need to split hairs here.
D. Remove the check for pacing and display behavior; send the updated `certificate_available_date` every time there is a change, no matter what the current display behavior is. Rejected because: We intentionally added this check in [PR #28275](https://github.com/openedx/edx-platform/pull/28275) because the task was not behaving as expected without it (specifically around self-paced courses). I assume this is still necessary.
**Relevant Prior Work**
The following PRs--in order--show how this section (and other relevant sections) of the code have been changed over time:
1. [Move cert date signals to avoid race conditions #26841](https://github.com/openedx/edx-platform/pull/26841)
2. [feat: Pass date in cert date update signal #26991](https://github.com/openedx/edx-platform/pull/26991)
3. [Fix certificate available date sync #28275](https://github.com/openedx/edx-platform/pull/28275)
4. [fix: Correct an issue where cert available date was not sent to Crede… #28524](https://github.com/openedx/edx-platform/pull/28524)
MICROBA-1818
When first building the Certificate Date Override feature, I set up the
CertificateDateOverride model to store the override dates as Dates
instead of DateTimes.
Turns out this is not how edX typically handles dates, and it’s causing
some minor headaches around needing to convert values. Also, using just
Dates causes timezone issues.
MICROBA-1488
When sending a GeneratedCertificate to Credentials, send the associated
CertificateDateOverride (if there is one), or else None. This
will be triggered after any save of a GeneratedCertificate, and after
any save or deletion of a single CertificateDateOverride.
Credentials will eventually store its own copy of this date override, or
edit or remove exiting date overrides.
* refactor: Merge the openedx certs app with lms one
Move the certs API from openedx into the lms certificates app.
Functionally, this is a no-op. Cleanup will happen in a subsequent
commit. This is simply a move.
* [fix] Fix certificate available date sync
We were syncing the course available date to every course in
credentials. Since credentials doesn't understand "self-paced" courses,
or course end behaviors, some certificates were time gated incorrectly.
This check make sure to check if the course is not self-paced, and has a
CDB of 'end' before syncing the CA date.
* [feat] Management command to populate credentials availability date
This is a command to populate the new CredentialsCertificate model's
available_date for every existing course_run.
* [feat] calling course certificates api from LMS
Now that CourseCertificates in credentials have a field for the
available_date, we need to make sure we are always updating that field
when it changes in studio. This PR adds a call to a new Credentials API
that will update the field each time the date change signal fires.
[MICROBA-1164]
* cast `course_key` as a string when scheduling the `revoke_program_certificates` task
* Update existing unit tests
* Move test utility method in test_tasks.py out from the middle of the test cases
* Fix spelling in test function name
Because the available date update to the CourseOverview happens inside a
view's signal and we have atomic requests on, the read that was
happening inside the task happened *before* the write was commited to
the database. To avoid the unknown bugs that would come from disabling
atomic transactions for that view (since it's large), this passes the
date we want down through the signals and tasks so we can skip the DB
read at the end.
COURSE_CERT_DATE_CHANGE was being called before saving the new data in
the course overview. The listeners were expecting to pull the data out
of the course overview, and thus were only right about half the time.
This moves the signal to trigger after the course publish signals are
handled.
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
