* feat: removing visible_date-to-creds updates per-cert
The credentials IDA now relies on the course certificate configuration
and (if present) `certificate_available_date` for displayability. We no
longer need to send `visible_date` updates for every awarded certificate
when a course overview changes.
* @justinhynes pointed out that the task queues might be populated with the to-be-removed task during a blue-green deployment, and it makes sense to remove the job that queues up the tasks slated for removal _before_ removing the code for those tasks.
* fixed a mock import order: 2 mocks were brought in an opposite order, but until this change, they
always had the same result so nobody had noticed.
FIXES: APER-3535
[APER-3229]
The monolith and the Credentials IDA keep independent records of a course runs certificate availability/visibility preferences. This PR aims to improve the communication between the monolith and the Credentiala IDA to keep the availability date/preference in sync with the monoliths records.
The current code is too strict and actually prevents valid updates in some configurations.
Additionally, the Credentials IDA doesn't understand the concept of "course pacing" (instructor-paced vs self-paced) and has troubles with courses with an availability date of "end". Instead of having to add the concept of course pacing (and syncing more data between the two systems), this PR proposes sending the end date of a course as the "certificate available date" to Credentials.
This way, the Credentials IDA can manage the visibility of awarded credentials in a course run with a display behavior of "end" using the existing feature set and models of the Credentials service.
[APER-1941]
We are aware of a product issue that causes a `certificate_available_date` (CAD) to be set for self-paced courses (and thus copied to the course-run's (course) certificate configuration) that causes an issue with learners' Program Records to be inaccurate. The stored CAD in Credentials is causing these certificates to be marked as "unearned" on the Program Record in Credentials, as the IDA believes the learner should *not* have access to them yet (but these certificates are available in the LMS).
A management command was recently introduced in Studio that can be used to clean/remove the `certificate_available_date` data from a course-run in Mongo. These updates aren't making it to the Credentials IDA because of an issue with our logic in the `update_certificate_visible_date_on_course_update` Celery task. This task assumes that we only want to send updates for *Instructor-Paced* courses that have a Certificate Display Behavior set to `end_with_date`. In reality, we need updates to pass to Credentials for _some_ self-paced courses with bad data.
This PR hopes to update our infrastructure to allow these updates to flow to Credentials.
* Improve logging for failed requests to the Credentials IDA's `course_certificates` endpoint when updating a course certificate configuration.
* Update docstrings and comments where appropriate
* Split the logic of the update_certificate_visible_date_on_course_update task into two tasks. The former task will continue to _just_ handle visible_date attribute updates. The latter (new) task will be dedicated to making the REST API call that updates the `certificate_available_date` data in Credentials.
* Update the `handle_course_cert_date_change` function wqhen the COURSE_CERT_DATE_CHANGE signal is received to queue both the "visible_date" and "certificate available date" Celery tasks.
* Update existing tests for the task changes.
**Previously**
When a course administrator changed the `certificates_display_behavior` (presumably to `end_with_date`) AND set the `certificate_available_date` in Studio, the `certificate_available_date` was not syncing to Credentials.
This was because we chose to send the `certificate_available_date` only if the course is self-paced and the `certificate_display_behavior` is set to `end_with_date`. [See PR #28275](https://github.com/openedx/edx-platform/pull/28275). However, we were checking those two conditions by looking at the relevant `CourseOverview`, which was not yet truly saved to reflect the updated display behavior at the time of the check due to atomic requests. [Read more about atomic requests and transactions here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#tying-transactions-to-http-requests-1); we have `ATOMIC_REQUESTS` set to `TRUE` in our codebase. Because the `certificate_display_behavior` was not (yet) `end_with_date`, the post to Credentials was not being fired.
**Solution**
To fix, this commit sends the `COURSE_CERT_DATE_CHANGE` signal `on_commit` instead, which waits until the transaction has completed and the update to the `CourseOverview` has been truly applied to the database. [Read more about `on_commit` here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#django.db.transaction.on_commit). Now, when the relevant `CourseOverview` is read, it will have the updated `certificate_display_behavior`.
See the [Django docs for how to test on_commit callbacks here](https://docs.djangoproject.com/en/3.2/topics/testing/tools/#django.test.TestCase.captureOnCommitCallbacks); this seems to be our first time using the built-in method.
This commit also cleans up some previous code that was meant to get around the problem caused by atomic requests, that is now unneccessary with this fix. It essentially reverses the work done in [PR #26991](https://github.com/openedx/edx-platform/pull/26991): we no longer need to explicitly pass the `certificate_available_date` since we can trust the `CourseOverview` to be properly updated.
**Rejected Solutions**
A. Simply publish the `COURSE_CERT_DATE_CHANGE` signal `on_commit`; no other changes. Rejected because: This would fix the problem, but leaves a lot of unnecessary code and some puzzling inconsistencies. I prefer the solution above because we are cleaning up behind ourselves.
B. Pass the new `certificate_display_behavior` along with the `certificate_available_date`; read those direclty instead of checking the (not-yet-properly-updated) `CourseOverview`. Rejected because: The pattern of passing the new `certificate_available_date` down through all these methods was put in place to get around the atomic requests problem. I believe `on_commit` to be a better solution to getting around that problem. I’d like to move away from passing data down through several functions / methods.
C. Start the celery task `on_commit` (rather than send the signal `on_commit`). Rejected because: The signal receiver basically only starts the celery task, and I find the break to be a bit more readable when sending the signal. No need to split hairs here.
D. Remove the check for pacing and display behavior; send the updated `certificate_available_date` every time there is a change, no matter what the current display behavior is. Rejected because: We intentionally added this check in [PR #28275](https://github.com/openedx/edx-platform/pull/28275) because the task was not behaving as expected without it (specifically around self-paced courses). I assume this is still necessary.
**Relevant Prior Work**
The following PRs--in order--show how this section (and other relevant sections) of the code have been changed over time:
1. [Move cert date signals to avoid race conditions #26841](https://github.com/openedx/edx-platform/pull/26841)
2. [feat: Pass date in cert date update signal #26991](https://github.com/openedx/edx-platform/pull/26991)
3. [Fix certificate available date sync #28275](https://github.com/openedx/edx-platform/pull/28275)
4. [fix: Correct an issue where cert available date was not sent to Crede… #28524](https://github.com/openedx/edx-platform/pull/28524)
MICROBA-1818
[MICROBA-1164]
* cast `course_key` as a string when scheduling the `revoke_program_certificates` task
* Update existing unit tests
* Move test utility method in test_tasks.py out from the middle of the test cases
* Fix spelling in test function name
Because the available date update to the CourseOverview happens inside a
view's signal and we have atomic requests on, the read that was
happening inside the task happened *before* the write was commited to
the database. To avoid the unknown bugs that would come from disabling
atomic transactions for that view (since it's large), this passes the
date we want down through the signals and tasks so we can skip the DB
read at the end.
COURSE_CERT_DATE_CHANGE was being called before saving the new data in
the course overview. The listeners were expecting to pull the data out
of the course overview, and thus were only right about half the time.
This moves the signal to trigger after the course publish signals are
handled.
The learner records feature had to be enabled/disabled via site-specific
configuration models, which is inconvenient for platforms that want to
disable this feature globally. Here, we introduce a
ENABLE_LEARNER_RECORDS feature toggle in the lms/cms settings that makes
it possible to disable this feature on all sites. Because this feature
toggle is set to True by default, this will not modify the behaviour of
existing platforms.
* INCR-250: Make compatible with Python 3.x without breaking Python 2.7 support --> openedx/core/djangoapps/programs
* INCR-250: Disable pylint warning and replace _f placeholder with actual name
* INCR-250: pylint format correction and fix over length line limit
