First PR to replace pytz with zoneinfo for UTC handling across codebase.
This PR migrates all UTC timezone handling from pytz to Python’s standard
library zoneinfo. The pytz library is now deprecated, and its documentation
recommends using zoneinfo for all new code. This update modernizes our
codebase, removes legacy pytz usage, and ensures compatibility with
current best practices for timezone management in Python 3.9+. No functional
changes to timezone logic - just a direct replacement for UTC handling.
https://github.com/openedx/edx-platform/issues/33980
* refactor: remove PII from log messages in programs tasks
[APER-3723]
Refactors a number of log statements from the Celery tasks in the Programs Django app. This removes username (considered PII) from the log statements and opts to use LMS User ID instead.
* fix: quality
* feat: removing visible_date-to-creds updates per-cert
The credentials IDA now relies on the course certificate configuration
and (if present) `certificate_available_date` for displayability. We no
longer need to send `visible_date` updates for every awarded certificate
when a course overview changes.
* @justinhynes pointed out that the task queues might be populated with the to-be-removed task during a blue-green deployment, and it makes sense to remove the job that queues up the tasks slated for removal _before_ removing the code for those tasks.
* fixed a mock import order: 2 mocks were brought in an opposite order, but until this change, they
always had the same result so nobody had noticed.
FIXES: APER-3535
* feat: switching to celery native backoff for cert awarding
Our homegrown backoff/retry was good enough for a while, but we ran into
a huge disabling event when too many changes were made simultaneously.
Since this code was first written, celery has built in good back
off/retry functionality, including jitter, to make sure that all the
retries don't happen simultaneously.
* Switches to using celery native backoff
* Refactors a huge try/catch block so the exception handling is on
smaller subsets of code
FIXES: APER-3510
* feat: switching to celery native backoff for cert awarding
* Fixed the grammar in a couple of comments
* fixed a couple of lending errors
FIXES: APER-3510
* feat: limiting PII in logs per code review
per code review
FIXES: APER-3510
* feat: code review feedback
* removing some more PII from logs, even where it was not requested
* circuit breaker returned from grading attempt if course key is bad
* add missing tests for that functionality
FIXES: APER-3510
* feat: improved logging
per code review, adding a log message explaining certificate mode
issues
FIXES: APER-3510
* feat: fixing a bad string
lint error, and left off the format string modifier
FIXES: APER-3510
* feat: style
linter error
FIXES: APER-3510
[APER-3229]
In a previous PR, I created a new utility function named `is_credentials_enabled()` that can be used to determine if use of the Credentials IDA is enabled by config in an Open edX instance.
This PR is some additional cleanup that replaces the direct import and use of the `CredentialsApiConfig` model with the new utility function.
I took some additional time to update some existing log messages to include more info while reducing our need to log PII. I've removed as much use of a learner's username as possible, replacing it with logging the learner's LMS User Id instead.
[APER-3229]
The monolith and the Credentials IDA keep independent records of a course runs certificate availability/visibility preferences. This PR aims to improve the communication between the monolith and the Credentiala IDA to keep the availability date/preference in sync with the monoliths records.
The current code is too strict and actually prevents valid updates in some configurations.
Additionally, the Credentials IDA doesn't understand the concept of "course pacing" (instructor-paced vs self-paced) and has troubles with courses with an availability date of "end". Instead of having to add the concept of course pacing (and syncing more data between the two systems), this PR proposes sending the end date of a course as the "certificate available date" to Credentials.
This way, the Credentials IDA can manage the visibility of awarded credentials in a course run with a display behavior of "end" using the existing feature set and models of the Credentials service.
* fixed one test to accommodate a slightly modified workflow
* allowed the automated linter to match current standards
Note: as part of this process, I realized a lot of the tests in `programs/tests/test_tasks.py` are somewhat problematic. Some aren't real unit tests (allow calls to something other than the system under test), at least one for that reason it won't run on local at all, and some mock the wrong part of the system or just don't match current flow. I'm not modifying them as part of this, but they should be looked at.
FIXES: APER-3146
To determine whether or not we need to revoke any program certificates, we first run get_revokable_program_uuids. This calls get_certified_programs, which calls get_credentials, which uses the OpenEdx Core utility method get_api_data. get_api_data makes the API call inside a try block, does raise_for_status also inside the try block, and then catches the exception, logs it, and returns an empty result to the caller.
This means that on a failure to call the credentials API, get_credentials can’t tell the difference between a failure to hit the API (because credentials is, as it sometimes is during a notify_programs run, overloaded), or a learner with no program certificates. In this particular case, this is absolute failure, incorrect behavior.
* Adds a new flag, `raise_on_error` which will make `get_api_data` log the exception and then re-raise the HTTPError, defaulting to false in order to avoid changing the behavior on any other callers
* Also: my editor reformatted all of the touched files to our modern code standards, and it seemed appropriate to let it do that.
* Also: added type hints in some cases, because they helped me write the code and debug. Our test suite definitely reports mypy results on type errors so we are verifying that hints are correct.
FIXES: APER-3146
[APER-1941]
We are aware of a product issue that causes a `certificate_available_date` (CAD) to be set for self-paced courses (and thus copied to the course-run's (course) certificate configuration) that causes an issue with learners' Program Records to be inaccurate. The stored CAD in Credentials is causing these certificates to be marked as "unearned" on the Program Record in Credentials, as the IDA believes the learner should *not* have access to them yet (but these certificates are available in the LMS).
A management command was recently introduced in Studio that can be used to clean/remove the `certificate_available_date` data from a course-run in Mongo. These updates aren't making it to the Credentials IDA because of an issue with our logic in the `update_certificate_visible_date_on_course_update` Celery task. This task assumes that we only want to send updates for *Instructor-Paced* courses that have a Certificate Display Behavior set to `end_with_date`. In reality, we need updates to pass to Credentials for _some_ self-paced courses with bad data.
This PR hopes to update our infrastructure to allow these updates to flow to Credentials.
* Improve logging for failed requests to the Credentials IDA's `course_certificates` endpoint when updating a course certificate configuration.
* Update docstrings and comments where appropriate
* Split the logic of the update_certificate_visible_date_on_course_update task into two tasks. The former task will continue to _just_ handle visible_date attribute updates. The latter (new) task will be dedicated to making the REST API call that updates the `certificate_available_date` data in Credentials.
* Update the `handle_course_cert_date_change` function wqhen the COURSE_CERT_DATE_CHANGE signal is received to queue both the "visible_date" and "certificate available date" Celery tasks.
* Update existing tests for the task changes.
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
When first building the Certificate Date Override feature, I set up the
CertificateDateOverride model to store the override dates as Dates
instead of DateTimes.
Turns out this is not how edX typically handles dates, and it’s causing
some minor headaches around needing to convert values. Also, using just
Dates causes timezone issues.
MICROBA-1488
When sending a GeneratedCertificate to Credentials, send the associated
CertificateDateOverride (if there is one), or else None. This
will be triggered after any save of a GeneratedCertificate, and after
any save or deletion of a single CertificateDateOverride.
Credentials will eventually store its own copy of this date override, or
edit or remove exiting date overrides.
* refactor: Merge the openedx certs app with lms one
Move the certs API from openedx into the lms certificates app.
Functionally, this is a no-op. Cleanup will happen in a subsequent
commit. This is simply a move.
feat: reimagine certificate display settings
The course settings `certificate_available_date` (CAD) and
`certificates_display_behavior` (CDB) were previously
acting indedependantly of one another. They now work in
tandem. This change:
- limits CDB to a dropdown
- removes "early_with_info" and adds "end_with_date"
- only takes CAD into account if "end_with_date" is selected
- Moves CDB to the main course schedule settings page
- updates CourseOverview model and CourseDetails objects to
validate these fields and choose sane defaults if they aren't
expected values
This work was previously done in bd9e7dd (complete with bugs), so this
version is toggleable via the ENABLE_V2_CERT_DISPLAY_SETTINGS setting
The course settings `certificate_available_date` (CAD) and
`certificates_display_behavior` (CDB) were previously acting indedependantly
of one another. They now work in tandem. This change:
- limits CDB to a dropdown
- removes "early_with_info" and adds "end_with_date"
- only takes CAD into account if "end_with_date" is selected
- Moves CDB to the main course schedule settings page
- updates CourseOverview to validate these fields and choose sane
defaults if they aren't expected values
Certificates will now show under the following circumstances:
"Immediately upon passing"
certificate_availability_date = null
certificates_display_behavior = "early_no_info"
"End date of course"
certificate_availability_date = null
certificates_display_behavior = "end"
"A date after the course end date"
certificate_availability_date = <date>
certificates_display_behavior = "end_with_date"
Sometimes learners have certificates in old course runs which've been
deleted. When this happens loading the learner's program progress can
result in 500 errors. This corrects those by choosing to count any
non-existent course the learner has certificates for as counting
towards program completion, effectively assuming that availability
dates have passed for all such courses.
Also fixes an error with a condition related to how we determine
whether an enrolled course is considered "in progress". The previous
version of the code had a bug that would result in a lot more courses
being marked "in progress" for the purpose of program completion than
actually were.
JIRA:EDUCATOR-5787
* [feat] calling course certificates api from LMS
Now that CourseCertificates in credentials have a field for the
available_date, we need to make sure we are always updating that field
when it changes in studio. This PR adds a call to a new Credentials API
that will update the field each time the date change signal fires.
notify_credentials command
In this PR we are removing a management command that has performance issues and duplicates logic not available in the notify_credentials management command.
count
[MICROBA-1163]
This change will correct an issue in the Program Dashboard where a user
would see a course as completed, but not see their Certificate because
it was not available to them yet.
[MICROBA-1164]
* cast `course_key` as a string when scheduling the `revoke_program_certificates` task
* Update existing unit tests
* Move test utility method in test_tasks.py out from the middle of the test cases
* Fix spelling in test function name
COURSE_CERT_DATE_CHANGE was being called before saving the new data in
the course overview. The listeners were expecting to pull the data out
of the course overview, and thus were only right about half the time.
This moves the signal to trigger after the course publish signals are
handled.
