* refactor(certificates): replace direct model imports with data classes and APIs
* fix: use Certificates API to create certificates
* docs: update docstring for get_certificate_for_user
* fix: remove trailing whitespace
---------
Co-authored-by: coder1918 <ram.chandra@wgu.edu>
Co-authored-by: Deborah Kaplan <deborahgu@users.noreply.github.com>
First PR to replace pytz with zoneinfo for UTC handling across codebase.
This PR migrates all UTC timezone handling from pytz to Python’s standard
library zoneinfo. The pytz library is now deprecated, and its documentation
recommends using zoneinfo for all new code. This update modernizes our
codebase, removes legacy pytz usage, and ensures compatibility with
current best practices for timezone management in Python 3.9+. No functional
changes to timezone logic - just a direct replacement for UTC handling.
https://github.com/openedx/edx-platform/issues/33980
* chore: improving programs rest API docs, type hints
reformatting the docstrings so they appear correctly in the openAPI
docs.
* moving them into the `GET`
* some cleanup and simplification of language
* removing real data that refers to specific hosts
regenerated the open API documentation.
also adding type hints to the API and telling `mypy` to check this
file. This was primarily because these models are confusing enough in
order to verify that the documentation was correct I had to do a lot of
jumping through the code.
FIXES: APER-3950
Moves the Program Dashboard APIs out of the deprecated remnants of the legacy learner dashboard, into the Programs djangoapp.
Keeps the old legacy routes for this API, left over from the deprecated remnants of the legacy learner dashboard, alongside future-proofed routes which will work when the deprecated, legacy Program Dashboard is eventually replaced with functionality in the Learner Dashboard MFE.
FIXES: APER-3949
PII Annotations are very out of date, this commit adds most that were
missing in edx-platform, and some additional annotations to the
safelist. It is not comprehensive, several other upstream Open edX
packages also need to be updated. It also does not include removing
annotations that have been moved upstream, or been removed entirely.
Those are separate follow-on tasks.
* refactor: remove PII from log messages in programs tasks
[APER-3723]
Refactors a number of log statements from the Celery tasks in the Programs Django app. This removes username (considered PII) from the log statements and opts to use LMS User ID instead.
* fix: quality
we have a periodic error that makes no sense when the certificate
available date is none. This improves logging for the API call. Because
the only thing being changed is a course certificate, there is no risk
of PII being exposed in the new logging.
* feat: removing visible_date-to-creds updates per-cert
The credentials IDA now relies on the course certificate configuration
and (if present) `certificate_available_date` for displayability. We no
longer need to send `visible_date` updates for every awarded certificate
when a course overview changes.
* @justinhynes pointed out that the task queues might be populated with the to-be-removed task during a blue-green deployment, and it makes sense to remove the job that queues up the tasks slated for removal _before_ removing the code for those tasks.
* fixed a mock import order: 2 mocks were brought in an opposite order, but until this change, they
always had the same result so nobody had noticed.
FIXES: APER-3535
* feat: linting before touching all these files
All these files are old enough, relative either to our current linting
rules or our current linter automation, that modifying anything in them
either makes the linter cranky or wants to reformat the entire file.
Rather than mixing cleanup with code changes, this commit just lints
this set of files to our current standards.
The LMS `certificates` app `README` has been, and the
openedx core `credentials` and `programs` READMEs have been created.
This will clarify not only what the limited responsibilities
of each of these django apps is, but also the way they interact with
each other.
FIXES: APER-2929
* feat: switching to celery native backoff for cert awarding
Our homegrown backoff/retry was good enough for a while, but we ran into
a huge disabling event when too many changes were made simultaneously.
Since this code was first written, celery has built in good back
off/retry functionality, including jitter, to make sure that all the
retries don't happen simultaneously.
* Switches to using celery native backoff
* Refactors a huge try/catch block so the exception handling is on
smaller subsets of code
FIXES: APER-3510
* feat: switching to celery native backoff for cert awarding
* Fixed the grammar in a couple of comments
* fixed a couple of lending errors
FIXES: APER-3510
* feat: limiting PII in logs per code review
per code review
FIXES: APER-3510
* feat: code review feedback
* removing some more PII from logs, even where it was not requested
* circuit breaker returned from grading attempt if course key is bad
* add missing tests for that functionality
FIXES: APER-3510
* feat: improved logging
per code review, adding a log message explaining certificate mode
issues
FIXES: APER-3510
* feat: fixing a bad string
lint error, and left off the format string modifier
FIXES: APER-3510
* feat: style
linter error
FIXES: APER-3510
[APER-3229]
In a previous PR, I created a new utility function named `is_credentials_enabled()` that can be used to determine if use of the Credentials IDA is enabled by config in an Open edX instance.
This PR is some additional cleanup that replaces the direct import and use of the `CredentialsApiConfig` model with the new utility function.
I took some additional time to update some existing log messages to include more info while reducing our need to log PII. I've removed as much use of a learner's username as possible, replacing it with logging the learner's LMS User Id instead.
[APER-3229]
The monolith and the Credentials IDA keep independent records of a course runs certificate availability/visibility preferences. This PR aims to improve the communication between the monolith and the Credentiala IDA to keep the availability date/preference in sync with the monoliths records.
The current code is too strict and actually prevents valid updates in some configurations.
Additionally, the Credentials IDA doesn't understand the concept of "course pacing" (instructor-paced vs self-paced) and has troubles with courses with an availability date of "end". Instead of having to add the concept of course pacing (and syncing more data between the two systems), this PR proposes sending the end date of a course as the "certificate available date" to Credentials.
This way, the Credentials IDA can manage the visibility of awarded credentials in a course run with a display behavior of "end" using the existing feature set and models of the Credentials service.
* fixed one test to accommodate a slightly modified workflow
* allowed the automated linter to match current standards
Note: as part of this process, I realized a lot of the tests in `programs/tests/test_tasks.py` are somewhat problematic. Some aren't real unit tests (allow calls to something other than the system under test), at least one for that reason it won't run on local at all, and some mock the wrong part of the system or just don't match current flow. I'm not modifying them as part of this, but they should be looked at.
FIXES: APER-3146
To determine whether or not we need to revoke any program certificates, we first run get_revokable_program_uuids. This calls get_certified_programs, which calls get_credentials, which uses the OpenEdx Core utility method get_api_data. get_api_data makes the API call inside a try block, does raise_for_status also inside the try block, and then catches the exception, logs it, and returns an empty result to the caller.
This means that on a failure to call the credentials API, get_credentials can’t tell the difference between a failure to hit the API (because credentials is, as it sometimes is during a notify_programs run, overloaded), or a learner with no program certificates. In this particular case, this is absolute failure, incorrect behavior.
* Adds a new flag, `raise_on_error` which will make `get_api_data` log the exception and then re-raise the HTTPError, defaulting to false in order to avoid changing the behavior on any other callers
* Also: my editor reformatted all of the touched files to our modern code standards, and it seemed appropriate to let it do that.
* Also: added type hints in some cases, because they helped me write the code and debug. Our test suite definitely reports mypy results on type errors so we are verifying that hints are correct.
FIXES: APER-3146
From the warning: Django now detects this configuration automatically.
You can remove default_app_config.
This also removes two warnings from shell startup.
[APER-1941]
We are aware of a product issue that causes a `certificate_available_date` (CAD) to be set for self-paced courses (and thus copied to the course-run's (course) certificate configuration) that causes an issue with learners' Program Records to be inaccurate. The stored CAD in Credentials is causing these certificates to be marked as "unearned" on the Program Record in Credentials, as the IDA believes the learner should *not* have access to them yet (but these certificates are available in the LMS).
A management command was recently introduced in Studio that can be used to clean/remove the `certificate_available_date` data from a course-run in Mongo. These updates aren't making it to the Credentials IDA because of an issue with our logic in the `update_certificate_visible_date_on_course_update` Celery task. This task assumes that we only want to send updates for *Instructor-Paced* courses that have a Certificate Display Behavior set to `end_with_date`. In reality, we need updates to pass to Credentials for _some_ self-paced courses with bad data.
This PR hopes to update our infrastructure to allow these updates to flow to Credentials.
* Improve logging for failed requests to the Credentials IDA's `course_certificates` endpoint when updating a course certificate configuration.
* Update docstrings and comments where appropriate
* Split the logic of the update_certificate_visible_date_on_course_update task into two tasks. The former task will continue to _just_ handle visible_date attribute updates. The latter (new) task will be dedicated to making the REST API call that updates the `certificate_available_date` data in Credentials.
* Update the `handle_course_cert_date_change` function wqhen the COURSE_CERT_DATE_CHANGE signal is received to queue both the "visible_date" and "certificate available date" Celery tasks.
* Update existing tests for the task changes.
**Previously**
When a course administrator changed the `certificates_display_behavior` (presumably to `end_with_date`) AND set the `certificate_available_date` in Studio, the `certificate_available_date` was not syncing to Credentials.
This was because we chose to send the `certificate_available_date` only if the course is self-paced and the `certificate_display_behavior` is set to `end_with_date`. [See PR #28275](https://github.com/openedx/edx-platform/pull/28275). However, we were checking those two conditions by looking at the relevant `CourseOverview`, which was not yet truly saved to reflect the updated display behavior at the time of the check due to atomic requests. [Read more about atomic requests and transactions here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#tying-transactions-to-http-requests-1); we have `ATOMIC_REQUESTS` set to `TRUE` in our codebase. Because the `certificate_display_behavior` was not (yet) `end_with_date`, the post to Credentials was not being fired.
**Solution**
To fix, this commit sends the `COURSE_CERT_DATE_CHANGE` signal `on_commit` instead, which waits until the transaction has completed and the update to the `CourseOverview` has been truly applied to the database. [Read more about `on_commit` here](https://docs.djangoproject.com/en/4.0/topics/db/transactions/#django.db.transaction.on_commit). Now, when the relevant `CourseOverview` is read, it will have the updated `certificate_display_behavior`.
See the [Django docs for how to test on_commit callbacks here](https://docs.djangoproject.com/en/3.2/topics/testing/tools/#django.test.TestCase.captureOnCommitCallbacks); this seems to be our first time using the built-in method.
This commit also cleans up some previous code that was meant to get around the problem caused by atomic requests, that is now unneccessary with this fix. It essentially reverses the work done in [PR #26991](https://github.com/openedx/edx-platform/pull/26991): we no longer need to explicitly pass the `certificate_available_date` since we can trust the `CourseOverview` to be properly updated.
**Rejected Solutions**
A. Simply publish the `COURSE_CERT_DATE_CHANGE` signal `on_commit`; no other changes. Rejected because: This would fix the problem, but leaves a lot of unnecessary code and some puzzling inconsistencies. I prefer the solution above because we are cleaning up behind ourselves.
B. Pass the new `certificate_display_behavior` along with the `certificate_available_date`; read those direclty instead of checking the (not-yet-properly-updated) `CourseOverview`. Rejected because: The pattern of passing the new `certificate_available_date` down through all these methods was put in place to get around the atomic requests problem. I believe `on_commit` to be a better solution to getting around that problem. I’d like to move away from passing data down through several functions / methods.
C. Start the celery task `on_commit` (rather than send the signal `on_commit`). Rejected because: The signal receiver basically only starts the celery task, and I find the break to be a bit more readable when sending the signal. No need to split hairs here.
D. Remove the check for pacing and display behavior; send the updated `certificate_available_date` every time there is a change, no matter what the current display behavior is. Rejected because: We intentionally added this check in [PR #28275](https://github.com/openedx/edx-platform/pull/28275) because the task was not behaving as expected without it (specifically around self-paced courses). I assume this is still necessary.
**Relevant Prior Work**
The following PRs--in order--show how this section (and other relevant sections) of the code have been changed over time:
1. [Move cert date signals to avoid race conditions #26841](https://github.com/openedx/edx-platform/pull/26841)
2. [feat: Pass date in cert date update signal #26991](https://github.com/openedx/edx-platform/pull/26991)
3. [Fix certificate available date sync #28275](https://github.com/openedx/edx-platform/pull/28275)
4. [fix: Correct an issue where cert available date was not sent to Crede… #28524](https://github.com/openedx/edx-platform/pull/28524)
MICROBA-1818
This is a first stage for removing the LegacyWaffle* classes.
LegacyWaffleFlag usage replaced with WaffleFlag;
LegacyWaffleSwitche usage replaced with WaffleSwitch;
New CourseWaffleFlag added to the temporary module __future__ as FutureCourseWaffleFlag;
Updated all the imports to use CourseWaffleFlag from the __future__ module;
BREAKING CHANGE: A number of toggle related constants (e.g. ENABLE_ACCESSIBILITY_POLICY_PAGE)
changed types. They were strings, and are now toggle instances (e.g. WaffleSwitch). Although the entire
refactor should be self-contained in edx-platform, if any plugins or dependencies were directly
using these constants, they will break. If this is the case, try to find a better publicized way of
exposing those toggles.
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences
In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.
For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
