The V2 libraries project had a few past iterations which were never
launched. This commit cleans up pieces from those which we don't need
for the real Libraries Relaunch MVP in Sumac:
* Remove ENABLE_LIBRARY_AUTHORING_MICROFRONTEND,
LIBRARY_AUTHORING_FRONTEND_URL, and
REDIRECT_TO_LIBRARY_AUTHORING_MICROFRONTEND, all of which are obsolete
now that library authoring has been merged into
https://github.com/openedx/frontend-app-authoring.
More details on the new Content Libraries configuration settings are
here: https://github.com/openedx/frontend-app-authoring/issues/1334
* Remove dangling support for syncing V2 (learning core-backed) library
content using the LibraryContentBlock. This code was all based on an
older understanding of V2 Content Libraries, where the libraries were
smaller and versioned as a whole rather then versioned by-item.
Reference to V2 libraries will be done on a per-block basis using
the upstream/downstream system, described here:
https://github.com/openedx/edx-platform/blob/master/docs/decisions/0020-upstream-downstream.rst
It's important that we remove this support now so that OLX course
authors don't stuble upon it and use it, which would be buggy and
complicate future migrations.
* Remove the "mode" parameter from LibraryContentBlock. The only
supported mode was and is "random". We will not be adding any further
modes. Going forward for V2, we will have an ItemBank block for
randomizing items (regardless of source), which can be synthesized
with upstream referenced as described above. Existing
LibraryContentBlocks will be migrated.
* Finally, some renamings:
* LibraryContentBlock -> LegacyLibraryContentBlock
* LibraryToolsService -> LegacyLibraryToolsService
* LibrarySummary -> LegacyLibrarySummary
Module names and the old OLX tag (library_content) are unchanged.
Closes: https://github.com/openedx/frontend-app-authoring/issues/1115
* feat: Be able to login to bare-metal studio easily.
Updating the documentation and the devstack.py files so that if you're
running bare-metal you can easily setup studio login via the LMS.
I also added the Ports that the various MFEs expect to the runserver
scripts so that it's easier to run those locally as well.
Co-authored-by: Kyle McCormick <kyle@axim.org>
Blockstore and all of its (experimental) functionality has been replaced with
openedx-learning, aka "Learning Core". This commit uninstalls the now-unused
openedx-blockstore package and removes all dangling references to it.
Note: This also removes the `copy_library_from_v1_to_v2` management command,
which has been broken ever since we switched from Blockstore to Learning Core.
Part of this DEPR: https://github.com/openedx/public-engineering/issues/238
The LMS and Studio need to set values for CORS_ALLOW_HEADERS so that the
MFEs can work properly, since preflight requests will need to send over
extra headers. Prior to this commit, CORS_ALLOW_HEADERS was being
redefined in multiple places in edx-platform and again in Tutor's config
because it was only being conditionally set if ENABLE_CORS_HEADERS was
True (which was a policy setting). But CORS_ALLOW_HEADERS is application
logic in that the value is determined by what the view needs, and won't
vary by deployment.
By consolidating this to always be defined in the common.py files, we
make sure that deployment environments don't have to define it. An
example of where this bit us was when course import in the course
authoring MFE did not work because Tutor was using an outdated value for
this setting.
A followup to this would be to just rip out the ENABLE_CORS_HEADERS
setting entirely, and just always have it on. But that would benefit
from a little more discovery to make sure there's no weird use case that
still requires it to be False (maybe something in the test suite?).
I was trying to test a communication pathway between the CMS and Credentials and requests were failing with the following error:
```
2024-02-07 19:25:46,524 ERROR 202 [celery.app.trace] [user 3] [ip 192.168.16.1] trace.py:270 - Task openedx.core.djangoapps.programs.tasks.update_credentials_course_certificate_configuration_available_date[898eae19-cd38-4857-a656-dc080d64a3c5] raised unexpected: ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=18150): Max retries exceeded with url: /api/v2/course_certificates/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc5b2c357c0>: Failed to establish a new connection: [Errno 111] Connection refused'))"))
Traceback (most recent call last):
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
```
I have updated the default value of the `CREDENTIALS_INTERNAL_SERVICE_URL` in our devstack settings to use the internal server URL and this fixed the issue.
BREAKING CHANGE: Removes all code, tests, and settings related to
indexing of V2 (blockstore-backed) content libraries in elasticsearch.
This includes indexing of top-level library metadata as well as indexing
of library block metadata. Operators who enabled the experimental
Library Authoring MFE *and* the experimental ENABLE_CONTENT_LIBRARY_INDEX
feature may notice that sorting, filtering, and searching of V2
libraries and their blocks may now be slower and/or less powerful.
The ENABLE_CONTENT_LIBRARY_INDEX feature was already disabled by
default, so most/all operators (including edx.org) should not notice
any difference.
Removed settings include:
* FEATURES['ENABLE_CONTENT_LIBRARY_INDEX']
* ENABLE_ELASTICSEARCH_FOR_TESTS
* TEST_ELASTICSEARCH_USE_SSL
* TEST_ELASTICSEARCH_HOST
* TEST_ELASTICSEARCH_PORT
For rationale, see the updated "Status" section of:
./openedx/core/djangoapps/content_libraries/docs/decisions/0001-index-libraries-in-elasticsearch.rst
* Update location in devstack-experimental to fix issue while re-indexing
* Updated the devstack.py to point to edx.devstack.elasticsearch710 host
* Enabed the ENABLE_COURSEWARE_INDEX and ENABLE_COURSEWARE_SEARCH
* feat: adding unenrollments to event bus
* fix: quality fixes
* fix: tweaks to pass tests
* fix: more tweaks for testing
---------
Co-authored-by: John Nagro <jnagro@edx.org>
Clearing the RequestCache was intended to address memory leaks in the
celery workers. Celery worker processes will process many tasks before
they are terminated. RequestCache cleanup typically happens in the
RequestCacheMiddleware class, and middleware never executes for celery.
To get around that issue, the CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION
setting was created to clear the RequestCache after every task was
successfully completed.
This works fine when celery is running as a separate process, as it's
set up to do in production. But during development, the
CELERY_ALWAYS_EAGER setting variable is set to True, meaning that
celery tasks are run in the same thread as the Django Request. This is
meant to make debugging easier, as task failures run as part of the
request cycle and will raise exceptions that are visible to the
browser.
However, celery tasks are triggered from many different actions. That
means that the RequestCache was being cleared many times during the
course of processing a request. This led to behavior that was
potentially slower, but also incorrect–the RequestCache was getting
flushed in a way that wouldn't happen in any deployed environment
because celery would be running in separate processes there. This came
up when trying to fix an issue around extra history records being
created during problem submissions:
https://discuss.openedx.org/t/extra-history-record-stored-on-each-problem-submission/8081
Furthermore, it's not necessary to prevent RequestCache memory leaks
when running in CELERY_AWLAYS_EAGER mode in development because the
middleware cleanup happens automatically–as everything is running as
part of the request/response cycle.
There are times in which we may want to run celery eagerly and still
clear the cache, such as testing. I have set
CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION = False in all dev and test
environments that already have CELERY_ALWAYS_EAGER = True. The unit
test that specifically tests whether the request cache is getting
cleared upon completion of a celery task then overrides
CLEAR_REQUEST_CACHE_ON_TASK_COMPLETION = True even though
CELERY_ALWAYS_EAGER = True for the sake of that specific testing
purpose.
Implements https://github.com/openedx/edx-platform/issues/30682
Produce signal only once transaction for a course publish is
committed, and only for actual courses (not libraries).
- Use newer openedx-events version that has a fix for None datetime
and that has CourseCatalogData without org, number.
- Add edx-event-bus-kafka -- specify recent version that drops
confluent-kafka from explicit deps, fixes common auth settings, and has
a multi-producer caching tweak.
- New functionality is behind toggle
As per https://github.com/openedx/openedx-events/issues/88 we're going to
try explicit dependencies on implementations for now, rather than solve
all the problems we'd encounter by using private dependencies.
Co-authored-by: Tim McCormack <tmccormack@edx.org>
Co-authored-by: Rebecca Graber <rgraber@edx.org>
In the `cms/envs/common.py` file the `CREDENTIALS_INTERNAL_SERVICE_URL` and `CREDENTIALS_PUBLIC_SERVICE_URL` settings are set to
`http://localhost:8005` which is incorrect for devstack. In devstack the Credentials IDA runs on port `18150`. This causes
issues with Studio being able to communicate with the Credentials IDA.
I've overriden the service URL settings in devstack.py to point to the correct port.
[MICROBA-1806]
We are aware of product issue where it is possible for a self-paced course-run to get have a `certificate_availability_date` created in the course settings. This can have an adverse effect on the Credentials IDA where a learner's Program Record does not correctly display the course certificates they have earned because of this data. This not only causes confusion for our learners, as it appears that a course certificate a learner can access and share in the LMS is displayed as unearned in the Credential's program record, but this can also cause issues when a learner attempts to share their program record through a credit pathway and the program record would not accurately reflect their program completion.
Unfortunately, the settings that manage the certificate availability date are hidden for self-paced courses in Studio (as they should only be used in instructor-paced courses).
For this reason, we are introducing a management command that will remove a certificate available date for a specified (self-paced) course-run. This will allow us to fix issues for individual learners while we work on a longer-term fix for the larger issue.
* Add new `clean_stale_certificate_available_dates` management command
* Add new `CleanStaleCertificateAvailabilityDates` Configuration Model
* Add tests for the new management command
* (Unrelated cleanup) Fix potential issue with private.py settings in the CMS being overwritten in devstack.py for developers using devstack.
* refactor: enterprise dependencies for EdxRestAPIClient replacement
This is a part of https://github.com/openedx/public-engineering/issues/42
- add settings for enterprise-backend-service DOT application
- update utils used by enterprise to get rid of EdxRestAPIClient
- original utils stays in the code (to keep edx-platform api
clients working) till the
https://github.com/openedx/public-engineering/issues/39 deprecation
work will be done
* fix: fix typo in the docstring
Previously, our rate-limiting code trusted the entire `X-Forwarded-For`
header, allowing a malicious client to spoof that header and evade
rate-limiting. This commit introduces a new module and setting
allowing us to make a more conservative choice of IPs.
- Create new `openedx.core.djangoapps.util.ip` module for producing
the IP "external chain" for requests based on the XFF header and the
REMOTE_ADDR.
- Include a function that gives the safest choice of IPs.
- Add new setting `CLOSEST_CLIENT_IP_FROM_HEADERS` for configuring how
the external chain is derived (i.e. setting the trust
boundary). Currently has a default, but we may want to make it
mandatory in the future.
- Change `django-ratelimit` code to use the proximate IP in the external
chain -- the one just outside the trust boundary.
Also:
- Change `XForwardedForMiddleware` to use more conservative choice for
its `REMOTE_ADDR` override
- Other adjustments to `XForwardedForMiddleware` as needed in order to
initialize new module and support code that needs the real
`REMOTE_ADDR` value
- Metrics for observability into the change (and XFF composition)
- Feature switch to restore legacy mode if needed
This also gives us a path forward to removing use of the django-ipware
package, which is no longer maintained and has a handful of bugs that make it
difficult to use safely.
Internal ticket: ARCHBOM-2056
Introduce a new CMS settings COURSEGRAPH_CONNECTION,
which allows operators to specify default connection paramters
for a Neo4j instance.
This has three purposes:
* The `./manage.py cms dump_to_neo4j` management command will be
much easier for developers and operators to type out because connection
arguments can now be omitted. Note that connection arguments, if
supplied, will override the arguments specified in CMS settings.
* The automatic push-to-coursegraph-on-publish-signal introduced in
subsequent commits can use these connection settings.
* The CourseGraph Django admin actions introduced in subsequent
commits can use these connection settings.
* feat: Add support for using the discussions MFE UI instead of existing UI
Adds a new course waffle flag that when set along with the discussions MFE URL shows the discussions MFE UI instead of the regular UI.
* test: add tests
* squash!: more consistent url name
- Update migration instructions
- Changes regarding redirect URLs and cookie domain are to permit the
site to run on multiple domains.
- Set LOGIN_URL in common so that it can be unset in environment overrides
This bypasses the "redirect to LMS" login/signup code, but does not yet
remove it; removal is covered by DEPR-166 so that this remains a
configuration-only change for now.
There should have no user-visible effect.
ref: ARCHBOM-1890
This changes the "Sign out" link on Studio to point to Studio's own logout
view, which clears the session and then redirects to LMS's logout page. The
LMS logout page then skips loading the Studio logout because it is seen in
the Referer header.
This change also brings Studio better into line with how other IDAs perform
their logouts.
Background:
After the rollout of Studio OAuth, logouts initiated on Studio failed to
actually log out Studio (but all other IDAs were logged out). This was
because the LMS logout view loads the logout pages of other IDAs but skips
any that is a *prefix* match on the Referer header, and browsers now often
send a truncated version of the Referer for privacy. Therefore, Studio was
always skipped when coming from Studio.
The fix is to make sure that Studio has already performed its logout by the
time the LMS logout page is loaded.
One wrinkle here is that the LMS logout view is activated by `/logout`, but
the correct logout view (provided by auth_backends) is activated by
`/logout/` -- with a trailing slash. This is fragile and unfortunate, but
can be cleaned up when we later remove other leftovers of Studio's previous
ability to handle logistration.
ref: ARCHBOM-1897
This also requires using different session cookie names in devstack for
LMS and Studio so that we can properly use the OAuth flow locally, rather
than just sharing a session cookie on the localhost domain. (Note that
ports are not used for cookie segmentation in browsers.)
This depends on the following provisioning PR:
https://github.com/edx/devstack/pull/825
Login redirect was using wrong url, thus making it impossible to login from studio. This settings change should fix that.
Useful information to include:
Which edX user roles will this change impact? "Developer"
Supporting information
https://openedx.atlassian.net/browse/ARCHBOM-1687
Testing instructions
Bring up devstack studio and press sign in. If it redirects you correctly to lms login page, this works!
Deprecated edx-platform import paths
(for example, `student` instead of
`common.djangoapps.student`) currently raise
warnings when used. We want to fully remove
support for those paths.
As an easily reversible way to initially remove
support, we add a new setting to LMS and Studio
called `ERROR_ON_DEPRECATED_EDX_PLATFORM_IMPORTS`,
defaulting to False. We set it to True for devstack
and will set it to True in Stage and Production
soon. If critical errors occur, we can easily
flip the setting back to False.
