We have a need to lock the version of Django for production and tests, but
also to test on newer versions of Django so that we can get the repo ready
for long-term-support releases.
We've been doing that by extracting the `django==x.y.z` from the
pip-compiled files and moving it to a django.txt that is then co-installed
but can be overridden during tests. The problem is that this can result
in broken packages.
The approach here is to have `make test-requirements` continue to
ensure a consistent set of packages, and then install a different
Django on top of that in the CI script -- and call `pip check` to make
sure that combination isn't broken.
Adding Django 4.0 to the unit-tests.yml matrix will now correctly
result in this error and a failing job:
`django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.`
The other half of this is to change other CI runners to remove their
ability to control the Django version, since it's complicated to make
this work, and we probably only need it in unit-tests.yml. Convert them
to just use `make test-requirements`.
Also:
- Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our
own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option
that it knows). This is needed to allow `make test-requirements` to do
the pip calls. An alternative would be to set a pip-options env var for
the make target to use, but `PIP_SRC` already exists.
- Remove outdated modifications to common_constraints
- Add comment explaining why pylint tests need dev-requirements
I found this useful because then I can run some basic dev commands
locally instead of needing a full container environment just to run
pylint or other basic dev tasks.
Co-authored-by: Kyle McCormick <kdmc@pm.me>
We've been seeing some cross-version caching issues in the
edx-platform-runner unit tests, which apparently run on a "dirty"
environment—the virtualenv does not get cleared out between runs,
which probably improves performance but allows installed dependencies
to "leak" between runs. This results in errors between master and older
open-releases but could also prevent us from noticing missing deps.
By using pip-sync in the new CI Make targets (as we already do for
the regular `make requirements` target) we can ensure that any stale
dependencies from runs by other branches (or older versions of the
code) are removed.
Calling `make local-requirements` at the end of each `*-requirements`
target rather than making it a prerequisite is necessary for using sync,
since otherwise the local reqs would be wiped out.
The `requirements` target is also deduplicated into the newer
`dev-requirements` (aliased to it, with both installing private deps now.)
Adding a prerequisite of `pre-requirements` allows us to simplify some
workflow calls slightly. This ends up being the bulk of the commit by
line count. The pip lockfile also wasn't being used in the Makefile, so
I added that to pre-requirements as well.
Also fix leading whitespace issue in Makefile.
This commits prepares edx-platform's experimental Dockerfile
for optional use in devstack. Presently, the image built by this
Dockerfile isn't used anywhere.
Notable changes:
* Drop the openedx/edx-platform image name in favor of
openedx/lms and openedx/cms.
* Drop the newrelic stages and tags.
* Create openedx/lms-dev and openedx/cms-dev image
variants which use Django devserver, install dev
requirements, and specify devstack Django settings.
* Add config files at (lms,cms)/envs/devstack-experimental.yml,
extracted from the existing edxapp docker image.
* Adds three new scripts, each of which replaces an Ansible
or Paver-supported function with a pure bash + Django
management command implementation.
These comments are meant for the Makefile maintainers to read, not the
user running make. We can prefix them with @ to keep make from echoing
them while running.
As we have upgraded pyjwt in edx-platform and also removed these pins from common constraint files in edx-l;int, so we don't need this anymore.
https://github.com/edx/edx-lint/pull/203
Pull Translations has been broken in edx-platform for a number of months
partially in part because of an import issue with local packages during
paver setup. This solves that problem by running make requirements as part
of the pull_translations make target.
Note: This should not be the default behavior. This is only a patch so the
translation job works again. The paver job being run (i18n_compilejs) does
the same install before it runs so we are doubling up there which is not
great.
PROD-2455
The edx-platform codebase already includes quite a few type annotations, but
they were not regularly checked. This may cause problems, when the annotations
themselves include errors (as we found out in some of the learning_sequences
annotations). So here, we add mypy as a dev requirement and introduce a make
command to run mypy regularly. Mypy runs on a very small portion of the total
edx-platform, as configured in mypy.ini. Our hope is that developers will add
more and more modules to this configuration file, until we can eventually run
mypy on the full code base.
See discussion: https://discuss.openedx.org/t/dev-notes-running-mypy-on-edx-platform/4860
We have decided not to pursue the decentralized devstack design right
now and so we want to cleanup files and task that were built as a part
of the process.
Related Ticket: ARCHBOM-1685
The requirements/edx/private.txt file is for dev's own private package
needs. There are two installation mechanisms in edx-platform, and
neither handled the file properly:
- `paver install_prereqs` had the wrong file name. The file was moved
almost three years ago, and paver wasn't kept up.
- `make requirements` used `private.*` which included private.in, which
pip-sync balks at.
I grew tired of modifying the Makefile to compile requirements without
upgrading them. Also, installing pip-tools should not be part of the
compile-requirements target, so a separate target was created.
When running "make pull_translations", this will print any
validation errors on the console, rather than stuffing them into
a .prob file.
This helps when looking at failing jenkins log files.
* Add and push Dockerfile; add decentralized devstack settings
Co-Authored-By: Diana Huang <dkh@edx.org>
Co-Authored-By: Kyle McCormick <kmccormick@edx.org>
* Remove Python requirements hack
Remove the attempted optimization to the installation of Python
package dependencies. The dependencies in edx-platform change
about three times per day, so this was of dubious value. And
because npm is run through nodeenv, which is a Python package,
the Python dependencies installation has to happen first.
* ARCHBOM-1439: Changing workdir to /edx/app/edxapp/edx-platform (#24835)
Context: The Dockerfile tries to stay in sych with legacy stuff.
In the ansible we configure the directory structure such that things
relating to the app but not in the codebase,
such as the env file wind up in /edx/app/edxapp/.
And the codebase winds up in /edx/app/edxapp/edx-platform.
I think due to accident, the dockerfile does
/edx/app/edx-platform/edx-platform instead of /edx/app/edxapp/edx-platform.
This commit tries to have Dockerfile more reflect what is currently happening in production
* Update ports for decentralized devstack ARCHBOM-1447 (#24841)
Switch from the LMS ports we've historically used for NGINX to those used for gunicorn, and fix the Studio ports to match the ones we've historically used for its gunicorn service. Also removed some leftover bits of the requirements hack.
Co-authored-by: Adam Blackwell <ablackwell@edx.org>
Co-authored-by: Diana Huang <dkh@edx.org>
Co-authored-by: jinder1s <msingh@edx.org>
Co-authored-by: Jeremy Bowman <jbowman@edx.org>
Co-authored-by: Manjinder Singh <49171515+jinder1s@users.noreply.github.com>
We introduce a new documentation target, where we use the featuretoggles
Sphinx extension from code-annotations to generate human-readable
documentation of feature toggles in edx-platform. The annotation report
is generated on-the-fly based on the standard feature toggle
configuration file in code-annotations.
In addition, we add new doc.in & doc.txt requirement files that will be
pip-installed by readthedocs to generate the documentation targets.
Cleaned up a few more things about the testing configuration now that we've deployed Django 2.2 to production:
* Go back to a single version of django-oauth-toolkit, tox doesn't need to vary it anymore
* Go back to PyPI release of django-method-override
* Remove no longer needed dependency on python-oauth2 (we already removed the package it was used by)
* Update prepare_xdist_nodes.sh with new Django requirements file options
* Revert "Use pip-sync to make sure that dep cache tarball can go safely stale"
This reverts commit d435f4cd3e.
* Revert "Extract worker setup into own shell script, as much as possible"
This reverts commit 0a079e757c.
This patch adds compilejs
management command to be executed
on pull translations enabling
availability of js/jsx translations
for lms/cms code.
PROD-960
Django-auth-toolkit update
Pinned the latest commit from github to get Django2.2 support in testing.
Pinned the zipp<2.0.0 to get Python<3.5 support too.
The API documentation decorators do not have to leak which solution we
use to generate the docs. Here, and as discussed in PR #21820, we rename
the `openapi` module to `apidocs`, and we make sure that this module
includes all the right functions to document API Views without referring
to Open API.
The xblock counting script had a requirements file that was not
previously being tracked and had an out of date version of requests.
Add it to the `upgrade` make target so that it stays up to date.
