We have a need to lock the version of Django for production and tests, but
also to test on newer versions of Django so that we can get the repo ready
for long-term-support releases.
We've been doing that by extracting the `django==x.y.z` from the
pip-compiled files and moving it to a django.txt that is then co-installed
but can be overridden during tests. The problem is that this can result
in broken packages.
The approach here is to have `make test-requirements` continue to
ensure a consistent set of packages, and then install a different
Django on top of that in the CI script -- and call `pip check` to make
sure that combination isn't broken.
Adding Django 4.0 to the unit-tests.yml matrix will now correctly
result in this error and a failing job:
`django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.`
The other half of this is to change other CI runners to remove their
ability to control the Django version, since it's complicated to make
this work, and we probably only need it in unit-tests.yml. Convert them
to just use `make test-requirements`.
Also:
- Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our
own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option
that it knows). This is needed to allow `make test-requirements` to do
the pip calls. An alternative would be to set a pip-options env var for
the make target to use, but `PIP_SRC` already exists.
- Remove outdated modifications to common_constraints
- Add comment explaining why pylint tests need dev-requirements
Packages can be added to package-lock.json that will fail to
install on certain systems. For example, the `fsevents` NPM
package, which only works on macOS, was listed as a requirement
in the file.
`npm install` will happily skip over such packages.
`npm clean-install`, however, will exit with a fatal error.
Throughout several doc pages, we have recently been encouraging
folks to use `clean-install` instead of `install`, because its
strictness makes it more reproducible. To ensure that `clean-install`
is working reliably at any given time, we should use `clean-install`
in our CI pipeline.
The requirements/edx/private.txt file is for dev's own private package
needs. There are two installation mechanisms in edx-platform, and
neither handled the file properly:
- `paver install_prereqs` had the wrong file name. The file was moved
almost three years ago, and paver wasn't kept up.
- `make requirements` used `private.*` which included private.in, which
pip-sync balks at.
Part of the notifier service deprecation (DEPR-106).
Also removed pdfminer from the package uninstall list, since we no longer install the package it conflicts with either.
enum34 is causing problems in python3.8 tests of edx-platform because it has incompatibility with recent versions of python, the error the error happens while importing the re module in these python versions.
This library is a backport of enum of python3.4 made for python<3.4, therefore is not needed anymore.
edx-val was updated to remove that dependency: https://github.com/edx/edx-val/pull/245
'npm install' command execution was changed from paver.easy.sh()
to subprocess.Popen() in commit 0dd13fad1b,
but exception handling was not updated accordingly. Due to this,
errors in 'npm install' execution were not detected.
Unpin several more outdated dependencies whose changelogs don't contain any significant backwards incompatible changes. Also add "moto" to the list of packages to uninstall from existing environments, since it requires a jsondiff version that clashes with the one we now use (triggering a harmless but distracting warning on dependency updates).
We can potentially stop using path.py/path altogether by switching to pathlib in the Python 3 standard library, but that merits a separate PR of its own.
Also, note that I'm not actually unpinning freezegun; different PRs restricted it in both constraints.txt and test.in, I'm just removing the latter redundant constraint.
Fix the issue that was preventing us from upgrading pytest. pytest does some manipulation of test packages that prevents `pkg_resources` from loading resources from them, but used to contain a workaround for the problem. That workaround was [removed](https://github.com/pytest-dev/pytest/issues/5392) in 4.6.0 as a performance enhancement when pytest switched from `pkg_resources` to `importlib-metadata` for its own entrypoint handling. This tripped up one of our test modules which defined classes that loaded templates from inside a test package. Moving these resources to the parent package fixes the problem.
More and more, `pkg_resources` is being abandoned in favor of `importlib-metadata` and `importlib_resources` as they have a simpler design with much better performance. However, `importlib_resources` doesn't support loading files from any directory which isn't itself a Python package (and doesn't allow direct use of paths including directories within the package). Jinja2 chose a [different approach](https://github.com/pallets/jinja/pull/1082) that we may want to emulate in our resource handling.
Also fixed usage of a removed `pytest.raises()` parameter and a bug in our configuration of the `common/lib` tests that became a problem after the upgrade.
to accelerate installation, people usually set npm registry to a faster mirror by running 'npm config set registry BLAH_BLAH'
and even if the customized config not available, by default it is 'NPM_REGISTRY = "https://registry.npmjs.org/"', so delete these lines always works.
The NO_PYTHON_UNINSTALL environment variable can now be used to disable the task that removes older versions of Python libraries. This task is not necessary for Docker images/containers since we always create a fresh virtualenv when we create the image.
