There was a `requirements/pip.txt` with old versions, and a newer
`requirements/edx/pip.txt` managed via a `pip.in` file. The old one was
used in most places, but came out of sync with pip-tools.txt, which was
managed properly. Eventually this caused a `pip check` failure due to the
mismatch.
This should resolve at least part of https://github.com/edx/edx-arch-experiments/issues/267
This PR moves pip.in and pip-tools.in and their corresponding pin files
up to the `requirements/` dir, since they should be shared between the edx
and sandbox environments. This also has the effect of upgrading pip to
match the version in the file we've been uselessly upgrading.
Other improvements:
- Remove `-q` option from pip and pip-sync calls, as it was hiding some
debugging information that would have resolved this sooner.
- Depend on `pre-requirements` from `compile-requirements`, rather than
from `upgrade`. (The base target is the one that actually needs it.)
This also lets us remove the explicit `pip install pip-tools` line.
- Install the recompiled pip and pip-tools files right away, not after the
loop. When we upgrade pip-tools, we want to use the upgraded version,
not the previous version. This requires moving the pip-tools.txt
recompilation outside of the loop and into its own explicit line.
- Don't upgrade pip if we're not running `make upgrade` (respect the
compile options).
- Remove apparently-unneeded `--no-emit-trusted-host --no-emit-index-url`
options (we don't pass trusted-host or index-url options).
We have a need to lock the version of Django for production and tests, but
also to test on newer versions of Django so that we can get the repo ready
for long-term-support releases.
We've been doing that by extracting the `django==x.y.z` from the
pip-compiled files and moving it to a django.txt that is then co-installed
but can be overridden during tests. The problem is that this can result
in broken packages.
The approach here is to have `make test-requirements` continue to
ensure a consistent set of packages, and then install a different
Django on top of that in the CI script -- and call `pip check` to make
sure that combination isn't broken.
Adding Django 4.0 to the unit-tests.yml matrix will now correctly
result in this error and a failing job:
`django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.`
The other half of this is to change other CI runners to remove their
ability to control the Django version, since it's complicated to make
this work, and we probably only need it in unit-tests.yml. Convert them
to just use `make test-requirements`.
Also:
- Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our
own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option
that it knows). This is needed to allow `make test-requirements` to do
the pip calls. An alternative would be to set a pip-options env var for
the make target to use, but `PIP_SRC` already exists.
- Remove outdated modifications to common_constraints
- Add comment explaining why pylint tests need dev-requirements
