See requirements/edx-sandbox/README.rst for more info
BREAKING CHANGE: edx-sandbox/py38.txt will not longer
be updated. Please install from either edx-sandbox/base.txt or
edx-sandbox/releases/*.txt instead.
We reimplement `make translations` so that it does not depend on the
presence of the `.git` directory. This fixes the Tutor build error:
> [production 18/34] RUN make clean_translations:
git clean -fdX conf/locale
fatal: not a git repository (or any of the parent directories): .git
make: *** [Makefile:57: clean_translations] Error 128
Why is the .git directory missing during the Tutor translations build?
Because if it were present, it would tie the translations' layer cache
up with git state, which would lead to the Docker layer being rebuilt
any time there was any change to git state.
This pull request pulls translations via atlas for `studio-frontend` and refactor `load_sfe_i18n_messages` to load new translations into `conf/plugins-locale/studio-frontend` instead of relying on deprecated node.js package bundled translations.
Sometimes this make target doubles or even triples the comment. I think
this is due to a temporary failure in the wget call, which is then ignored
and an additional comment is added onto the top. If this happens multiple
times in a row, you get multiple additions.
Removing the `|| touch` should fix this, and surface any errors that are
happening.
* fix: include new assets.txt file in make upgrade
* test: update click version to resolve upgrade job failure
* chore: Updating Python Requirements (#32861)
To make edx-platform more consistent with other IDAs and to
help deprecate more of paver, we are adding the ability to
run `make migrate` in a local environment as a replacement
for `paver update_db`.
https://github.com/openedx/devstack/issues/1085
* Swagger was renamed to OpenAPI at some point so use the new name for
clarity.
* Prefix with `lms` to make it clear that these are the APIs from the
LMS and may not all be available in the CMS.
* This project was not being published anywhere.
* The previous commit that adds openapi generation capabilities to the
`docs/guides` replaces what this was doing.
In different environments echo seems to handle escape characters differently.
When generating the common constraints file, on some systems it outputs the
"\n" character as-is. This change switches echo to printf, which is supposed to
be more consistent.
This adds a Make target that should simplify the common task of
upgrading a single dependency. Sometimes people manually edit the pin
files, which we would like to avoid; hopefully this will make it
easier for them to do the right thing.
The GitHub workflow should also make it easier for people on Mac to
recompile requirements in a Linux environment, reducing the number of
times spurious dependency changes show up in the pin files (due to
OS-dependent requirements.)
Also, separate upgrade/downgrade instructions and simplify the latter.
(Min constraints are rare and we usually move beyond them quickly.)
There was a `requirements/pip.txt` with old versions, and a newer
`requirements/edx/pip.txt` managed via a `pip.in` file. The old one was
used in most places, but came out of sync with pip-tools.txt, which was
managed properly. Eventually this caused a `pip check` failure due to the
mismatch.
This should resolve at least part of https://github.com/edx/edx-arch-experiments/issues/267
This PR moves pip.in and pip-tools.in and their corresponding pin files
up to the `requirements/` dir, since they should be shared between the edx
and sandbox environments. This also has the effect of upgrading pip to
match the version in the file we've been uselessly upgrading.
Other improvements:
- Remove `-q` option from pip and pip-sync calls, as it was hiding some
debugging information that would have resolved this sooner.
- Depend on `pre-requirements` from `compile-requirements`, rather than
from `upgrade`. (The base target is the one that actually needs it.)
This also lets us remove the explicit `pip install pip-tools` line.
- Install the recompiled pip and pip-tools files right away, not after the
loop. When we upgrade pip-tools, we want to use the upgraded version,
not the previous version. This requires moving the pip-tools.txt
recompilation outside of the loop and into its own explicit line.
- Don't upgrade pip if we're not running `make upgrade` (respect the
compile options).
- Remove apparently-unneeded `--no-emit-trusted-host --no-emit-index-url`
options (we don't pass trusted-host or index-url options).
We have a need to lock the version of Django for production and tests, but
also to test on newer versions of Django so that we can get the repo ready
for long-term-support releases.
We've been doing that by extracting the `django==x.y.z` from the
pip-compiled files and moving it to a django.txt that is then co-installed
but can be overridden during tests. The problem is that this can result
in broken packages.
The approach here is to have `make test-requirements` continue to
ensure a consistent set of packages, and then install a different
Django on top of that in the CI script -- and call `pip check` to make
sure that combination isn't broken.
Adding Django 4.0 to the unit-tests.yml matrix will now correctly
result in this error and a failing job:
`django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.`
The other half of this is to change other CI runners to remove their
ability to control the Django version, since it's complicated to make
this work, and we probably only need it in unit-tests.yml. Convert them
to just use `make test-requirements`.
Also:
- Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our
own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option
that it knows). This is needed to allow `make test-requirements` to do
the pip calls. An alternative would be to set a pip-options env var for
the make target to use, but `PIP_SRC` already exists.
- Remove outdated modifications to common_constraints
- Add comment explaining why pylint tests need dev-requirements
I found this useful because then I can run some basic dev commands
locally instead of needing a full container environment just to run
pylint or other basic dev tasks.
Co-authored-by: Kyle McCormick <kdmc@pm.me>
We've been seeing some cross-version caching issues in the
edx-platform-runner unit tests, which apparently run on a "dirty"
environment—the virtualenv does not get cleared out between runs,
which probably improves performance but allows installed dependencies
to "leak" between runs. This results in errors between master and older
open-releases but could also prevent us from noticing missing deps.
By using pip-sync in the new CI Make targets (as we already do for
the regular `make requirements` target) we can ensure that any stale
dependencies from runs by other branches (or older versions of the
code) are removed.
Calling `make local-requirements` at the end of each `*-requirements`
target rather than making it a prerequisite is necessary for using sync,
since otherwise the local reqs would be wiped out.
The `requirements` target is also deduplicated into the newer
`dev-requirements` (aliased to it, with both installing private deps now.)
Adding a prerequisite of `pre-requirements` allows us to simplify some
workflow calls slightly. This ends up being the bulk of the commit by
line count. The pip lockfile also wasn't being used in the Makefile, so
I added that to pre-requirements as well.
Also fix leading whitespace issue in Makefile.
This commits prepares edx-platform's experimental Dockerfile
for optional use in devstack. Presently, the image built by this
Dockerfile isn't used anywhere.
Notable changes:
* Drop the openedx/edx-platform image name in favor of
openedx/lms and openedx/cms.
* Drop the newrelic stages and tags.
* Create openedx/lms-dev and openedx/cms-dev image
variants which use Django devserver, install dev
requirements, and specify devstack Django settings.
* Add config files at (lms,cms)/envs/devstack-experimental.yml,
extracted from the existing edxapp docker image.
* Adds three new scripts, each of which replaces an Ansible
or Paver-supported function with a pure bash + Django
management command implementation.
These comments are meant for the Makefile maintainers to read, not the
user running make. We can prefix them with @ to keep make from echoing
them while running.
