Sometimes this make target doubles or even triples the comment. I think
this is due to a temporary failure in the wget call, which is then ignored
and an additional comment is added onto the top. If this happens multiple
times in a row, you get multiple additions.
Removing the `|| touch` should fix this, and surface any errors that are
happening.
* fix: include new assets.txt file in make upgrade
* test: update click version to resolve upgrade job failure
* chore: Updating Python Requirements (#32861)
To make edx-platform more consistent with other IDAs and to
help deprecate more of paver, we are adding the ability to
run `make migrate` in a local environment as a replacement
for `paver update_db`.
https://github.com/openedx/devstack/issues/1085
* Swagger was renamed to OpenAPI at some point so use the new name for
clarity.
* Prefix with `lms` to make it clear that these are the APIs from the
LMS and may not all be available in the CMS.
* This project was not being published anywhere.
* The previous commit that adds openapi generation capabilities to the
`docs/guides` replaces what this was doing.
In different environments echo seems to handle escape characters differently.
When generating the common constraints file, on some systems it outputs the
"\n" character as-is. This change switches echo to printf, which is supposed to
be more consistent.
This adds a Make target that should simplify the common task of
upgrading a single dependency. Sometimes people manually edit the pin
files, which we would like to avoid; hopefully this will make it
easier for them to do the right thing.
The GitHub workflow should also make it easier for people on Mac to
recompile requirements in a Linux environment, reducing the number of
times spurious dependency changes show up in the pin files (due to
OS-dependent requirements.)
Also, separate upgrade/downgrade instructions and simplify the latter.
(Min constraints are rare and we usually move beyond them quickly.)
There was a `requirements/pip.txt` with old versions, and a newer
`requirements/edx/pip.txt` managed via a `pip.in` file. The old one was
used in most places, but came out of sync with pip-tools.txt, which was
managed properly. Eventually this caused a `pip check` failure due to the
mismatch.
This should resolve at least part of https://github.com/edx/edx-arch-experiments/issues/267
This PR moves pip.in and pip-tools.in and their corresponding pin files
up to the `requirements/` dir, since they should be shared between the edx
and sandbox environments. This also has the effect of upgrading pip to
match the version in the file we've been uselessly upgrading.
Other improvements:
- Remove `-q` option from pip and pip-sync calls, as it was hiding some
debugging information that would have resolved this sooner.
- Depend on `pre-requirements` from `compile-requirements`, rather than
from `upgrade`. (The base target is the one that actually needs it.)
This also lets us remove the explicit `pip install pip-tools` line.
- Install the recompiled pip and pip-tools files right away, not after the
loop. When we upgrade pip-tools, we want to use the upgraded version,
not the previous version. This requires moving the pip-tools.txt
recompilation outside of the loop and into its own explicit line.
- Don't upgrade pip if we're not running `make upgrade` (respect the
compile options).
- Remove apparently-unneeded `--no-emit-trusted-host --no-emit-index-url`
options (we don't pass trusted-host or index-url options).
We have a need to lock the version of Django for production and tests, but
also to test on newer versions of Django so that we can get the repo ready
for long-term-support releases.
We've been doing that by extracting the `django==x.y.z` from the
pip-compiled files and moving it to a django.txt that is then co-installed
but can be overridden during tests. The problem is that this can result
in broken packages.
The approach here is to have `make test-requirements` continue to
ensure a consistent set of packages, and then install a different
Django on top of that in the CI script -- and call `pip check` to make
sure that combination isn't broken.
Adding Django 4.0 to the unit-tests.yml matrix will now correctly
result in this error and a failing job:
`django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.`
The other half of this is to change other CI runners to remove their
ability to control the Django version, since it's complicated to make
this work, and we probably only need it in unit-tests.yml. Convert them
to just use `make test-requirements`.
Also:
- Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our
own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option
that it knows). This is needed to allow `make test-requirements` to do
the pip calls. An alternative would be to set a pip-options env var for
the make target to use, but `PIP_SRC` already exists.
- Remove outdated modifications to common_constraints
- Add comment explaining why pylint tests need dev-requirements
I found this useful because then I can run some basic dev commands
locally instead of needing a full container environment just to run
pylint or other basic dev tasks.
Co-authored-by: Kyle McCormick <kdmc@pm.me>
We've been seeing some cross-version caching issues in the
edx-platform-runner unit tests, which apparently run on a "dirty"
environment—the virtualenv does not get cleared out between runs,
which probably improves performance but allows installed dependencies
to "leak" between runs. This results in errors between master and older
open-releases but could also prevent us from noticing missing deps.
By using pip-sync in the new CI Make targets (as we already do for
the regular `make requirements` target) we can ensure that any stale
dependencies from runs by other branches (or older versions of the
code) are removed.
Calling `make local-requirements` at the end of each `*-requirements`
target rather than making it a prerequisite is necessary for using sync,
since otherwise the local reqs would be wiped out.
The `requirements` target is also deduplicated into the newer
`dev-requirements` (aliased to it, with both installing private deps now.)
Adding a prerequisite of `pre-requirements` allows us to simplify some
workflow calls slightly. This ends up being the bulk of the commit by
line count. The pip lockfile also wasn't being used in the Makefile, so
I added that to pre-requirements as well.
Also fix leading whitespace issue in Makefile.
This commits prepares edx-platform's experimental Dockerfile
for optional use in devstack. Presently, the image built by this
Dockerfile isn't used anywhere.
Notable changes:
* Drop the openedx/edx-platform image name in favor of
openedx/lms and openedx/cms.
* Drop the newrelic stages and tags.
* Create openedx/lms-dev and openedx/cms-dev image
variants which use Django devserver, install dev
requirements, and specify devstack Django settings.
* Add config files at (lms,cms)/envs/devstack-experimental.yml,
extracted from the existing edxapp docker image.
* Adds three new scripts, each of which replaces an Ansible
or Paver-supported function with a pure bash + Django
management command implementation.
These comments are meant for the Makefile maintainers to read, not the
user running make. We can prefix them with @ to keep make from echoing
them while running.
As we have upgraded pyjwt in edx-platform and also removed these pins from common constraint files in edx-l;int, so we don't need this anymore.
https://github.com/edx/edx-lint/pull/203
Pull Translations has been broken in edx-platform for a number of months
partially in part because of an import issue with local packages during
paver setup. This solves that problem by running make requirements as part
of the pull_translations make target.
Note: This should not be the default behavior. This is only a patch so the
translation job works again. The paver job being run (i18n_compilejs) does
the same install before it runs so we are doubling up there which is not
great.
PROD-2455
The edx-platform codebase already includes quite a few type annotations, but
they were not regularly checked. This may cause problems, when the annotations
themselves include errors (as we found out in some of the learning_sequences
annotations). So here, we add mypy as a dev requirement and introduce a make
command to run mypy regularly. Mypy runs on a very small portion of the total
edx-platform, as configured in mypy.ini. Our hope is that developers will add
more and more modules to this configuration file, until we can eventually run
mypy on the full code base.
See discussion: https://discuss.openedx.org/t/dev-notes-running-mypy-on-edx-platform/4860
We have decided not to pursue the decentralized devstack design right
now and so we want to cleanup files and task that were built as a part
of the process.
Related Ticket: ARCHBOM-1685
The requirements/edx/private.txt file is for dev's own private package
needs. There are two installation mechanisms in edx-platform, and
neither handled the file properly:
- `paver install_prereqs` had the wrong file name. The file was moved
almost three years ago, and paver wasn't kept up.
- `make requirements` used `private.*` which included private.in, which
pip-sync balks at.
I grew tired of modifying the Makefile to compile requirements without
upgrading them. Also, installing pip-tools should not be part of the
compile-requirements target, so a separate target was created.
When running "make pull_translations", this will print any
validation errors on the console, rather than stuffing them into
a .prob file.
This helps when looking at failing jenkins log files.
* Add and push Dockerfile; add decentralized devstack settings
Co-Authored-By: Diana Huang <dkh@edx.org>
Co-Authored-By: Kyle McCormick <kmccormick@edx.org>
* Remove Python requirements hack
Remove the attempted optimization to the installation of Python
package dependencies. The dependencies in edx-platform change
about three times per day, so this was of dubious value. And
because npm is run through nodeenv, which is a Python package,
the Python dependencies installation has to happen first.
* ARCHBOM-1439: Changing workdir to /edx/app/edxapp/edx-platform (#24835)
Context: The Dockerfile tries to stay in sych with legacy stuff.
In the ansible we configure the directory structure such that things
relating to the app but not in the codebase,
such as the env file wind up in /edx/app/edxapp/.
And the codebase winds up in /edx/app/edxapp/edx-platform.
I think due to accident, the dockerfile does
/edx/app/edx-platform/edx-platform instead of /edx/app/edxapp/edx-platform.
This commit tries to have Dockerfile more reflect what is currently happening in production
* Update ports for decentralized devstack ARCHBOM-1447 (#24841)
Switch from the LMS ports we've historically used for NGINX to those used for gunicorn, and fix the Studio ports to match the ones we've historically used for its gunicorn service. Also removed some leftover bits of the requirements hack.
Co-authored-by: Adam Blackwell <ablackwell@edx.org>
Co-authored-by: Diana Huang <dkh@edx.org>
Co-authored-by: jinder1s <msingh@edx.org>
Co-authored-by: Jeremy Bowman <jbowman@edx.org>
Co-authored-by: Manjinder Singh <49171515+jinder1s@users.noreply.github.com>
We introduce a new documentation target, where we use the featuretoggles
Sphinx extension from code-annotations to generate human-readable
documentation of feature toggles in edx-platform. The annotation report
is generated on-the-fly based on the standard feature toggle
configuration file in code-annotations.
In addition, we add new doc.in & doc.txt requirement files that will be
pip-installed by readthedocs to generate the documentation targets.
Cleaned up a few more things about the testing configuration now that we've deployed Django 2.2 to production:
* Go back to a single version of django-oauth-toolkit, tox doesn't need to vary it anymore
* Go back to PyPI release of django-method-override
* Remove no longer needed dependency on python-oauth2 (we already removed the package it was used by)
* Update prepare_xdist_nodes.sh with new Django requirements file options
* Revert "Use pip-sync to make sure that dep cache tarball can go safely stale"
This reverts commit d435f4cd3e.
* Revert "Extract worker setup into own shell script, as much as possible"
This reverts commit 0a079e757c.
