Unclear if there was a change in the focal repositories
or if there was an issue with something else. The noble
repositories don't support 7.0, so we're stuck here
until we upgrade to 8.0.
Operators Note: In newer versions of ubuntu the MD4 hashing algorithm
is disabled by default. To enable it the openssl config needs to be
updated in a manner similar to what's being done here. Alternatively,
you can set the `FEATURES['ENABLE_BLAKE2B_HASHING']` setting to `True`
which will switch to a newer hashing algorithm where MD4 was previously
used.
Because this hashing is being used as a part of the edx-platform caching
mechanism, this will effectively clear the cache for the items that use
this hash. The will impact any items where the cache key might have been
too big to store in memcache so it's hard to predict exactly which items
will be impacted.
BREAKING CHANGE: See the operator note above for more details as this
may break for users transitioning from Ubuntu 20.04 to newer versions.
This code does not have any dependencies that are specific to any specific
version of ubuntu. So instead of testing on a specific version and then needing
to do work to keep the versions up-to-date, we switch to the ubuntu-latest
target which should be sufficient for testing purposes.
This work is being done as a part of https://github.com/openedx/platform-roadmap/issues/377
closes https://github.com/openedx/edx-platform/issues/35314
When a shard of unit-tests.yml fails, we want the `success` job to be
maked "Failed" (not "Skipped"). That's because "Failed" blocks the PR
from merging, whereas "Skipped" does not. This change ensures that
`success` always runs to completion rather than being cancelled as soon
as a unit test shard fails or is cancelled.
From https://github.com/marketplace/actions/alls-green#options:
> Important: For this to work properly, it is a must to have the job always run,
> otherwise GitHub will make it skipped when any of the dependencies fail. In
> some contexts, skipped is interpreted as success which may lead to undersired,
> unobvious and even dangerous (as in security breach "dangerous") side-effects.
Closes https://github.com/openedx/edx-platform/issues/34789
This reverts commit 52adce48d1 because we were getting this in the build pipeline:
```
WARNING: lxml 5.2.1 does not provide the extra 'html-clean'
ERROR: Exception:
Traceback (most recent call last):
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
status = self.run(options, args)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 203, in wrapper
return func(self, options, args)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 315, in run
requirement_set = resolver.resolve(
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 366, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 221, in _attempt_to_pin_criterion
satisfied = all(
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 222, in <genexpr>
self._p.is_satisfied_by(requirement=r, candidate=candidate)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 178, in is_satisfied_by
return requirement.is_satisfied_by(candidate)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/requirements.py", line 84, in is_satisfied_by
assert candidate.name == self.name, (
AssertionError: Internal issue: Candidate is not for this requirement lxml[html-clean,html-clean] vs lxml[html-clean]
WARNING: You are using pip version 21.2.1; however, version 24.0 is available.
You should consider upgrading via the '/edx/app/edxapp/venvs/edxapp/bin/python -m pip install --upgrade pip' command.
```
Currently, ./xmodule/ unit tests are only run with LMS settings. However,
./common/ and ./xmodule/ are run twice: once with LMS settings and once with
CMS settings.
Just like ./common/ and ./openedx/, the unit tests in ./xmodule/ validate
behavior in both LMS and CMS. So, order to fully test ./xmodule/, we should to
run its tests with CMS settings too.
This will enable us to better validate certain LibraryContentBlocks behaviors
being touched by https://github.com/openedx/edx-platform/pull/33263 which can't
be expressed under LMS settings.
Also in this commit:
* refactor: rename the shards to be clear whether they're running under LMS or CMS
* docs: correct comments regarding conditions under which codejail's
test_cant_do_something_forbidden is skipped.
* test: update a unit test which was using the now-deleted library_sourced block to use
library_content block instead.
- Remove pylint warning suppression from SHAKE-128 hexdigest calls; these
are no longer needed as of astroid 2.12.12. For background, see issue
<https://github.com/PyCQA/pylint/issues/4039>. Also, correct a comment
that referred to SHAKE-256 instead of SHAKE-128.
- Replace "released" with "beta" in several places where there was a copy
& paste error in dark_lang. Add mention of `beta_lang` to a docstring
where it was omitted.
- Remove comment regarding Mongo startup; the code it referred to has since
been removed.
- Fix typos.
We have a need to lock the version of Django for production and tests, but
also to test on newer versions of Django so that we can get the repo ready
for long-term-support releases.
We've been doing that by extracting the `django==x.y.z` from the
pip-compiled files and moving it to a django.txt that is then co-installed
but can be overridden during tests. The problem is that this can result
in broken packages.
The approach here is to have `make test-requirements` continue to
ensure a consistent set of packages, and then install a different
Django on top of that in the CI script -- and call `pip check` to make
sure that combination isn't broken.
Adding Django 4.0 to the unit-tests.yml matrix will now correctly
result in this error and a failing job:
`django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.`
The other half of this is to change other CI runners to remove their
ability to control the Django version, since it's complicated to make
this work, and we probably only need it in unit-tests.yml. Convert them
to just use `make test-requirements`.
Also:
- Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our
own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option
that it knows). This is needed to allow `make test-requirements` to do
the pip calls. An alternative would be to set a pip-options env var for
the make target to use, but `PIP_SRC` already exists.
- Remove outdated modifications to common_constraints
- Add comment explaining why pylint tests need dev-requirements
If a job "needs" earlier jobs, and one of the earlier jobs fails, then
the "needs" job will be marked as Skipped. A required check that is
skipped doesn't block merging.
The alls-green action has the correct logic to fail the aggregation job
if any of its required jobs fail.
We've been seeing some cross-version caching issues in the
edx-platform-runner unit tests, which apparently run on a "dirty"
environment—the virtualenv does not get cleared out between runs,
which probably improves performance but allows installed dependencies
to "leak" between runs. This results in errors between master and older
open-releases but could also prevent us from noticing missing deps.
By using pip-sync in the new CI Make targets (as we already do for
the regular `make requirements` target) we can ensure that any stale
dependencies from runs by other branches (or older versions of the
code) are removed.
Calling `make local-requirements` at the end of each `*-requirements`
target rather than making it a prerequisite is necessary for using sync,
since otherwise the local reqs would be wiped out.
The `requirements` target is also deduplicated into the newer
`dev-requirements` (aliased to it, with both installing private deps now.)
Adding a prerequisite of `pre-requirements` allows us to simplify some
workflow calls slightly. This ends up being the bulk of the commit by
line count. The pip lockfile also wasn't being used in the Makefile, so
I added that to pre-requirements as well.
Also fix leading whitespace issue in Makefile.
