There are three packages which edx-platform needs in order to run which
were being installed transitively via paver.in. Since we are removing
paver.in, these dependencies need to be transferred into kernel.in:
* psutil
* pymemcache
* wrapt
Part of: https://github.com/openedx/edx-platform/issues/34467
Since `make pull_translations` is now performed at build time (in the new
Atlas workflow), it will generally only have access to base dependencies.
However, it depends on the `i18n_tools` command that was only installed at
the testing layer. This moves the dependency from testing to base.
(There is no increase in artifact size, as ora2 already depends on it and
is installed at the base later.)
This also adds `API_ACCESS_MANAGER_EMAIL` to the minimal settings
so that `make pull_translations` can be run using production envs in a
local environment (to test the lack of dependency on developer-level deps.)
This supports OEP-58.
This was to support an experiment in using coverage test contexts to
record what tests ran each line of code (informally known as
who-tests-what, or wtw). It never became operational.
Upgrade transifex-client version.
This is only made in order to run the tests with a transifex-client version compatible with python3.8
The actual problem that we have in edx-platform is that the last version of transifex-client has as constraint python-slugify<2.0.0 which is not compatible with the current requierements of edx-platform on the other hand, the current version used in edx-platform is not compatible with python3.8
The last time we tried this upgrade we encountered timeouts on the quality job, which it now appears were due to the worker running pylint common running out of memory and killing the Jenkins process. Switching to a different worker type with double the RAM (8 GB vs. 4 GB) seems to have fixed this; about 5.5 GB was used. Upstream is aware of the high memory usage on large projects, it's apparently due primarily to a cache of parsed modules: https://github.com/PyCQA/pylint/issues/1495 .
Even after disabling some of the new checks that have been added, the new version of pylint found about twice as much to complain about. Just bumping the threshold for now to unblock the Django upgrade, we can try automated utilities like pyupgrade to fix some of these later.
Unpin several more outdated dependencies whose changelogs don't contain any significant backwards incompatible changes. Also add "moto" to the list of packages to uninstall from existing environments, since it requires a jsondiff version that clashes with the one we now use (triggering a harmless but distracting warning on dependency updates).
We can potentially stop using path.py/path altogether by switching to pathlib in the Python 3 standard library, but that merits a separate PR of its own.
Also, note that I'm not actually unpinning freezegun; different PRs restricted it in both constraints.txt and test.in, I'm just removing the latter redundant constraint.
* Added pytest-json-report plugin
- modifying app-opts in setup.cfg
- adding hook to all conftest.py files in repo
- setting report to be saved to test_root/log/warnings.json
- Writing custom logic to save json report to avoid overwrite if pytest called twice
This was created to allow us to easily parse through test warnings in jenkins
Fix the issue that was preventing us from upgrading pytest. pytest does some manipulation of test packages that prevents `pkg_resources` from loading resources from them, but used to contain a workaround for the problem. That workaround was [removed](https://github.com/pytest-dev/pytest/issues/5392) in 4.6.0 as a performance enhancement when pytest switched from `pkg_resources` to `importlib-metadata` for its own entrypoint handling. This tripped up one of our test modules which defined classes that loaded templates from inside a test package. Moving these resources to the parent package fixes the problem.
More and more, `pkg_resources` is being abandoned in favor of `importlib-metadata` and `importlib_resources` as they have a simpler design with much better performance. However, `importlib_resources` doesn't support loading files from any directory which isn't itself a Python package (and doesn't allow direct use of paths including directories within the package). Jinja2 chose a [different approach](https://github.com/pallets/jinja/pull/1082) that we may want to emulate in our resource handling.
Also fixed usage of a removed `pytest.raises()` parameter and a bug in our configuration of the `common/lib` tests that became a problem after the upgrade.
Some change in the requirements here may be responsible for increased
CPU and memory usage. This is causing production LMS for edx.org to run
into lots of performance issues. Reverting for now until we can narrow
it down to which change caused the issue.
