The last time we tried this upgrade we encountered timeouts on the quality job, which it now appears were due to the worker running pylint common running out of memory and killing the Jenkins process. Switching to a different worker type with double the RAM (8 GB vs. 4 GB) seems to have fixed this; about 5.5 GB was used. Upstream is aware of the high memory usage on large projects, it's apparently due primarily to a cache of parsed modules: https://github.com/PyCQA/pylint/issues/1495 .
Even after disabling some of the new checks that have been added, the new version of pylint found about twice as much to complain about. Just bumping the threshold for now to unblock the Django upgrade, we can try automated utilities like pyupgrade to fix some of these later.
Remove duplicate constraints in requirements/constraints.txt uncovered by the recent change in pip-compile output format. I sorted the pinned dependencies by name to try and make it more obvious if this happens again. I also upgraded to pip-tools 4.5.1, which removes the line numbers from the enhanced pip-compile output added in 4.5.0, which should reduce future diff churn and merge conflicts but means that there's a large diff this one last time.
Also unpin edx-search again after the previous change to do that was apparently broken by a merge conflict.
Unpin several more outdated dependencies whose changelogs don't contain any significant backwards incompatible changes. Also add "moto" to the list of packages to uninstall from existing environments, since it requires a jsondiff version that clashes with the one we now use (triggering a harmless but distracting warning on dependency updates).
We can potentially stop using path.py/path altogether by switching to pathlib in the Python 3 standard library, but that merits a separate PR of its own.
Also, note that I'm not actually unpinning freezegun; different PRs restricted it in both constraints.txt and test.in, I'm just removing the latter redundant constraint.
* Added pytest-json-report plugin
- modifying app-opts in setup.cfg
- adding hook to all conftest.py files in repo
- setting report to be saved to test_root/log/warnings.json
- Writing custom logic to save json report to avoid overwrite if pytest called twice
This was created to allow us to easily parse through test warnings in jenkins
In the script used to create the bok-choy database cache files, we've been passing in the parameters to omit timestamps that change for every run in some of the mysqldump commands but not all of them. Use them consistently so we can stop creating new redundant automated bok-choy DB cache PRs on almost every merge to edx-platform.
Some change in the requirements here may be responsible for increased
CPU and memory usage. This is causing production LMS for edx.org to run
into lots of performance issues. Reverting for now until we can narrow
it down to which change caused the issue.
