We have a need to lock the version of Django for production and tests, but
also to test on newer versions of Django so that we can get the repo ready
for long-term-support releases.
We've been doing that by extracting the `django==x.y.z` from the
pip-compiled files and moving it to a django.txt that is then co-installed
but can be overridden during tests. The problem is that this can result
in broken packages.
The approach here is to have `make test-requirements` continue to
ensure a consistent set of packages, and then install a different
Django on top of that in the CI script -- and call `pip check` to make
sure that combination isn't broken.
Adding Django 4.0 to the unit-tests.yml matrix will now correctly
result in this error and a failing job:
`django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.`
The other half of this is to change other CI runners to remove their
ability to control the Django version, since it's complicated to make
this work, and we probably only need it in unit-tests.yml. Convert them
to just use `make test-requirements`.
Also:
- Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our
own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option
that it knows). This is needed to allow `make test-requirements` to do
the pip calls. An alternative would be to set a pip-options env var for
the make target to use, but `PIP_SRC` already exists.
- Remove outdated modifications to common_constraints
- Add comment explaining why pylint tests need dev-requirements
Packages can be added to package-lock.json that will fail to
install on certain systems. For example, the `fsevents` NPM
package, which only works on macOS, was listed as a requirement
in the file.
`npm install` will happily skip over such packages.
`npm clean-install`, however, will exit with a fatal error.
Throughout several doc pages, we have recently been encouraging
folks to use `clean-install` instead of `install`, because its
strictness makes it more reproducible. To ensure that `clean-install`
is working reliably at any given time, we should use `clean-install`
in our CI pipeline.
- Moving xmodule folder to root as we're dissolving sub-projects of common folder in edx-platform
- More info: https://openedx.atlassian.net/browse/BOM-2579
- -e common/lib/xmodule has been removed from the requirements as xmodule has itself become the part of edx-platform and not being installed through requirements
- The test files common/lib/xmodule/test_files/ have been removed as they are not being used anymore
This was causing issue with Django 3.2, as Django has restricted to only use language from the pre-defined set of languages provided by Django.
BOM-2870
Does 3 things:
(1) Use django for modulestore tests
(2) Use normal LMS settings for modulestore tests instead of openedx/tests/settings.py
(3) Simplify some TestCase subclasses by converting them to use ModuleStoreTestCase
Details and rationale:
(1) Currently parts of the modulestore test suite are designed to run "without django", although there is still a lot of django functionality imported at times, and many of the tests do in fact use django. But for the upcoming PR #27565 (moving split's course indexes from MongoDB to MySQL), we will need to always have Django enabled. So this commit paves the way for that change.
(2) The previous tests that did use Django used a special settings file, openedx/tests/settings.py which made some debugging confusing because those tests had quite different django settings than other tests. This change deletes that file and runs the tests using the LMS test settings.
(3) The test suite also contains many different ways of initializing and testing a modulestore, with significant differences in their configuration, and also a lot of repetition. I find this makes understanding, debugging and writing tests more difficult. So this commit also reduces the number of different "test case using modulestore" base classes:
* Simplifies MixedWithOptionsTestCase and MixedSplitTestCase by making them simple subclasses of ModuleStoreTestCase.
* Removes PureModulestoreTestCase.
* build: Removed the diff-quality step
Applied lint-amnesty on all the warnings
Removed pylint thresholds comparison code and related tests
Co-authored-by: Usama Sadiq <usama.sadiq@arbisoft.com>
When we tried upgrading Jenkins workers from Ubuntu 16.04 to 20.04, `diff-cover` started silently failing with a return code of `1`. We suspect this is due to https://github.com/Bachmann1234/diff_cover/issues/153 , so we're adding the workaround described in that ticket.
The requirements/edx/private.txt file is for dev's own private package
needs. There are two installation mechanisms in edx-platform, and
neither handled the file properly:
- `paver install_prereqs` had the wrong file name. The file was moved
almost three years ago, and paver wasn't kept up.
- `make requirements` used `private.*` which included private.in, which
pip-sync balks at.
On missing sass directory, the following kind of warning was printed:
Sass dir '[{'sass_source_dir': path('lms/static/sass'), 'css_destination_dir': Path('/openedx/themes/indigo/lms/static/css'), 'lookup_paths': [Path('/openedx/themes/indigo/lms/static/sas│.. s/partials'), path('lms/static/sass/partials'), path('lms/static/sass')]}, {'sass_source_dir': Path('/openedx/themes/indigo/lms/static/sass'), 'css_destination_dir': Path('/openedx/themes│. /indigo/lms/static/css'), 'lookup_paths': [Path('/openedx/themes/indigo/lms/static/sass/partials'), path('lms/static/sass/partials'), path('lms/static/sass')]}, {'sass_source_dir': Path('│ /openedx/themes/indigo/lms/static/certificates/sass'), 'css_destination_dir': Path('/openedx/themes/indigo/lms/static/certificates/css'), 'lookup_paths': [Path('/openedx/themes/indigo/lms│ /static/sass/partials'), Path('/openedx/themes/indigo/lms/static/sass')]}]' does not exists, skipping sass compilation for '/openedx/themes/indigo'
Which was rather inconvenient. Instead, we now print the following
warning:
Sass dir '/openedx/themes/indigo/lms/static/certificates/sass' does not exists, skipping sass compilation for '/openedx/themes/indigo'
This should solve the problem of getting `nodejs: not found` when
running `paver test_js_run -s lms` in lms-shell, which cropped up in
the past few days.
I don't know why this stopped working recently, but I don't have `nodejs`
on my recently re-created devstack, and some people with an older devstack
have both. Maybe a recent node.js release got rid of an alias.
Part of the notifier service deprecation (DEPR-106).
Also removed pdfminer from the package uninstall list, since we no longer install the package it conflicts with either.
