b8a34f0c57
We have a need to lock the version of Django for production and tests, but also to test on newer versions of Django so that we can get the repo ready for long-term-support releases. We've been doing that by extracting the `django==x.y.z` from the pip-compiled files and moving it to a django.txt that is then co-installed but can be overridden during tests. The problem is that this can result in broken packages. The approach here is to have `make test-requirements` continue to ensure a consistent set of packages, and then install a different Django on top of that in the CI script -- and call `pip check` to make sure that combination isn't broken. Adding Django 4.0 to the unit-tests.yml matrix will now correctly result in this error and a failing job: `django-splash 1.2.1 has requirement Django<4.0, but you have django 4.0.8.` The other half of this is to change other CI runners to remove their ability to control the Django version, since it's complicated to make this work, and we probably only need it in unit-tests.yml. Convert them to just use `make test-requirements`. Also: - Simplify handling of `pip --src` by setting `PIP_SRC` (rather than our own `PIP_SRC_DIR`, which pip ignores because `--src-dir` isn't an option that it knows). This is needed to allow `make test-requirements` to do the pip calls. An alternative would be to set a pip-options env var for the make target to use, but `PIP_SRC` already exists. - Remove outdated modifications to common_constraints - Add comment explaining why pylint tests need dev-requirements
66 lines
2.4 KiB
Bash
66 lines
2.4 KiB
Bash
#!/usr/bin/env bash
|
|
|
|
set -e
|
|
|
|
source $HOME/jenkins_env
|
|
|
|
# Clear the mongo database
|
|
# Note that this prevents us from running jobs in parallel on a single worker.
|
|
mongo --quiet --eval 'db.getMongo().getDBNames().forEach(function(i){db.getSiblingDB(i).dropDatabase()})'
|
|
|
|
# Ensure we have fetched a target branch (origin/master, unless testing
|
|
# a release branc) Some of the reporting tools compare the checked out
|
|
# branch to a target branch; depending on how the GitHub plugin refspec
|
|
# is configured, this may not already be fetched.
|
|
if [ ! -z ${TARGET_BRANCH+x} ]; then
|
|
TARGET_BRANCH_WITHOUT_ORIGIN=$(echo "${TARGET_BRANCH}" | sed 's:^origin/::')
|
|
git fetch origin $TARGET_BRANCH_WITHOUT_ORIGIN:refs/remotes/origin/$TARGET_BRANCH_WITHOUT_ORIGIN
|
|
fi
|
|
|
|
# Reset the jenkins worker's virtualenv back to the
|
|
# state it was in when the instance was spun up.
|
|
if [ -z ${PYTHON_VERSION+x} ] || [[ ${PYTHON_VERSION} == 'null' ]]; then
|
|
VENV_PATH="$HOME"
|
|
ARCHIVED_VENV="edx-venv_clean.tar.gz"
|
|
else
|
|
VENV_PATH="$HOME/edx-venv-$PYTHON_VERSION"
|
|
ARCHIVED_VENV="edx-venv_clean-$PYTHON_VERSION.tar.gz"
|
|
fi
|
|
|
|
if [ -e $HOME/$ARCHIVED_VENV ]; then
|
|
rm -rf $VENV_PATH/edx-venv
|
|
tar -C $VENV_PATH -xf $HOME/$ARCHIVED_VENV
|
|
fi
|
|
|
|
# Load the npm packages from the time the worker was built
|
|
# into the npm cache. This is an attempt to reduce the number
|
|
# of times that npm gets stuck during an installation, by
|
|
# reducing the number of packages that npm needs to fetch.
|
|
if [ -e $HOME/edx-npm-cache_clean.tar.gz ]; then
|
|
echo "Loading archived npm packages into the local npm cache"
|
|
rm -rf $HOME/.npm
|
|
tar -C $HOME -xf $HOME/edx-npm-cache_clean.tar.gz
|
|
fi
|
|
|
|
# Activate the Python virtualenv
|
|
source $VENV_PATH/edx-venv/bin/activate
|
|
|
|
# Hack to fix up egg-link files given that the virtualenv is not relocatable
|
|
sed -i "s|^/home/jenkins/shallow-clone|`pwd`|" -- \
|
|
$VENV_PATH/edx-venv/lib/python*/site-packages/*.egg-link
|
|
make test-requirements
|
|
|
|
# add the node packages dir to PATH
|
|
PATH=$PATH:node_modules/.bin
|
|
|
|
echo "node version is `node --version`"
|
|
echo "npm version is `npm --version`"
|
|
|
|
# Log any paver or ansible command timing
|
|
TIMESTAMP=$(date +%s)
|
|
SHARD_NUM=${SHARD:="all"}
|
|
export PAVER_TIMER_LOG="test_root/log/timing.paver.$TEST_SUITE.$SHARD_NUM.log"
|
|
export ANSIBLE_TIMER_LOG="test_root/log/timing.ansible.$TIMESTAMP.log"
|
|
|
|
echo "This node is `curl http://169.254.169.254/latest/meta-data/hostname`"
|