diff --git a/docs/en_us/internal/jenkins/builds.md b/docs/en_us/internal/jenkins/builds.md index 102bd313e1..839a42c830 100644 --- a/docs/en_us/internal/jenkins/builds.md +++ b/docs/en_us/internal/jenkins/builds.md @@ -1,72 +1 @@ -## Starting and checking the status of builds -There are three ways that builds start in our jenkins testing infastructure for edx-platform. - -##### 1) Automatic builds for Pull Requests -* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people). - If you are not, someone will start a build for you during the pull request review process. You will still be able to - view the build results as described below. - -* How it gets started - - >* When you submit a pull request to the edx-platform repository, a jenkins build will - > automatically start and run unit and acceptance tests at the most recent commit. - >* When you add a new commit to the PR, a new build will be run for those changes. - >* Sometimes it may take a little while for the build to start. That usually just means that - > jenkins is pretty busy. - -* How it is reported - - >* You will know a build is started if you see this: - > - > ![Running Tests](jenkins_images/started_tests.png) - > - >* When it is finished you will see either a green checkmark or a red X, indicating that the - > build either passed or failed respectively. - > - > ![Passed Tests](jenkins_images/passed_tests.png) - > - > ![Failed Tests](jenkins_images/failed_tests.png) - >* You can click on 'details' to take you to the jenkins build report. - -##### 2) Manually started builds for pull requests - -* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people). - -* How it gets started - - >1. Go to [edx-all-tests-manual-pr](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/) - >2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the - > upper right corner of the page. Else, the 'log in' link will be there. - >3. Click 'Build with Parameters' in the left navigation column. - > - > ![Build with Params](jenkins_images/build_w_params.png) - > - >4. Enter the PR number from edx-platform that you want to test. - >5. Click on 'Build'. - -* How it is reported - - >* This will be reported the same as Automatic builds for Pull Requests are. (See point 1 of this - > section.) - >* When you start the build, it will redirect you to the log page. You can watch this page for - > results as well. - -##### 3) Manually started builds for commits - -* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people). - -* How it gets started - - >1. Go to [edx-all-tests-manual-commit](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/) - >2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the - > upper right corner of the page. Else, the 'log in' link will be there. - >3. Click 'Build with Parameters' in the left navigation column. - >4. Enter the commit hash that you want to test. - >5. Click on 'Build'. - -* How it is reported - - >* When you start the build, it will redirect you to the log page. You can watch this page for - > results. - >* The results will also be reported to github, and will show up next to your commit on a PR or - > other places that the commit is listed (search results, etc.). +See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide diff --git a/docs/en_us/internal/jenkins/jenkins.md b/docs/en_us/internal/jenkins/jenkins.md index b8465193bd..839a42c830 100644 --- a/docs/en_us/internal/jenkins/jenkins.md +++ b/docs/en_us/internal/jenkins/jenkins.md @@ -1,5 +1 @@ -## Guides for edx-platform tests on Jenkins - -* [Starting and seeing the status of builds](builds.md) -* [Finding reports (coverage, pep8/pylint, test results)](results.md) -* [Troubleshooting failures/errors](troubleshooting.md) \ No newline at end of file +See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide diff --git a/docs/en_us/internal/jenkins/jenkins_images/build_link.png b/docs/en_us/internal/jenkins/jenkins_images/build_link.png deleted file mode 100644 index a5f1bc0aa5..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/build_link.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/build_w_params.png b/docs/en_us/internal/jenkins/jenkins_images/build_w_params.png deleted file mode 100644 index 87abe49fc0..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/build_w_params.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/console_log.png b/docs/en_us/internal/jenkins/jenkins_images/console_log.png deleted file mode 100644 index 89b8979822..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/console_log.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/coverage_report_nav_links.png b/docs/en_us/internal/jenkins/jenkins_images/coverage_report_nav_links.png deleted file mode 100644 index c5093d29a4..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/coverage_report_nav_links.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/failed_shard.png b/docs/en_us/internal/jenkins/jenkins_images/failed_shard.png deleted file mode 100644 index f9407de638..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/failed_shard.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/failed_tests.png b/docs/en_us/internal/jenkins/jenkins_images/failed_tests.png deleted file mode 100644 index ea12e28602..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/failed_tests.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/passed_tests.png b/docs/en_us/internal/jenkins/jenkins_images/passed_tests.png deleted file mode 100644 index f718b1038b..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/passed_tests.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/pep8_pylint_graph.png b/docs/en_us/internal/jenkins/jenkins_images/pep8_pylint_graph.png deleted file mode 100644 index 1565d1eb8f..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/pep8_pylint_graph.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/quality_artifacts.png b/docs/en_us/internal/jenkins/jenkins_images/quality_artifacts.png deleted file mode 100644 index 5948f42a0c..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/quality_artifacts.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/quality_report_nav_links.png b/docs/en_us/internal/jenkins/jenkins_images/quality_report_nav_links.png deleted file mode 100644 index fe985c78b0..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/quality_report_nav_links.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/quality_shard.png b/docs/en_us/internal/jenkins/jenkins_images/quality_shard.png deleted file mode 100644 index eba7580759..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/quality_shard.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/started_tests.png b/docs/en_us/internal/jenkins/jenkins_images/started_tests.png deleted file mode 100644 index df3c208043..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/started_tests.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/test_results_link.png b/docs/en_us/internal/jenkins/jenkins_images/test_results_link.png deleted file mode 100644 index b56e233053..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/test_results_link.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/test_results_page.png b/docs/en_us/internal/jenkins/jenkins_images/test_results_page.png deleted file mode 100644 index 04adbea89d..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/test_results_page.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/unit-bok-choy-shards.png b/docs/en_us/internal/jenkins/jenkins_images/unit-bok-choy-shards.png deleted file mode 100644 index 52cc7214f1..0000000000 Binary files a/docs/en_us/internal/jenkins/jenkins_images/unit-bok-choy-shards.png and /dev/null differ diff --git a/docs/en_us/internal/jenkins/results.md b/docs/en_us/internal/jenkins/results.md index c5f9e63825..839a42c830 100644 --- a/docs/en_us/internal/jenkins/results.md +++ b/docs/en_us/internal/jenkins/results.md @@ -1,125 +1 @@ -# Looking at build results -_build log_ will refer to something from the following urls: - -``` -https://jenkins.testeng.edx.org/job/edx-all-tests-auto-pr/[NUMBER]/ -https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/[NUMBER]/ -https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/[NUMBER]/ -``` - -_build report_ will refer to something with the following urls: - -``` -https://jenkins.testeng.edx.org/job/edx-platform-report/[NUMBER]/ -``` - -Note that navigating to a build via github (clicking 'details') while it is in a pending or failed state -will take you to a _build log_, but doing so after it has reported back as passed will bring you -to a _build report_. To get to a _build log_ from a _build report_, click on the build number in -the 'Upstream Builds' section: - ->![Upstream Builds](jenkins_images/build_link.png) - - -## Coverage Reports - -Note: Coverage reports are only generated when your tests pass. - -* How to find: - - __From the build report__ - - > On the left side you will see: - > - > * 'Coverage.py Report' which shows the html reports produced by the coverage - > package. - > * 'Diff Coverage Report' which shows the coverage of _new_ lines of code - > - > ![Coverage Nav Links](jenkins_images/coverage_report_nav_links.png) - - __From the build log__ - - >1. Click on one of the unit test or bok-choy shards. - > - > ![Unit and BokChoy Shards](jenkins_images/unit-bok-choy-shards.png) - > - >2. Click on 'Build Artifacts'. - >3. You can find the generated html in the reports folder. - > - For unit tests: - > + reports/cms/cover - > + reports/common/lib/calc/cover - > + reports/common/lib/capa/cover - > + reports/common/lib/xmodule/cover - > + reports/lms/cover - > - For bok-choy: - > + reports/bok_choy/cover - -## Code Quality Reports - -These are generated regardless of test restults. -Note that a passing build does __not__ mean that you have not introduced new pep8/pylint violations. - -* How to find: - - __From the build report__ - > On the left side you should see: - > * 'Diff Quality' which shows _new_ violations - > * 'Violations' which shows _all_ violations - > - > ![Quality Nav Links](jenkins_images/quality_report_nav_links.png) - > - > There will also be a graph on the build report page showing trends of pep8 and pylint - > violations. - > - > ![Quality Graph](jenkins_images/pep8_pylint_graph.png) - - __From the build log__ - - > 1. Click on the quality shard to get to the report for the pep8 and pylint results. - > - > ![Quality Shard](jenkins_images/quality_shard.png) - > - > 2. Under 'Build Artifacts', you should see the reports. - > - > ![Quality Reports](jenkins_images/quality_artifacts.png) - > - - + "diff\_quality\_pep8.html" and "diff\_quality\_pylint.html" are probably the most - useful to look at. - - If don't see these here, click on 'Build Artifacts' and check in - 'reports/diff_quality' - - These show how many _new_ violations have been introduced in your changes. - + You can also download the full reports (files ending in '.report') - - These show _all_ of the found violations, not just newly introduced ones. - - -## Test Results - -###### Finding results for individual tests - -* If your build fails, the shards that had failures will have a red dot in the chart on the build log. - - > ![Failed Unit Tests](jenkins_images/failed_shard.png) - -* Click on the failed shard, and at the bottom of the new page you should see something like this: - - > ![Test Results Link](jenkins_images/test_results_link.png) - -* The 'Test Result' link (above) will take you to a summary of the test results (below). - - > ![Test Results Page](jenkins_images/test_results_page.png) - -* From this page you can click through the test names to get to the console log for each test. - -###### Console output for the shard - -Examining the console output of a build can provide more information about the test results. For example, -sometimes a worker will have an error during setup for the tests. If this happens, you will see a red dot for the shard, but no -test failures reported (since the tests did not run). In this case, checking the console log can be helful to determine the -reason for the build failure. It can sometimes be useful to review even if you can find the individual error message for a test. - - -To find it from the build log, click on the 'Console Output' link in the left navigation column. - -> ![Console Log](jenkins_images/console_log.png) - +See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide diff --git a/docs/en_us/internal/jenkins/troubleshooting.md b/docs/en_us/internal/jenkins/troubleshooting.md index 27e3f3d4fb..839a42c830 100644 --- a/docs/en_us/internal/jenkins/troubleshooting.md +++ b/docs/en_us/internal/jenkins/troubleshooting.md @@ -1,32 +1 @@ -# Troubleshooting failing jenkins builds - -###### Check the logs to find out why your build failed. - * See [here](results.md#console-output-for-the-shard) for how to find the log. - * If the log shows the build erroring out before tests begin running, report this to the test - engineering team. Be sure to share the link to your build log. - -###### Run tests on devstack - * If a build fails on jenkins, try to reporoduce and debug the failures in devstack. - * See the [testing guide](../testing.md) for how to run the tests. - - For lettuce acceptance test debugging, see [this section](../testing.md#debugging-acceptance-tests-on-vagrant). - * For more debugging tips, see the [testing FAQ](https://github.com/edx/edx-platform/wiki/Test-engineering-FAQ). - There you'll find tips for: - - setting breakpoints in tests - - visually debugging acceptance tests - - running tests from a single file - - and more.. - - * "What's devstack?" ... See [here](https://github.com/edx/configuration/wiki/edX-Developer-Stack) - -###### Check the 'flaky test' list - * Known flaky tests can be found with filters on the openedx Jira site - - [Known unresolved flaky tests](https://openedx.atlassian.net/issues/?filter=10600) - - [Recently resolved flaky tests](https://openedx.atlassian.net/issues/?filter=11001) - * Remember that a test being listed as flaky doesn't mean that it can't fail for other reasons. Look into - the logs and confirm that it is failing for the same reason as listed in the issue ticket. - * If your build has failure that is a recently resolved flaky test, try rebasing from master. (A new auto-pr - build will start when you do this on an open PR.) - -###### Check if the failure is occurring on the master branch - * Tests run for the master branch are [here](https://jenkins.testeng.edx.org/job/edx-all-tests-auto-master/). - You can inspect them the same way you would a PR build. +See: https://openedx.atlassian.net/wiki/display/TE/Jenkins+Guide