diff --git a/docs/en_us/internal/jenkins/builds.md b/docs/en_us/internal/jenkins/builds.md new file mode 100644 index 0000000000..102bd313e1 --- /dev/null +++ b/docs/en_us/internal/jenkins/builds.md @@ -0,0 +1,72 @@ +## Starting and checking the status of builds +There are three ways that builds start in our jenkins testing infastructure for edx-platform. + +##### 1) Automatic builds for Pull Requests +* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people). + If you are not, someone will start a build for you during the pull request review process. You will still be able to + view the build results as described below. + +* How it gets started + + >* When you submit a pull request to the edx-platform repository, a jenkins build will + > automatically start and run unit and acceptance tests at the most recent commit. + >* When you add a new commit to the PR, a new build will be run for those changes. + >* Sometimes it may take a little while for the build to start. That usually just means that + > jenkins is pretty busy. + +* How it is reported + + >* You will know a build is started if you see this: + > + > ![Running Tests](jenkins_images/started_tests.png) + > + >* When it is finished you will see either a green checkmark or a red X, indicating that the + > build either passed or failed respectively. + > + > ![Passed Tests](jenkins_images/passed_tests.png) + > + > ![Failed Tests](jenkins_images/failed_tests.png) + >* You can click on 'details' to take you to the jenkins build report. + +##### 2) Manually started builds for pull requests + +* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people). + +* How it gets started + + >1. Go to [edx-all-tests-manual-pr](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/) + >2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the + > upper right corner of the page. Else, the 'log in' link will be there. + >3. Click 'Build with Parameters' in the left navigation column. + > + > ![Build with Params](jenkins_images/build_w_params.png) + > + >4. Enter the PR number from edx-platform that you want to test. + >5. Click on 'Build'. + +* How it is reported + + >* This will be reported the same as Automatic builds for Pull Requests are. (See point 1 of this + > section.) + >* When you start the build, it will redirect you to the log page. You can watch this page for + > results as well. + +##### 3) Manually started builds for commits + +* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people). + +* How it gets started + + >1. Go to [edx-all-tests-manual-commit](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/) + >2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the + > upper right corner of the page. Else, the 'log in' link will be there. + >3. Click 'Build with Parameters' in the left navigation column. + >4. Enter the commit hash that you want to test. + >5. Click on 'Build'. + +* How it is reported + + >* When you start the build, it will redirect you to the log page. You can watch this page for + > results. + >* The results will also be reported to github, and will show up next to your commit on a PR or + > other places that the commit is listed (search results, etc.). diff --git a/docs/en_us/internal/jenkins/jenkins.md b/docs/en_us/internal/jenkins/jenkins.md new file mode 100644 index 0000000000..b8465193bd --- /dev/null +++ b/docs/en_us/internal/jenkins/jenkins.md @@ -0,0 +1,5 @@ +## Guides for edx-platform tests on Jenkins + +* [Starting and seeing the status of builds](builds.md) +* [Finding reports (coverage, pep8/pylint, test results)](results.md) +* [Troubleshooting failures/errors](troubleshooting.md) \ No newline at end of file diff --git a/docs/en_us/internal/jenkins/jenkins_images/build_link.png b/docs/en_us/internal/jenkins/jenkins_images/build_link.png new file mode 100644 index 0000000000..a5f1bc0aa5 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/build_link.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/build_w_params.png b/docs/en_us/internal/jenkins/jenkins_images/build_w_params.png new file mode 100644 index 0000000000..87abe49fc0 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/build_w_params.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/console_log.png b/docs/en_us/internal/jenkins/jenkins_images/console_log.png new file mode 100644 index 0000000000..89b8979822 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/console_log.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/coverage_report_nav_links.png b/docs/en_us/internal/jenkins/jenkins_images/coverage_report_nav_links.png new file mode 100644 index 0000000000..c5093d29a4 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/coverage_report_nav_links.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/failed_shard.png b/docs/en_us/internal/jenkins/jenkins_images/failed_shard.png new file mode 100644 index 0000000000..f9407de638 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/failed_shard.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/failed_tests.png b/docs/en_us/internal/jenkins/jenkins_images/failed_tests.png new file mode 100644 index 0000000000..ea12e28602 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/failed_tests.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/passed_tests.png b/docs/en_us/internal/jenkins/jenkins_images/passed_tests.png new file mode 100644 index 0000000000..f718b1038b Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/passed_tests.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/pep8_pylint_graph.png b/docs/en_us/internal/jenkins/jenkins_images/pep8_pylint_graph.png new file mode 100644 index 0000000000..1565d1eb8f Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/pep8_pylint_graph.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/quality_artifacts.png b/docs/en_us/internal/jenkins/jenkins_images/quality_artifacts.png new file mode 100644 index 0000000000..5948f42a0c Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/quality_artifacts.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/quality_report_nav_links.png b/docs/en_us/internal/jenkins/jenkins_images/quality_report_nav_links.png new file mode 100644 index 0000000000..fe985c78b0 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/quality_report_nav_links.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/quality_shard.png b/docs/en_us/internal/jenkins/jenkins_images/quality_shard.png new file mode 100644 index 0000000000..eba7580759 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/quality_shard.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/started_tests.png b/docs/en_us/internal/jenkins/jenkins_images/started_tests.png new file mode 100644 index 0000000000..df3c208043 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/started_tests.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/test_results_link.png b/docs/en_us/internal/jenkins/jenkins_images/test_results_link.png new file mode 100644 index 0000000000..b56e233053 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/test_results_link.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/test_results_page.png b/docs/en_us/internal/jenkins/jenkins_images/test_results_page.png new file mode 100644 index 0000000000..04adbea89d Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/test_results_page.png differ diff --git a/docs/en_us/internal/jenkins/jenkins_images/unit-bok-choy-shards.png b/docs/en_us/internal/jenkins/jenkins_images/unit-bok-choy-shards.png new file mode 100644 index 0000000000..52cc7214f1 Binary files /dev/null and b/docs/en_us/internal/jenkins/jenkins_images/unit-bok-choy-shards.png differ diff --git a/docs/en_us/internal/jenkins/results.md b/docs/en_us/internal/jenkins/results.md new file mode 100644 index 0000000000..c5f9e63825 --- /dev/null +++ b/docs/en_us/internal/jenkins/results.md @@ -0,0 +1,125 @@ +# Looking at build results +_build log_ will refer to something from the following urls: + +``` +https://jenkins.testeng.edx.org/job/edx-all-tests-auto-pr/[NUMBER]/ +https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/[NUMBER]/ +https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/[NUMBER]/ +``` + +_build report_ will refer to something with the following urls: + +``` +https://jenkins.testeng.edx.org/job/edx-platform-report/[NUMBER]/ +``` + +Note that navigating to a build via github (clicking 'details') while it is in a pending or failed state +will take you to a _build log_, but doing so after it has reported back as passed will bring you +to a _build report_. To get to a _build log_ from a _build report_, click on the build number in +the 'Upstream Builds' section: + +>![Upstream Builds](jenkins_images/build_link.png) + + +## Coverage Reports + +Note: Coverage reports are only generated when your tests pass. + +* How to find: + + __From the build report__ + + > On the left side you will see: + > + > * 'Coverage.py Report' which shows the html reports produced by the coverage + > package. + > * 'Diff Coverage Report' which shows the coverage of _new_ lines of code + > + > ![Coverage Nav Links](jenkins_images/coverage_report_nav_links.png) + + __From the build log__ + + >1. Click on one of the unit test or bok-choy shards. + > + > ![Unit and BokChoy Shards](jenkins_images/unit-bok-choy-shards.png) + > + >2. Click on 'Build Artifacts'. + >3. You can find the generated html in the reports folder. + > - For unit tests: + > + reports/cms/cover + > + reports/common/lib/calc/cover + > + reports/common/lib/capa/cover + > + reports/common/lib/xmodule/cover + > + reports/lms/cover + > - For bok-choy: + > + reports/bok_choy/cover + +## Code Quality Reports + +These are generated regardless of test restults. +Note that a passing build does __not__ mean that you have not introduced new pep8/pylint violations. + +* How to find: + + __From the build report__ + > On the left side you should see: + > * 'Diff Quality' which shows _new_ violations + > * 'Violations' which shows _all_ violations + > + > ![Quality Nav Links](jenkins_images/quality_report_nav_links.png) + > + > There will also be a graph on the build report page showing trends of pep8 and pylint + > violations. + > + > ![Quality Graph](jenkins_images/pep8_pylint_graph.png) + + __From the build log__ + + > 1. Click on the quality shard to get to the report for the pep8 and pylint results. + > + > ![Quality Shard](jenkins_images/quality_shard.png) + > + > 2. Under 'Build Artifacts', you should see the reports. + > + > ![Quality Reports](jenkins_images/quality_artifacts.png) + > + + + "diff\_quality\_pep8.html" and "diff\_quality\_pylint.html" are probably the most + useful to look at. + - If don't see these here, click on 'Build Artifacts' and check in + 'reports/diff_quality' + - These show how many _new_ violations have been introduced in your changes. + + You can also download the full reports (files ending in '.report') + - These show _all_ of the found violations, not just newly introduced ones. + + +## Test Results + +###### Finding results for individual tests + +* If your build fails, the shards that had failures will have a red dot in the chart on the build log. + + > ![Failed Unit Tests](jenkins_images/failed_shard.png) + +* Click on the failed shard, and at the bottom of the new page you should see something like this: + + > ![Test Results Link](jenkins_images/test_results_link.png) + +* The 'Test Result' link (above) will take you to a summary of the test results (below). + + > ![Test Results Page](jenkins_images/test_results_page.png) + +* From this page you can click through the test names to get to the console log for each test. + +###### Console output for the shard + +Examining the console output of a build can provide more information about the test results. For example, +sometimes a worker will have an error during setup for the tests. If this happens, you will see a red dot for the shard, but no +test failures reported (since the tests did not run). In this case, checking the console log can be helful to determine the +reason for the build failure. It can sometimes be useful to review even if you can find the individual error message for a test. + + +To find it from the build log, click on the 'Console Output' link in the left navigation column. + +> ![Console Log](jenkins_images/console_log.png) + diff --git a/docs/en_us/internal/jenkins/troubleshooting.md b/docs/en_us/internal/jenkins/troubleshooting.md new file mode 100644 index 0000000000..27e3f3d4fb --- /dev/null +++ b/docs/en_us/internal/jenkins/troubleshooting.md @@ -0,0 +1,32 @@ +# Troubleshooting failing jenkins builds + +###### Check the logs to find out why your build failed. + * See [here](results.md#console-output-for-the-shard) for how to find the log. + * If the log shows the build erroring out before tests begin running, report this to the test + engineering team. Be sure to share the link to your build log. + +###### Run tests on devstack + * If a build fails on jenkins, try to reporoduce and debug the failures in devstack. + * See the [testing guide](../testing.md) for how to run the tests. + - For lettuce acceptance test debugging, see [this section](../testing.md#debugging-acceptance-tests-on-vagrant). + * For more debugging tips, see the [testing FAQ](https://github.com/edx/edx-platform/wiki/Test-engineering-FAQ). + There you'll find tips for: + - setting breakpoints in tests + - visually debugging acceptance tests + - running tests from a single file + - and more.. + + * "What's devstack?" ... See [here](https://github.com/edx/configuration/wiki/edX-Developer-Stack) + +###### Check the 'flaky test' list + * Known flaky tests can be found with filters on the openedx Jira site + - [Known unresolved flaky tests](https://openedx.atlassian.net/issues/?filter=10600) + - [Recently resolved flaky tests](https://openedx.atlassian.net/issues/?filter=11001) + * Remember that a test being listed as flaky doesn't mean that it can't fail for other reasons. Look into + the logs and confirm that it is failing for the same reason as listed in the issue ticket. + * If your build has failure that is a recently resolved flaky test, try rebasing from master. (A new auto-pr + build will start when you do this on an open PR.) + +###### Check if the failure is occurring on the master branch + * Tests run for the master branch are [here](https://jenkins.testeng.edx.org/job/edx-all-tests-auto-master/). + You can inspect them the same way you would a PR build.