Merge pull request #4843 from edx/clytwynec/jenkins_how_to_doc

Clytwynec/jenkins how to doc

docs/en_us/internal/jenkins/builds.md

@@ -0,0 +1,72 @@
+## Starting and checking the status of builds
+There are three ways that builds start in our jenkins testing infastructure for edx-platform.  
+
+##### 1) Automatic builds for Pull Requests
+* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people).
+  If you are not, someone will start a build for you during the pull request review process. You will still be able to
+  view the build results as described below.
+
+* How it gets started  
+  
+  >* When you submit a pull request to the edx-platform repository, a jenkins build will
+  >  automatically start and run unit and acceptance tests at the most recent commit.
+  >* When you add a new commit to the PR, a new build will be run for those changes.
+  >* Sometimes it may take a little while for the build to start. That usually just means that
+  >  jenkins is pretty busy.
+
+* How it is reported  
+  
+  >* You will know a build is started if you see this:  
+  >
+  >  ![Running Tests](jenkins_images/started_tests.png)
+  >
+  >* When it is finished you will see either a green checkmark or a red X, indicating that the
+  >  build either passed or failed respectively.  
+  >
+  >  ![Passed Tests](jenkins_images/passed_tests.png) 
+  >
+  >  ![Failed Tests](jenkins_images/failed_tests.png)  
+  >* You can click on 'details' to take you to the jenkins build report.
+    
+##### 2) Manually started builds for pull requests
+
+* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people).
+
+* How it gets started  
+
+  >1. Go to [edx-all-tests-manual-pr](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/)
+  >2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the
+  >   upper right corner of the page. Else, the 'log in' link will be there.
+  >3. Click 'Build with Parameters' in the left navigation column.
+  >
+  >  ![Build with Params](jenkins_images/build_w_params.png)
+  >
+  >4. Enter the PR number from edx-platform that you want to test.
+  >5. Click on 'Build'.
+  
+* How it is reported  
+
+  >* This will be reported the same as Automatic builds for Pull Requests are. (See point 1 of this
+  >  section.)
+  >* When you start the build, it will redirect you to the log page.  You can watch this page for
+  >  results as well. 
+  
+##### 3) Manually started builds for commits
+
+* Permissions Required: You must be a _public_ member of the [edx organization on github](https://github.com/orgs/edx/people).
+
+* How it gets started
+  
+  >1. Go to [edx-all-tests-manual-commit](https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/)
+  >2. Make sure you are __logged in__. If you are already logged in, your username and a 'log out' link will be in the
+  >   upper right corner of the page. Else, the 'log in' link will be there.
+  >3. Click 'Build with Parameters' in the left navigation column.
+  >4. Enter the commit hash that you want to test.
+  >5. Click on 'Build'.
+  
+* How it is reported
+  
+  >* When you start the build, it will redirect you to the log page.  You can watch this page for
+  >  results. 
+  >* The results will also be reported to github, and will show up next to your commit on a PR or
+  >  other places that the commit is listed (search results, etc.). 

docs/en_us/internal/jenkins/jenkins.md

@@ -0,0 +1,5 @@
+## Guides for edx-platform tests on Jenkins
+
+* [Starting and seeing the status of builds](builds.md)
+* [Finding reports (coverage, pep8/pylint, test results)](results.md)
+* [Troubleshooting failures/errors](troubleshooting.md)

docs/en_us/internal/jenkins/results.md

@@ -0,0 +1,125 @@
+# Looking at build results
+_build log_ will refer to something from the following urls:
+
+```
+https://jenkins.testeng.edx.org/job/edx-all-tests-auto-pr/[NUMBER]/
+https://jenkins.testeng.edx.org/job/edx-all-tests-manual-pr/[NUMBER]/
+https://jenkins.testeng.edx.org/job/edx-all-tests-manual-commit/[NUMBER]/ 
+```
+
+_build report_ will refer to something with the following urls:
+
+```
+https://jenkins.testeng.edx.org/job/edx-platform-report/[NUMBER]/
+```
+
+Note that navigating to a build via github (clicking 'details') while it is in a pending or failed state
+will take you to a _build log_, but doing so after it has reported back as passed will bring you
+to a _build report_.  To get to a _build log_ from a _build report_, click on the build number in
+the 'Upstream Builds' section:
+
+>![Upstream Builds](jenkins_images/build_link.png)  
+
+
+## Coverage Reports
+
+Note: Coverage reports are only generated when your tests pass.
+  
+* How to find:
+
+  __From the build report__
+
+    > On the left side you will see:  
+    >
+    > * 'Coverage.py Report' which shows the html reports produced by the coverage
+    > package.
+    > * 'Diff Coverage Report' which shows the coverage of _new_ lines of code  
+    >
+    >   ![Coverage Nav Links](jenkins_images/coverage_report_nav_links.png)
+
+  __From the build log__ 
+
+    >1. Click on one of the unit test or bok-choy shards.
+    >
+    >  ![Unit and BokChoy Shards](jenkins_images/unit-bok-choy-shards.png)
+    >
+    >2. Click on 'Build Artifacts'.
+    >3. You can find the generated html in the reports folder.
+    >    - For unit tests: 
+    >      + reports/cms/cover
+    >      + reports/common/lib/calc/cover
+    >      + reports/common/lib/capa/cover
+    >      + reports/common/lib/xmodule/cover
+    >      + reports/lms/cover
+    >    - For bok-choy: 
+    >      + reports/bok_choy/cover
+
+## Code Quality Reports  
+
+These are generated regardless of test restults.
+Note that a passing build does __not__ mean that you have not introduced new pep8/pylint violations.
+
+* How to find:
+
+  __From the build report__
+  > On the left side you should see:
+  >   * 'Diff Quality' which shows _new_ violations
+  >   * 'Violations' which shows _all_ violations
+  >
+  >   ![Quality Nav Links](jenkins_images/quality_report_nav_links.png)
+  >
+  > There will also be a graph on the build report page showing trends of pep8 and pylint
+  > violations.
+  >
+  >   ![Quality Graph](jenkins_images/pep8_pylint_graph.png)
+
+  __From the build log__
+
+  > 1. Click on the quality shard to get to the report for the pep8 and pylint results.  
+  >
+  >   ![Quality Shard](jenkins_images/quality_shard.png)
+  > 
+  > 2. Under 'Build Artifacts', you should see the reports.  
+  >
+  >   ![Quality Reports](jenkins_images/quality_artifacts.png)
+  >
+
+    + "diff\_quality\_pep8.html" and "diff\_quality\_pylint.html" are probably the most 
+      useful to look at. 
+      - If don't see these here, click on 'Build Artifacts' and check in 
+      'reports/diff_quality'
+      - These show how many _new_ violations have been introduced in your changes.
+    + You can also download the full reports (files ending in '.report')
+      - These show _all_ of the found violations, not just newly introduced ones.
+
+
+## Test Results
+
+###### Finding results for individual tests
+
+* If your build fails, the shards that had failures will have a red dot in the chart on the build log.
+ 
+  > ![Failed Unit Tests](jenkins_images/failed_shard.png)
+
+* Click on the failed shard, and at the bottom of the new page you should see something like this:
+ 
+  > ![Test Results Link](jenkins_images/test_results_link.png)
+ 
+* The 'Test Result' link (above) will take you to a summary of the test results (below).
+  
+  > ![Test Results Page](jenkins_images/test_results_page.png)
+ 
+* From this page you can click through the test names to get to the console log for each test.
+
+###### Console output for the shard
+
+Examining the console output of a build can provide more information about the test results. For example,
+sometimes a worker will have an error during setup for the tests. If this happens, you will see a red dot for the shard, but no
+test failures reported (since the tests did not run).  In this case, checking the console log can be helful to determine the
+reason for the build failure. It can sometimes be useful to review even if you can find the individual error message for a test.
+
+
+To find it from the build log, click on the 'Console Output' link in the left navigation column.
+
+> ![Console Log](jenkins_images/console_log.png)
+

docs/en_us/internal/jenkins/troubleshooting.md

@@ -0,0 +1,32 @@
+# Troubleshooting failing jenkins builds
+
+###### Check the logs to find out why your build failed.
+  * See [here](results.md#console-output-for-the-shard) for how to find the log.
+  * If the log shows the build erroring out before tests begin running, report this to the test 
+    engineering team. Be sure to share the link to your build log.
+
+###### Run tests on devstack
+  * If a build fails on jenkins, try to reporoduce and debug the failures in devstack.
+  * See the [testing guide](../testing.md) for how to run the tests.
+    - For lettuce acceptance test debugging, see [this section](../testing.md#debugging-acceptance-tests-on-vagrant).
+  * For more debugging tips, see the [testing FAQ](https://github.com/edx/edx-platform/wiki/Test-engineering-FAQ).
+    There you'll find tips for:
+      - setting breakpoints in tests
+      - visually debugging acceptance tests
+      - running tests from a single file
+      - and more..
+
+  * "What's devstack?" ... See [here](https://github.com/edx/configuration/wiki/edX-Developer-Stack)
+
+###### Check the 'flaky test' list
+  * Known flaky tests can be found with filters on the openedx Jira site
+    - [Known unresolved flaky tests](https://openedx.atlassian.net/issues/?filter=10600)
+    - [Recently resolved flaky tests](https://openedx.atlassian.net/issues/?filter=11001)
+  * Remember that a test being listed as flaky doesn't mean that it can't fail for other reasons. Look into
+    the logs and confirm that it is failing for the same reason as listed in the issue ticket.
+  * If your build has failure that is a recently resolved flaky test, try rebasing from master. (A new auto-pr
+    build will start when you do this on an open PR.)
+
+###### Check if the failure is occurring on the master branch
+  * Tests run for the master branch are [here](https://jenkins.testeng.edx.org/job/edx-all-tests-auto-master/).
+    You can inspect them the same way you would a PR build.