Andal.LND/edx-platform
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update based on feedback.

Browse Source
This commit is contained in:
Feanil Patel
2020-03-19 16:12:23 -04:00
parent 82c50458a1
commit d0f584ab7e
1 changed files with 39 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
docs/decisions/0003-reduce-bokchoy-testing.rst
View File

@@ -1,58 +1,67 @@
Status
~~~~~~
======
Draft
Context
~~~~~~~
=======
edx-platform bokchoy tests are slow, flaky and difficult to debug. A quick assessment of their value shows that they might be more trouble than they are worth. And that we might get the same benefit with far fewer tests.
Baseline Data:(Last 7 days)
---------------------------
Total number of builds: 253(across 106 PRs)
Failures: 49(across 24 PRs)
True Failures: 10(across 6 PRs)
Failures that wouldn’t be caught by other test: 3(on 1 PR)
Color
=====
Of the real failures found, there was one PR which had a failure that was only found via bokchoy and a115 tests.
- This PR made a JS change which would have broken many pages from loading.
Recommendation
Baseline Data:
--------------
As an experiment, we should not run bokchoy tests but continue to run a11y tests which will reduce the total number of tests significantly but continue to act as a smoke test for issues that can be caused by the fact that our frontend in edx-platform is still quite highly coupled together.
This data was collected based on the results of bokchoy tests run across all edx-platform PRs over the last 7 days.
We'll run in this mode for a month while we collect more data according to the test plan below. This should give us either the confidence to significantly reduce the number of bokchoy tests or good reasons not to.
* Total number of builds: 253(across 106 PRs)
* Failures: 49(across 24 PRs)
* True Failures: 10(across 6 PRs)
* Failures that wouldn’t be caught by other test: 3(on 1 PR)
Color
~~~~~
Of the real failures found, there was one PR which had a failure that was only found via bokchoy and a11y tests.
* This PR made a JS change which would have broken many pages from loading.
Recommendation
==============
Based on the info we have so far, we should only run a suite of smoke tests in bokchoy that ensure the frontend is not entirely broken.
For the experiment, we will use the a11y bokchoy tests as simple stand-in for a suite of smoke tests, because it is already a much smaller suite of happy path tests.
During the experiment, if we find we are missing coverage via a regression, we will first add a missing Python or JavaScript unit test where possible. Only if this isn't possible would we add to the smoke suite of bokchoy tests.
We'll run in this mode for a month while we collect more data according to the test plan below. This should give us either the confidence to significantly reduce the number of bokchoy tests or good reasons not to.
Test Plan
=========
---------
1. Deactivate bokchoy tests on master and all PRs but leave a11y tests running.
- The a11y tests will act as a proxy for the small number of UI tests that would catch most major issues.
#. Deactivate bokchoy tests on master and all PRs but leave a11y tests running.
2. Collect data on which issues bokchoy would have caught by running them manually font-of-band).
- On a Daily cadense for 1 month.
* The a11y tests will act as a proxy for the small number of UI tests that would catch most major issues.
3. Assess Impact of change.
- We'll record the number of issues that bokchoy would have prevented.
- Both True issues and false positives(flakiness).
#. Collect data on which issues bokchoy would have caught by running them manually out-of-band).
* We'll look at the failures on the out-of-band bokchoy job to find any true failures that would be caught by the removed tests.
* On a Daily cadense for 1 month.
#. Assess Impact of change.
* We'll record the number of issues that bokchoy would have detected, when we manually run the bokchoy job out-of-band.
* Both True issues and false positives(flakiness).
Outcome: Decision on whether or not to reduce the number of bokchoy tests.
Experiment Results
------------------
==================
TBD
Consequences
------------
============
TBD