Andal.LND/edx-platform
6
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update testing documentation

Browse Source
This commit is contained in:
Jay Zoldak
2014-04-14 10:38:42 -04:00
parent c1f75fe1a3
commit 7ff5d618d0
1 changed files with 39 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
docs/en_us/internal/testing.md
View File

@@ -5,6 +5,16 @@
We maintain three kinds of tests: unit tests, integration tests,
and acceptance tests.
Overall, you want to write the tests that **maximize coverage**
while **minimizing maintenance**.
In practice, this usually means investing heavily
in unit tests, which tend to be the most robust to changes in the code base.
![Test Pyramid](test_pyramid.png)
The pyramid above shows the relative number of unit tests, integration tests,
and acceptance tests. Most of our tests are unit tests or integration tests.
### Unit Tests
* Each test case should be concise: setup, execute, check, and teardown.
@@ -31,8 +41,8 @@ You do not need to test every possible input--that's what unit
tests are for. Instead, focus on testing the "happy path"
to verify that the components work together correctly.
* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/) to simulate
HTTP requests to the server.
* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/)
to simulate HTTP requests to the server.
### UI Acceptance Tests
* Use these to test that major program features are working correctly.
@@ -41,15 +51,9 @@ HTTP requests to the server.
these tests simulate user interactions through the browser using
[splinter](http://splinter.cobrateam.info/).
Overall, you want to write the tests that **maximize coverage**
while **minimizing maintenance**.
In practice, this usually means investing heavily
in unit tests, which tend to be the most robust to changes in the code base.
![Test Pyramid](test_pyramid.png)
The pyramid above shows the relative number of unit tests, integration tests,
and acceptance tests. Most of our tests are unit tests or integration tests.
* We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) to
write end-user acceptance tests directly in Python, using the framework to
maximize reliability and maintainability.
## Test Locations
@@ -65,9 +69,10 @@ and test. For example, the test for `src/views/module.coffee`
should be written in `spec/views/module_spec.coffee`.
* UI acceptance tests:
- Set up and helper methods: `common/djangoapps/terrain`
- Tests: located in `features` subpackage within a Django app.
- Set up and helper methods, and stubs for external services: `common/djangoapps/terrain`
- Lettuce Tests: located in `features` subpackage within a Django app.
For example: `lms/djangoapps/courseware/features`
- Bok Choy Tests: Artifacts are located under `common/test/acceptance`
## Factories
@@ -199,75 +204,56 @@ To run JavaScript tests in your default browser:
These rake commands call through to a custom test runner. For more info, see [js-test-tool](https://github.com/edx/js-test-tool).
### Running Acceptance Tests (Bok Choy)
### Running Bok Choy Acceptance Tests
We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) for acceptance testing.
Bok Choy is a UI-level acceptance test framework for writing robust [Selenium](http://docs.seleniumhq.org/) tests in [Python](https://www.python.org/).
Bok Choy makes your acceptance tests reliable and maintainable by utilizing the Page Object and Promise design patterns.
**Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver) installed to run the tests in Chrome. The tests are confirmed to run with Chrome (not Chromium) version 28.0.1500.71 with ChromeDriver version 2.1.210398.
**Prerequisites**:
* These prerequisites are all automatically installed and available in [Devstack](https://github.com/edx/configuration/wiki/edX-Developer-Stack),
the supported development enviornment for the edX Platform.
* Chromedriver and Chrome (see Running Lettuce Acceptance Tests below for the latest tested versions)
* Mongo
* Memcache
* mySQL
To run all the acceptance tests:
To run all the bok choy acceptance tests:
rake test:bok_choy
To run tests faster by not collecting static files, you can use:
Once the database has been set up and the static files collected, you can use the 'fast'
option to skip those tasks. This option can also be used with any of the test specs below:
rake test:bok_choy:fast
To run single test:
To run single test, specify the name of the test file. For example:
rake test:bok_choy[TEST_FILE_NAME]
example
rake test:bok_choy[test_lms.py]
To run single test faster by not collecting static files:
To run single test faster by not repeating setup tasks:
rake test:bok_choy:fast[TEST_FILE_NAME]
rake test:bok_choy:fast[test_lms.py]
To test only specific feature:
To test only a certain feature, specify the file and the testcase class:
rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME]
example
rake test:bok_choy[test_lms.py:RegistrationTest]
To test only specific feature faster by not collecting static files:
rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME]
example
rake test:bok_choy:fast[test_lms.py:RegistrationTest]
To test only specific scenario:
To execute only a certain test case, specify the file name, class, and
test case method:
rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME.FUNCTION_NAME]
example
rake test:bok_choy[test_lms.py:RegistrationTest.test_register]
To test only specific scenario faster by not collecting static files:
rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME.FUNCTION_NAME]
example
rake test:bok_choy:fast[test_lms.py:RegistrationTest.test_register]
During acceptance test execution, log files are written to
test_root/log/bok_choy_lms.log
test_root/log/bok_choy_ora.log
test_root/log/bok_choy_studio.log
test_root/log/bok_choy_xqueue.log
test_root/log/bok_choy_comments.log
Failed test's screenshots are saved to:
test_root/log
During acceptance test execution, log files and also screenshots of failed tests
are captured in test_root/log.
To put a debugging breakpoint in a test use:
from nose.tools import set_trace; set_trace()
### Running Acceptance Tests
### Running Lettuce Acceptance Tests
We use [Lettuce](http://lettuce.it/) for acceptance testing.
Most of our tests use [Splinter](http://splinter.cobrateam.info/)
@@ -276,8 +262,7 @@ uses [Selenium](http://docs.seleniumhq.org/) to control the Chrome browser.
**Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver)
installed to run the tests in Chrome. The tests are confirmed to run
with Chrome (not Chromium) version 28.0.1500.71 with ChromeDriver
version 2.1.210398.
with Chrome (not Chromium) version 34.0.1847.116 with ChromeDriver version 2.6.232917.
To run all the acceptance tests:
rake test:acceptance