diff --git a/docs/en_us/internal/testing.md b/docs/en_us/internal/testing.md index 13c1d06b0c..bd85acddc4 100644 --- a/docs/en_us/internal/testing.md +++ b/docs/en_us/internal/testing.md @@ -5,6 +5,16 @@ We maintain three kinds of tests: unit tests, integration tests, and acceptance tests. +Overall, you want to write the tests that **maximize coverage** +while **minimizing maintenance**. +In practice, this usually means investing heavily +in unit tests, which tend to be the most robust to changes in the code base. + +![Test Pyramid](test_pyramid.png) + +The pyramid above shows the relative number of unit tests, integration tests, +and acceptance tests. Most of our tests are unit tests or integration tests. + ### Unit Tests * Each test case should be concise: setup, execute, check, and teardown. @@ -31,8 +41,8 @@ You do not need to test every possible input--that's what unit tests are for. Instead, focus on testing the "happy path" to verify that the components work together correctly. -* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/) to simulate -HTTP requests to the server. +* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/) +to simulate HTTP requests to the server. ### UI Acceptance Tests * Use these to test that major program features are working correctly. @@ -41,15 +51,9 @@ HTTP requests to the server. these tests simulate user interactions through the browser using [splinter](http://splinter.cobrateam.info/). -Overall, you want to write the tests that **maximize coverage** -while **minimizing maintenance**. -In practice, this usually means investing heavily -in unit tests, which tend to be the most robust to changes in the code base. - -![Test Pyramid](test_pyramid.png) - -The pyramid above shows the relative number of unit tests, integration tests, -and acceptance tests. Most of our tests are unit tests or integration tests. +* We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) to +write end-user acceptance tests directly in Python, using the framework to +maximize reliability and maintainability. ## Test Locations @@ -65,9 +69,10 @@ and test. For example, the test for `src/views/module.coffee` should be written in `spec/views/module_spec.coffee`. * UI acceptance tests: - - Set up and helper methods: `common/djangoapps/terrain` - - Tests: located in `features` subpackage within a Django app. + - Set up and helper methods, and stubs for external services: `common/djangoapps/terrain` + - Lettuce Tests: located in `features` subpackage within a Django app. For example: `lms/djangoapps/courseware/features` + - Bok Choy Tests: Artifacts are located under `common/test/acceptance` ## Factories @@ -199,75 +204,56 @@ To run JavaScript tests in your default browser: These rake commands call through to a custom test runner. For more info, see [js-test-tool](https://github.com/edx/js-test-tool). -### Running Acceptance Tests (Bok Choy) +### Running Bok Choy Acceptance Tests We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) for acceptance testing. Bok Choy is a UI-level acceptance test framework for writing robust [Selenium](http://docs.seleniumhq.org/) tests in [Python](https://www.python.org/). Bok Choy makes your acceptance tests reliable and maintainable by utilizing the Page Object and Promise design patterns. -**Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver) installed to run the tests in Chrome. The tests are confirmed to run with Chrome (not Chromium) version 28.0.1500.71 with ChromeDriver version 2.1.210398. +**Prerequisites**: +* These prerequisites are all automatically installed and available in [Devstack](https://github.com/edx/configuration/wiki/edX-Developer-Stack), +the supported development enviornment for the edX Platform. +* Chromedriver and Chrome (see Running Lettuce Acceptance Tests below for the latest tested versions) +* Mongo +* Memcache +* mySQL -To run all the acceptance tests: + +To run all the bok choy acceptance tests: rake test:bok_choy -To run tests faster by not collecting static files, you can use: +Once the database has been set up and the static files collected, you can use the 'fast' +option to skip those tasks. This option can also be used with any of the test specs below: rake test:bok_choy:fast -To run single test: +To run single test, specify the name of the test file. For example: - rake test:bok_choy[TEST_FILE_NAME] - example rake test:bok_choy[test_lms.py] -To run single test faster by not collecting static files: +To run single test faster by not repeating setup tasks: - rake test:bok_choy:fast[TEST_FILE_NAME] rake test:bok_choy:fast[test_lms.py] -To test only specific feature: +To test only a certain feature, specify the file and the testcase class: - rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME] - example - rake test:bok_choy[test_lms.py:RegistrationTest] - -To test only specific feature faster by not collecting static files: - - rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME] - example rake test:bok_choy:fast[test_lms.py:RegistrationTest] -To test only specific scenario: +To execute only a certain test case, specify the file name, class, and +test case method: - rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME.FUNCTION_NAME] - example - rake test:bok_choy[test_lms.py:RegistrationTest.test_register] - -To test only specific scenario faster by not collecting static files: - - rake test:bok_choy:fast[TEST_FILE_NAME:CLASS_NAME.FUNCTION_NAME] - example rake test:bok_choy:fast[test_lms.py:RegistrationTest.test_register] -During acceptance test execution, log files are written to - - test_root/log/bok_choy_lms.log - test_root/log/bok_choy_ora.log - test_root/log/bok_choy_studio.log - test_root/log/bok_choy_xqueue.log - test_root/log/bok_choy_comments.log - -Failed test's screenshots are saved to: - - test_root/log +During acceptance test execution, log files and also screenshots of failed tests +are captured in test_root/log. To put a debugging breakpoint in a test use: from nose.tools import set_trace; set_trace() -### Running Acceptance Tests +### Running Lettuce Acceptance Tests We use [Lettuce](http://lettuce.it/) for acceptance testing. Most of our tests use [Splinter](http://splinter.cobrateam.info/) @@ -276,8 +262,7 @@ uses [Selenium](http://docs.seleniumhq.org/) to control the Chrome browser. **Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver) installed to run the tests in Chrome. The tests are confirmed to run -with Chrome (not Chromium) version 28.0.1500.71 with ChromeDriver -version 2.1.210398. +with Chrome (not Chromium) version 34.0.1847.116 with ChromeDriver version 2.6.232917. To run all the acceptance tests: rake test:acceptance