diff --git a/doc/development.md b/doc/development.md index e89d8b19af..95cc32329c 100644 --- a/doc/development.md +++ b/doc/development.md @@ -12,7 +12,6 @@ This will read the `Gemfile` and install all of the gems specified there. Run the following:: pip install -r requirements.txt - pip install -r test-requirements.txt ### Binaries @@ -52,77 +51,13 @@ or with additional options: *N.B.* You may have to escape the `[` characters, depending on your shell: `rake "lms[test,5000]"` -## Running tests - -### Python Tests - -This runs all the tests (long, uses collectstatic): - - rake test - -If if you aren't changing static files, can run `rake test` once, then run - - rake fasttest_lms - -or - - rake fasttest_cms - -xmodule can be tested independently, with this: - - rake test_common/lib/xmodule - -To run a single django test class: - - django-admin.py test --settings=lms.envs.test --pythonpath=. lms/djangoapps/courseware/tests/tests.py:TestViewAuth - -To run a single django test: - - django-admin.py test --settings=lms.envs.test --pythonpath=. lms/djangoapps/courseware/tests/tests.py:TestViewAuth.test_dark_launch - - -To run a single nose test file: - - nosetests common/lib/xmodule/xmodule/tests/test_stringify.py - -To run a single nose test: - - nosetests common/lib/xmodule/xmodule/tests/test_stringify.py:test_stringify - - -Very handy: if you uncomment the `pdb=1` line in `setup.cfg`, it will drop you into pdb on error. This lets you go up and down the stack and see what the values of the variables are. Check out http://docs.python.org/library/pdb.html - - -### Javascript Tests - -These commands start a development server with jasmine testing enabled, and launch your default browser -pointing to those tests - - rake browse_jasmine_{lms,cms} - -To run the tests headless, you must install phantomjs (http://phantomjs.org/download.html). - - rake phantomjs_jasmine_{lms,cms} - -If the `phantomjs` binary is not on the path, set the `PHANTOMJS_PATH` environment variable to point to it - - PHANTOMJS_PATH=/path/to/phantomjs rake phantomjs_jasmine_{lms,cms} - - -## Getting More Information - -Run the following to see a list of all rake tasks available and their arguments +To get a full list of available rake tasks, use: rake -T -## Testing using queue servers - -When testing problems that use a queue server on AWS (e.g. sandbox-xqueue.edx.org), you'll need to run your server on your public IP, like so. - -`django-admin.py runserver --settings=lms.envs.dev --pythonpath=. 0.0.0.0:8000` - -When you connect to the LMS, you need to use the public ip. Use `ifconfig` to figure out the numnber, and connect e.g. to `http://18.3.4.5:8000/` +## Running Tests +See `testing.md` for instructions on running the test suite. ## Content development diff --git a/doc/test_pyramid.png b/doc/test_pyramid.png new file mode 100644 index 0000000000..5c0eea5762 Binary files /dev/null and b/doc/test_pyramid.png differ diff --git a/doc/testing.md b/doc/testing.md index 694a9e8231..84175fee3d 100644 --- a/doc/testing.md +++ b/doc/testing.md @@ -1,68 +1,218 @@ # Testing -Testing is good. Here is some useful info about how we set up tests. -More info is [on the wiki](https://edx-wiki.atlassian.net/wiki/display/ENG/Test+Engineering) +## Overview -## Backend code +We maintain three kinds of tests: unit tests, integration tests, +and acceptance tests. -- The python unit tests can be run via rake tasks. -See development.md for more info on how to do this. +### Unit Tests -## Frontend code +* Each test case should be concise: setup, execute, check, and teardown. +If you find yourself writing tests with many steps, consider refactoring +the unit under tests into smaller units, and then testing those individually. -### Jasmine +* As a rule of thumb, your unit tests should cover every code branch. -We're using Jasmine to unit/integration test the JavaScript files. -More info [on the wiki](https://edx-wiki.atlassian.net/wiki/display/ENG/Jasmine) +* Mock or patch external dependencies. +We use [voidspace mock](http://www.voidspace.org.uk/python/mock/). -All the specs are written in CoffeeScript to be consistent with the code. -To access the test cases, start the server using the settings file **jasmine.py** using this command: - `rake django-admin[runserver,lms,jasmine,12345]` +* We unit test Python code (using [unittest](http://docs.python.org/2/library/unittest.html)) and +Javascript (using [Jasmine](http://pivotal.github.io/jasmine/)) -Then navigate to `http://localhost:12345/_jasmine/` to see the test results. +### Integration Tests +* Test several units at the same time. +Note that you can still mock or patch dependencies +that are not under test! For example, you might test that +`LoncapaProblem`, `NumericalResponse`, and `CorrectMap` in the +`capa` package work together, while still mocking out template rendering. -All the JavaScript codes must have test coverage. Both CMS and LMS -has its own test directory in `{cms,lms}/static/coffee/spec` If you haven't -written a JavaScript test before, you can look at those example files as a -starting point. Also, these materials might be helpful for you: +* Use integration tests to ensure that units are hooked up correctly. +You do not need to test every possible input--that's what unit +tests are for. Instead, focus on testing the "happy path" +to verify that the components work together correctly. -CMS Note: For consistency, you're advised to use the same directory structure -for implementation and test. For example, test for `src/views/module.coffee` +* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/) to simulate +HTTP requests to the server. + +### UI Acceptance Tests +* Use these to test that major program features are working correctly. + +* We use [lettuce](http://lettuce.it/) to write BDD-style tests. Most of +these tests simulate user interactions through the browser using +[splinter](http://splinter.cobrateam.info/). + +Overall, you want to write the tests that **maximize coverage** +while **minimizing maintenance**. +In practice, this usually means investing heavily +in unit tests, which tend to be the most robust to changes in the code base. + +![Test Pyramid](test_pyramid.png) + +The pyramid above shows the relative number of unit tests, integration tests, +and acceptance tests. Most of our tests are unit tests or integration tests. + +## Test Locations + +* Python unit and integration tests: Located in +subpackages called `tests`. +For example, the tests for the `capa` package are located in +`common/lib/capa/capa/tests`. + +* Javascript unit tests: Located in `spec` folders. For example, +`common/lib/xmodule/xmodule/js/spec` and `{cms,lms}/static/coffee/spec` +For consistency, you should use the same directory structure for implementation +and test. For example, the test for `src/views/module.coffee` should be written in `spec/views/module_spec.coffee`. -* http://pivotal.github.com/jasmine -* http://railscasts.com/episodes/261-testing-javascript-with-jasmine?view=asciicast -* http://a-developer-life.blogspot.com/2011/05/jasmine-part-1-unit-testing-javascript.html +* UI acceptance tests: + - Set up and helper methods: `common/djangoapps/terrain` + - Tests: located in `features` subpackage within a Django app. + For example: `lms/djangoapps/courseware/features` -If you're finishing a feature that contains JavaScript code snippets and do not -sure how to test, please feel free to open up a pull request and asking people -for help. (However, the best way to do it would be writing your test first, then -implement your feature - Test Driven Development.) -### BDD style acceptance tests with Lettuce +## Factories -We're using Lettuce for end user acceptance testing of features. -More info [on the wiki](https://edx-wiki.atlassian.net/wiki/display/ENG/Lettuce+Acceptance+Testing) +Many tests delegate set-up to a "factory" class. For example, +there are factories for creating courses, problems, and users. +This encapsulates set-up logic from tests. -Lettuce is a port of Cucumber. We're using it to drive Splinter, which is a python wrapper to Selenium. -To execute the automated test scripts, you'll need to start up the django server separately, then launch the tests. -Do both use the settings file named **acceptance.py**. +Factories are often implemented using [FactoryBoy](https://readthedocs.org/projects/factoryboy/) -What this will do is to use a sqllite database named mitx_all/db/test_mitx.db. -That way it can be flushed etc. without messing up your dev db. -Note that this also means that you need to syncdb and migrate the db first before starting the server to initialize it if it does not yet exist. +In general, factories should be located close to the code they use. +For example, the factory for creating problem XML definitions + is located in `common/lib/capa/capa/tests/response_xml_factory.py` +because the `capa` package handles problem XML. + + +# Running Tests + +Before running tests, ensure that you have all the dependencies. You can install dependencies using: + + pip install -r requirements.txt + + +## Running Python Unit tests + +We use [nose](https://nose.readthedocs.org/en/latest/) through +the [django-nose plugin](https://pypi.python.org/pypi/django-nose) +to run the test suite. + +You can run tests using `rake` commands. For example, + + rake test + +runs all the tests. It also runs `collectstatic`, which prepares the static files used by the site (for example, compiling Coffeescript to Javascript). + +You can also run the tests without `collectstatic`, which tends to be faster: + + rake fasttest_lms + +or + + rake fasttest_cms + +xmodule can be tested independently, with this: + + rake test_common/lib/xmodule + +To run a single django test class: + + django-admin.py test --settings=lms.envs.test --pythonpath=. lms/djangoapps/courseware/tests/tests.py:TestViewAuth + +To run a single django test: + + django-admin.py test --settings=lms.envs.test --pythonpath=. lms/djangoapps/courseware/tests/tests.py:TestViewAuth.test_dark_launch + + +To run a single nose test file: + + nosetests common/lib/xmodule/xmodule/tests/test_stringify.py + +To run a single nose test: + + nosetests common/lib/xmodule/xmodule/tests/test_stringify.py:test_stringify + + +Very handy: if you uncomment the `pdb=1` line in `setup.cfg`, it will drop you into pdb on error. This lets you go up and down the stack and see what the values of the variables are. Check out [the pdb documentation](http://docs.python.org/library/pdb.html) + +### Running Javascript Unit Tests + +These commands start a development server with jasmine testing enabled, and launch your default browser +pointing to those tests + + rake browse_jasmine_{lms,cms} + +To run the tests headless, you must install [phantomjs](http://phantomjs.org/download.html), then run: + + rake phantomjs_jasmine_{lms,cms} + +If the `phantomjs` binary is not on the path, set the `PHANTOMJS_PATH` environment variable to point to it + + PHANTOMJS_PATH=/path/to/phantomjs rake phantomjs_jasmine_{lms,cms} + +Once you have run the `rake` command, your browser should open to +to `http://localhost/_jasmine/`, which displays the test results. + +**Troubleshooting**: If you get an error message while running the `rake` task, +try running `bundle install` to install the required ruby gems. + +### Running Acceptance Tests + +We use [Lettuce](http://lettuce.it/) for acceptance testing. +Most of our tests use [Splinter](http://splinter.cobrateam.info/) +to simulate UI browser interactions. Splinter, in turn, +uses [Selenium](http://docs.seleniumhq.org/) to control the browser. + +**Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver) +installed to run the tests in Chrome. + +Before running the tests, you need to set up the test database: -1. Set up the test database (only needs to be done once): rm ../db/test_mitx.db rake django-admin[syncdb,lms,acceptance,--noinput] rake django-admin[migrate,lms,acceptance,--noinput] -2. Start up the django server separately in a shell - rake lms[acceptance] +To run the acceptance tests: -3. Then in another shell, run the tests in different ways as below. Lettuce comes with a new django-admin command called _harvest_. See the [lettuce django docs](http://lettuce.it/recipes/django-lxml.html) for more details. -* All tests in a specified feature folder: `django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/portal/features/` -* Only the specified feature's scenarios: `django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/courseware/features/high-level-tabs.feature` +1. Start the Django server locally using the settings in **acceptance.py**: -4. Troubleshooting -* If you get an error msg that says something about harvest not being a command, you probably are missing a requirement. Pip install (test-requirements.txt) and/or brew install as needed. \ No newline at end of file + rake lms[acceptance] + +2. In another shell, run the tests: + + django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/portal/features/ + +To test only a specific feature: + + django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/courseware/features/high-level-tabs.feature + +**Troubleshooting**: If you get an error message that says something about harvest not being a command, you probably are missing a requirement. +Try running: + + pip install -r requirements.txt + + +## Viewing Test Coverage + +We currently collect test coverage information for Python unit/integration tests. + +To view test coverage: + +1. Run the test suite: + + rake test + +2. Generate reports: + + rake coverage:html + +3. HTML reports are located in the `reports` folder. + + +## Testing using queue servers + +When testing problems that use a queue server on AWS (e.g. sandbox-xqueue.edx.org), you'll need to run your server on your public IP, like so. + +`django-admin.py runserver --settings=lms.envs.dev --pythonpath=. 0.0.0.0:8000` + +When you connect to the LMS, you need to use the public ip. Use `ifconfig` to figure out the number, and connect e.g. to `http://18.3.4.5:8000/`