diff --git a/AUTHORS b/AUTHORS index 89fc2d959b..c700eab277 100644 --- a/AUTHORS +++ b/AUTHORS @@ -82,3 +82,5 @@ Adam Palay Ian Hoover Mukul Goyal Robert Marks +Yarko Tymciurak + diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 468db0607c..68308980ad 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -5,6 +5,16 @@ These are notable changes in edx-platform. This is a rolling list of changes, in roughly chronological order, most recent first. Add your entries at or near the top. Include a label indicating the component affected. +LMS: Added user preferences (arbitrary user/key/value tuples, for which +which user/key is unique) and a REST API for reading users and +preferences. Access to the REST API is restricted by use of the +X-Edx-Api-Key HTTP header (which must match settings.EDX_API_KEY; if +the setting is not present, the API is disabled). + +LMS: Added endpoints for AJAX requests to enable/disable notifications +(which are not yet implemented) and a one-click unsubscribe page. + +Common: Add a manage.py that knows about edx-platform specific settings and projects Common: Added *experimental* support for jsinput type. diff --git a/README.md b/README.md index 2208fe1cad..4ab07b3550 100644 --- a/README.md +++ b/README.md @@ -12,30 +12,35 @@ installation process. 1. Make sure you have plenty of available disk space, >5GB 2. Install Git: http://git-scm.com/downloads -3. Install VirtualBox: https://www.virtualbox.org/wiki/Download_Old_Builds_4_2 - (you need version 4.2.12, as later/earlier versions might not work well with - Vagrant) +3. Install VirtualBox: https://www.virtualbox.org/wiki/Downloads + See http://docs.vagrantup.com/v2/providers/index.html for a list of supported + Providers. You should use VirtualBox >= 4.2.12. + (Windows: later/earlier VirtualBox versions than 4.2.12 have been reported to not work well with + Vagrant. If this is still a problem, you can + install 4.2.12 from https://www.virtualbox.org/wiki/Download_Old_Builds_4_2). 4. Install Vagrant: http://www.vagrantup.com/ (Vagrant 1.2.2 or later) 5. Open a terminal 6. Download the project: `git clone git://github.com/edx/edx-platform.git` 7. Enter the project directory: `cd edx-platform/` -8. (Windows only) Run the commands to +8. (Windows only) Run the commands to [deal with line endings and symlinks under Windows](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#dealing-with-line-endings-and-symlinks-under-windows) -9. Start: `vagrant up` +9. Create the development environment and start it: `vagrant up` -The last step might require your host machine's administrator password to setup NFS. +The initial `vagrant up` will download a Linux image, then boot and ask for your +host machine's administrator password to setup file sharing between your computer and the VM. +Once file sharing is established, `edx-platform/scripts/create-dev-env.sh` will +install dependencies and configure the VM. +This will take a while; go grab a coffee. -Afterwards, it will download an image, install all the dependencies and configure -the VM. It will take a while, go grab a coffee. +When complete, you should see a _"Success!"_ message. +If not, refer to the +[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting). -Once completed, hopefully you should see a "Success!" message indicating that the -installation went fine. (If not, refer to the -[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).) +Your development environment is initialized only on the first bring-up. +Subsequently `vagrant up` commands will boot your virtual machine normally. -Note: by default, the VM will get the IP `192.168.20.40`. If you need to use a -different IP, you can edit the file `Vagrantfile`. If you have already started the -VM with `vagrant up`, see "Stopping and restarting the VM" below to take the change -into account. +Note: by default, the VM will get the IP `192.168.20.40`. +You can change this in your `Vagrantfile` (the startup message will reflect your VM's actual IP). Accessing the VM ---------------- @@ -46,15 +51,24 @@ Once the installation is finished, to log into the virtual machine: $ vagrant ssh ``` -Note: This won't work from Windows, install install PuTTY from -http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html instead. Then -connect to 127.0.0.1, port 2222, using vagrant/vagrant as a user/password. +Note: This won't work from Windows. Instead, install PuTTY from +http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html. Then +connect to 192.168.20.40, port 2222, using vagrant/vagrant as a user/password. + Using edX --------- -Once inside the VM, you can start Studio and LMS with the following commands -(from the `/opt/edx/edx-platform` folder): +When you login to your VM, you are in +`/opt/edx/edx-platform` by default, which is shared from your host workspace. +Your host computer contains the edx-project development code and repository. +Your VM runs edx-platform code mounted from your host, so +you can develop by editing on your host. + +After logging into your VM with `vagrant ssh`, +start the _Studio_ and +_Learning management system (LMS)_ +servers (run these from `/opt/edx/edx-platform`): Learning management system (LMS): @@ -62,46 +76,85 @@ Learning management system (LMS): $ rake lms[cms.dev,0.0.0.0:8000] ``` -Studio: +Studio (CMS): ``` $ rake cms[dev,0.0.0.0:8001] ``` -Once started, open the following URLs in your browser: +The servers will come up to these URLs: -* Learning management system (LMS): http://192.168.20.40:8000/ -* Studio (CMS): http://192.168.20.40:8001/ +- LMS: http://192.168.20.40:8000/ +- CMS: http://192.168.20.40:8001/ -You can develop by editing the files directly in the `edx-platform/` directory you -downloaded before, you don't need to connect to the VM to edit them (the VM uses -those files to run edX, mirroring the folder in `/opt/edx/edx-platform`). +Your VM's port 8000 is forwarded to host port 9000 +so you can also access the LMS with [http://localhost:9000/](). +Similarly, VM port 8001 is forwarded to host port 9001. +These are set in your `Vagrantfile`. -You may also want to create a super-user with: - -``` -$ rake django-admin["createsuperuser"] -``` - -Also note that if you register a new user through the web interface, -the activiation email will be posted to your VM's terminal window (search for -lines similar to): +Note that when you register a new user through the web interface, +by default the activiation email will be appear on your VM's terminal. +Search for lines similar to: ``` Subject: Your account for edX Studio From: registration@edx.org ``` -and find the activation URL for the account you've created. +and find the activation URL. See the [Frequently Asked Questions](https://github.com/edx/edx-platform/wiki/Frequently-Asked-Questions) for more usage tips. +Django admin & debug toolbar +----------------------------- + +You can enable admin logins and the debug_toolbar by editing +`lms/envs/common.py`: + +- enable ADMIN login page by setting: + - ``` + 'ENABLE_DJANGO_ADMIN_SITE': True +``` + + +- enable debug toolbar by uncommenting: + - ``` + # 'debug_toolbar.middleware.DebugToolbarMiddleware', +``` + +These are also defined in `lms/envs/dev.py`, +and usually active on localhost. + +To get at your VM's 127.0.0.1, explicitly forward one of VM's available localhost ports to your computer. +Instead of `vagrant ssh`, login with: + +``` +$ ssh -L 6080:127.0.0.1:8080 vagrant@192.168.20.40 +``` + +The password is _vagrant_. + +From your VM, start the LMS as a localhost instance: + +``` +$ rake lms[cms.dev,127.0.0.1:8080] +``` + +You should see the debug toolbar now on [http:/localhost:6080/](). +You should now also see a login on [http://localhost:6080/admin/]() +You will need a privileged user for the admin login. +You can create a CMS/LMS super-user with: +``` +$ ./manage.py lms createsuperuser +``` + + Stopping & starting ------------------- -To stop the VM (from your `edx-platform/` directory): +To stop the VM (from your `edx-platform/` directory): ``` $ vagrant halt ``` @@ -112,16 +165,27 @@ To restart: $ vagrant up ``` -or, to start without attempting to update the dependencies: +To suspend and resume tasks in progress on your VM: +``` +$ vagrant suspend +$ # and later... +$ vagrant resume +``` +Your development environment is normally created once, on first `vagrant up`. +You can continue to fetch changes in edx-platform +as you work with your VM. +To re-create your VM and create a fresh development environment: ``` -$ vagrant up --no-provision +$ vagrant destroy +$ vagrant up # will make a new VM ``` + Troubleshooting --------------- -If anything doesn't work as expected, see the +If anything doesn't work as expected, see the [troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting). Installation - Advanced @@ -229,23 +293,12 @@ or any other process management tool. Configuring Your Project ------------------------ -We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our -project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T` -to view a summary. - Before you run your project, you need to create a sqlite database, create -tables in that database, run database migrations, and populate templates for -CMS templates. Fortunately, `rake` will do all of this for you! Just run: +tables in that database, and run database migrations. Fortunately, `django` +will do all of this for you - $ rake django-admin[syncdb] - $ rake django-admin[migrate] - -If you are running these commands using the [`zsh`](http://www.zsh.org/) shell, -zsh will assume that you are doing -[shell globbing](https://en.wikipedia.org/wiki/Glob_%28programming%29), search for -a file in your directory named `django-adminsyncdb` or `django-adminmigrate`, -and fail. To fix this, just surround the argument with quotation marks, so that -you're running `rake "django-admin[syncdb]"`. + $ ./manage.py lms syncdb --migrate + $ ./manage.py cms syncdb --migrate Run Your Project ---------------- @@ -253,6 +306,10 @@ edX has two components: Studio, the course authoring system; and the LMS (learning management system) used by students. These two systems communicate through the MongoDB database, which stores course information. +We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our +project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T` +to view a summary. + To run Studio, run: $ rake cms diff --git a/cms/djangoapps/contentstore/features/checklists.feature b/cms/djangoapps/contentstore/features/checklists.feature index 3767144c99..10db23c4fa 100644 --- a/cms/djangoapps/contentstore/features/checklists.feature +++ b/cms/djangoapps/contentstore/features/checklists.feature @@ -8,7 +8,7 @@ Feature: Course checklists Scenario: A course author can mark tasks as complete Given I have opened Checklists Then I can check and uncheck tasks in a checklist - And They are correctly selected after I reload the page + And They are correctly selected after reloading the page Scenario: A task can link to a location within Studio Given I have opened Checklists diff --git a/cms/djangoapps/contentstore/features/checklists.py b/cms/djangoapps/contentstore/features/checklists.py index 13d3ca99b7..e8dcd755a3 100644 --- a/cms/djangoapps/contentstore/features/checklists.py +++ b/cms/djangoapps/contentstore/features/checklists.py @@ -45,7 +45,7 @@ def i_can_check_and_uncheck_tasks(step): verifyChecklist2Status(2, 7, 29) -@step('They are correctly selected after I reload the page$') +@step('They are correctly selected after reloading the page$') def tasks_correctly_selected_after_reload(step): reload_the_page(step) verifyChecklist2Status(2, 7, 29) diff --git a/cms/djangoapps/contentstore/features/common.py b/cms/djangoapps/contentstore/features/common.py index 756adad7c4..d357c8ae96 100644 --- a/cms/djangoapps/contentstore/features/common.py +++ b/cms/djangoapps/contentstore/features/common.py @@ -209,7 +209,8 @@ def i_created_a_video_component(step): world.create_component_instance( step, '.large-video-icon', 'video', - '.xmodule_VideoModule' + '.xmodule_VideoModule', + has_multiple_templates=False ) @@ -238,6 +239,17 @@ def save_button_disabled(step): assert world.css_has_class(button_css, disabled) +@step('I confirm the prompt') +def confirm_the_prompt(step): + prompt_css = 'a.button.action-primary' + world.css_click(prompt_css) + + +@step(u'I am shown a (.*)$') +def i_am_shown_a_notification(step, notification_type): + assert world.is_css_present('.wrapper-%s' % notification_type) + + def type_in_codemirror(index, text): world.css_click(".CodeMirror", index=index) g = world.css_find("div.CodeMirror.CodeMirror-focused > div > textarea") diff --git a/cms/djangoapps/contentstore/features/component.feature b/cms/djangoapps/contentstore/features/component.feature index 2291712f2d..a30ce96ae6 100644 --- a/cms/djangoapps/contentstore/features/component.feature +++ b/cms/djangoapps/contentstore/features/component.feature @@ -67,3 +67,21 @@ Feature: Component Adding When I will confirm all alerts And I delete all components Then I see no components + + Scenario: I see a prompt on delete + Given I have opened a new course in studio + And I am editing a new unit + And I add the following components: + | Component | + | Discussion | + And I delete a component + Then I am shown a prompt + + Scenario: I see a notification on save + Given I have opened a new course in studio + And I am editing a new unit + And I add the following components: + | Component | + | Discussion | + And I edit and save a component + Then I am shown a notification diff --git a/cms/djangoapps/contentstore/features/component.py b/cms/djangoapps/contentstore/features/component.py index 217ad84591..15727dd992 100644 --- a/cms/djangoapps/contentstore/features/component.py +++ b/cms/djangoapps/contentstore/features/component.py @@ -41,6 +41,17 @@ def see_no_components(steps): assert world.is_css_not_present('li.component') +@step(u'I delete a component') +def delete_one_component(step): + world.css_click('a.delete-button') + + +@step(u'I edit and save a component') +def edit_and_save_component(step): + world.css_click('.edit-button') + world.css_click('.save-button') + + def step_selector_list(data_type, path, index=1): selector_list = ['a[data-type="{}"]'.format(data_type)] if index != 1: diff --git a/cms/djangoapps/contentstore/features/component_settings_editor_helpers.py b/cms/djangoapps/contentstore/features/component_settings_editor_helpers.py index 2f1788c6a4..2b206e4466 100644 --- a/cms/djangoapps/contentstore/features/component_settings_editor_helpers.py +++ b/cms/djangoapps/contentstore/features/component_settings_editor_helpers.py @@ -7,10 +7,16 @@ from terrain.steps import reload_the_page @world.absorb -def create_component_instance(step, component_button_css, category, expected_css, boilerplate=None): - click_new_component_button(step, component_button_css) - click_component_from_menu(category, boilerplate, expected_css) +def create_component_instance(step, component_button_css, category, + expected_css, boilerplate=None, + has_multiple_templates=True): + click_new_component_button(step, component_button_css) + + if has_multiple_templates: + click_component_from_menu(category, boilerplate, expected_css) + + assert_equal(1, len(world.css_find(expected_css))) @world.absorb def click_new_component_button(step, component_button_css): @@ -34,7 +40,6 @@ def click_component_from_menu(category, boilerplate, expected_css): elements = world.css_find(elem_css) assert_equal(len(elements), 1) world.css_click(elem_css) - assert_equal(1, len(world.css_find(expected_css))) @world.absorb diff --git a/cms/djangoapps/contentstore/features/studio-overview-togglesection.feature b/cms/djangoapps/contentstore/features/course-overview.feature similarity index 86% rename from cms/djangoapps/contentstore/features/studio-overview-togglesection.feature rename to cms/djangoapps/contentstore/features/course-overview.feature index e746f3629a..b3041b9b18 100644 --- a/cms/djangoapps/contentstore/features/studio-overview-togglesection.feature +++ b/cms/djangoapps/contentstore/features/course-overview.feature @@ -1,7 +1,7 @@ -Feature: Overview Toggle Section - In order to quickly view the details of a course's section or to scan the inventory of sections +Feature: Course Overview + In order to quickly view the details of a course's section and set release dates and grading As a course author - I want to toggle the visibility of each section's subsection details in the overview listing + I want to use the course overview page Scenario: The default layout for the overview page is to show sections in expanded view Given I have a course with multiple sections @@ -57,3 +57,9 @@ Feature: Overview Toggle Section And I click the "Expand All Sections" link Then I see the "Collapse All Sections" link And all sections are expanded + + Scenario: Notification is shown on grading status changes + Given I have a course with 1 section + When I navigate to the course overview page + And I change an assignment's grading status + Then I am shown a notification diff --git a/cms/djangoapps/contentstore/features/studio-overview-togglesection.py b/cms/djangoapps/contentstore/features/course-overview.py similarity index 95% rename from cms/djangoapps/contentstore/features/studio-overview-togglesection.py rename to cms/djangoapps/contentstore/features/course-overview.py index 41e39513ea..10fa6453b2 100644 --- a/cms/djangoapps/contentstore/features/studio-overview-togglesection.py +++ b/cms/djangoapps/contentstore/features/course-overview.py @@ -118,3 +118,9 @@ def all_sections_are_collapsed(step): subsections = world.css_find(subsection_locator) for index in range(len(subsections)): assert_false(world.css_visible(subsection_locator, index=index)) + + +@step(u"I change an assignment's grading status") +def change_grading_status(step): + world.css_find('a.menu-toggle').click() + world.css_find('.menu li').first.click() diff --git a/cms/djangoapps/contentstore/features/discussion-editor.py b/cms/djangoapps/contentstore/features/discussion-editor.py index 8e4becb62e..13927a7d89 100644 --- a/cms/djangoapps/contentstore/features/discussion-editor.py +++ b/cms/djangoapps/contentstore/features/discussion-editor.py @@ -9,7 +9,8 @@ def i_created_discussion_tag(step): world.create_component_instance( step, '.large-discussion-icon', 'discussion', - '.xmodule_DiscussionModule' + '.xmodule_DiscussionModule', + has_multiple_templates=False ) diff --git a/cms/djangoapps/contentstore/features/html-editor.py b/cms/djangoapps/contentstore/features/html-editor.py index b03388c89a..c3e0afa480 100644 --- a/cms/djangoapps/contentstore/features/html-editor.py +++ b/cms/djangoapps/contentstore/features/html-editor.py @@ -14,4 +14,4 @@ def i_created_blank_html_page(step): @step('I see only the HTML display name setting$') def i_see_only_the_html_display_name(step): - world.verify_all_setting_entries([['Display Name', "Blank HTML Page", False]]) + world.verify_all_setting_entries([['Display Name', "Text", False]]) diff --git a/cms/djangoapps/contentstore/features/problem-editor.py b/cms/djangoapps/contentstore/features/problem-editor.py index 64b2ec9b5c..565a35f802 100644 --- a/cms/djangoapps/contentstore/features/problem-editor.py +++ b/cms/djangoapps/contentstore/features/problem-editor.py @@ -170,7 +170,8 @@ def edit_latex_source(step): @step('my change to the High Level Source is persisted') def high_level_source_persisted(step): def verify_text(driver): - return world.css_text('.problem') == 'hi' + css_sel = '.problem div>span' + return world.css_text(css_sel) == 'hi' world.wait_for(verify_text) diff --git a/cms/djangoapps/contentstore/features/section.feature b/cms/djangoapps/contentstore/features/section.feature index 84a9bb991d..a08b490c6d 100644 --- a/cms/djangoapps/contentstore/features/section.feature +++ b/cms/djangoapps/contentstore/features/section.feature @@ -33,4 +33,5 @@ Feature: Create Section And I have added a new section When I will confirm all alerts And I press the "section" delete icon + And I confirm the prompt Then the section does not exist diff --git a/cms/djangoapps/contentstore/features/subsection.feature b/cms/djangoapps/contentstore/features/subsection.feature index a11467e3f9..9f5793dbe7 100644 --- a/cms/djangoapps/contentstore/features/subsection.feature +++ b/cms/djangoapps/contentstore/features/subsection.feature @@ -38,4 +38,5 @@ Feature: Create Subsection And I see my subsection on the Courseware page When I will confirm all alerts And I press the "subsection" delete icon + And I confirm the prompt Then the subsection does not exist diff --git a/cms/djangoapps/contentstore/features/video-editor.py b/cms/djangoapps/contentstore/features/video-editor.py index e0f76b30ad..93d638e621 100644 --- a/cms/djangoapps/contentstore/features/video-editor.py +++ b/cms/djangoapps/contentstore/features/video-editor.py @@ -7,7 +7,7 @@ from lettuce import world, step @step('I see the correct settings and default values$') def i_see_the_correct_settings_and_values(step): world.verify_all_setting_entries([['Default Speed', 'OEoXaMPEzfM', False], - ['Display Name', 'Video Title', False], + ['Display Name', 'Video', False], ['Download Track', '', False], ['Download Video', '', False], ['Show Captions', 'True', False], diff --git a/cms/djangoapps/contentstore/tests/test_contentstore.py b/cms/djangoapps/contentstore/tests/test_contentstore.py index ce7e886220..a51110163d 100644 --- a/cms/djangoapps/contentstore/tests/test_contentstore.py +++ b/cms/djangoapps/contentstore/tests/test_contentstore.py @@ -312,6 +312,23 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase): self.assertGreater(len(course.textbooks), 0) + def test_default_tabs_on_create_course(self): + module_store = modulestore('direct') + CourseFactory.create(org='edX', course='999', display_name='Robot Super Course') + course_location = Location(['i4x', 'edX', '999', 'course', 'Robot_Super_Course', None]) + + course = module_store.get_item(course_location) + + expected_tabs = [] + expected_tabs.append({u'type': u'courseware'}) + expected_tabs.append({u'type': u'course_info', u'name': u'Course Info'}) + expected_tabs.append({u'type': u'textbooks'}) + expected_tabs.append({u'type': u'discussion', u'name': u'Discussion'}) + expected_tabs.append({u'type': u'wiki', u'name': u'Wiki'}) + expected_tabs.append({u'type': u'progress', u'name': u'Progress'}) + + self.assertEqual(course.tabs, expected_tabs) + def test_static_tab_reordering(self): module_store = modulestore('direct') CourseFactory.create(org='edX', course='999', display_name='Robot Super Course') diff --git a/cms/djangoapps/contentstore/tests/test_course_updates.py b/cms/djangoapps/contentstore/tests/test_course_updates.py index 4f92806871..30114496c8 100644 --- a/cms/djangoapps/contentstore/tests/test_course_updates.py +++ b/cms/djangoapps/contentstore/tests/test_course_updates.py @@ -36,8 +36,11 @@ class CourseUpdateTest(CourseTestCase): 'provided_id': payload['id']}) content += '
div

p

' payload['content'] = content + # POST requests were coming in w/ these header values causing an error; so, repro error here resp = self.client.post(first_update_url, json.dumps(payload), - "application/json") + "application/json", + HTTP_X_HTTP_METHOD_OVERRIDE="PUT", + REQUEST_METHOD="POST") self.assertHTMLEqual(content, json.loads(resp.content)['content'], "iframe w/ div") diff --git a/cms/djangoapps/contentstore/tests/test_crud.py b/cms/djangoapps/contentstore/tests/test_crud.py new file mode 100644 index 0000000000..84643f7787 --- /dev/null +++ b/cms/djangoapps/contentstore/tests/test_crud.py @@ -0,0 +1,186 @@ +''' +Created on May 7, 2013 + +@author: dmitchell +''' +import unittest +from xmodule import templates +from xmodule.modulestore.tests import persistent_factories +from xmodule.course_module import CourseDescriptor +from xmodule.modulestore.django import modulestore +from xmodule.seq_module import SequenceDescriptor +from xmodule.x_module import XModuleDescriptor +from xmodule.capa_module import CapaDescriptor +from xmodule.modulestore.locator import CourseLocator, BlockUsageLocator +from xmodule.modulestore.exceptions import ItemNotFoundError +from xmodule.html_module import HtmlDescriptor + + +class TemplateTests(unittest.TestCase): + """ + Test finding and using the templates (boilerplates) for xblocks. + """ + + def test_get_templates(self): + found = templates.all_templates() + self.assertIsNotNone(found.get('course')) + self.assertIsNotNone(found.get('about')) + self.assertIsNotNone(found.get('html')) + self.assertIsNotNone(found.get('problem')) + self.assertEqual(len(found.get('course')), 0) + self.assertEqual(len(found.get('about')), 1) + self.assertGreaterEqual(len(found.get('html')), 2) + self.assertGreaterEqual(len(found.get('problem')), 10) + dropdown = None + for template in found['problem']: + self.assertIn('metadata', template) + self.assertIn('display_name', template['metadata']) + if template['metadata']['display_name'] == 'Dropdown': + dropdown = template + break + self.assertIsNotNone(dropdown) + self.assertIn('markdown', dropdown['metadata']) + self.assertIn('data', dropdown) + self.assertRegexpMatches(dropdown['metadata']['markdown'], r'^Dropdown.*') + self.assertRegexpMatches(dropdown['data'], r'\s*

Dropdown.*') + + def test_get_some_templates(self): + self.assertEqual(len(SequenceDescriptor.templates()), 0) + self.assertGreater(len(HtmlDescriptor.templates()), 0) + self.assertIsNone(SequenceDescriptor.get_template('doesntexist.yaml')) + self.assertIsNone(HtmlDescriptor.get_template('doesntexist.yaml')) + self.assertIsNotNone(HtmlDescriptor.get_template('announcement.yaml')) + + def test_factories(self): + test_course = persistent_factories.PersistentCourseFactory.create(org='testx', prettyid='tempcourse', + display_name='fun test course', user_id='testbot') + self.assertIsInstance(test_course, CourseDescriptor) + self.assertEqual(test_course.display_name, 'fun test course') + index_info = modulestore('split').get_course_index_info(test_course.location) + self.assertEqual(index_info['org'], 'testx') + self.assertEqual(index_info['prettyid'], 'tempcourse') + + test_chapter = persistent_factories.ItemFactory.create(display_name='chapter 1', + parent_location=test_course.location) + self.assertIsInstance(test_chapter, SequenceDescriptor) + # refetch parent which should now point to child + test_course = modulestore('split').get_course(test_chapter.location) + self.assertIn(test_chapter.location.usage_id, test_course.children) + + def test_temporary_xblocks(self): + """ + Test using load_from_json to create non persisted xblocks + """ + test_course = persistent_factories.PersistentCourseFactory.create(org='testx', prettyid='tempcourse', + display_name='fun test course', user_id='testbot') + + test_chapter = XModuleDescriptor.load_from_json({'category': 'chapter', + 'metadata': {'display_name': 'chapter n'}}, + test_course.system, parent_xblock=test_course) + self.assertIsInstance(test_chapter, SequenceDescriptor) + self.assertEqual(test_chapter.display_name, 'chapter n') + self.assertIn(test_chapter, test_course.get_children()) + + # test w/ a definition (e.g., a problem) + test_def_content = 'boo' + test_problem = XModuleDescriptor.load_from_json({'category': 'problem', + 'definition': {'data': test_def_content}}, + test_course.system, parent_xblock=test_chapter) + self.assertIsInstance(test_problem, CapaDescriptor) + self.assertEqual(test_problem.data, test_def_content) + self.assertIn(test_problem, test_chapter.get_children()) + test_problem.display_name = 'test problem' + self.assertEqual(test_problem.display_name, 'test problem') + + def test_persist_dag(self): + """ + try saving temporary xblocks + """ + test_course = persistent_factories.PersistentCourseFactory.create(org='testx', prettyid='tempcourse', + display_name='fun test course', user_id='testbot') + test_chapter = XModuleDescriptor.load_from_json({'category': 'chapter', + 'metadata': {'display_name': 'chapter n'}}, + test_course.system, parent_xblock=test_course) + test_def_content = 'boo' + test_problem = XModuleDescriptor.load_from_json({'category': 'problem', + 'definition': {'data': test_def_content}}, + test_course.system, parent_xblock=test_chapter) + # better to pass in persisted parent over the subdag so + # subdag gets the parent pointer (otherwise 2 ops, persist dag, update parent children, + # persist parent + persisted_course = modulestore('split').persist_xblock_dag(test_course, 'testbot') + self.assertEqual(len(persisted_course.children), 1) + persisted_chapter = persisted_course.get_children()[0] + self.assertEqual(persisted_chapter.category, 'chapter') + self.assertEqual(persisted_chapter.display_name, 'chapter n') + self.assertEqual(len(persisted_chapter.children), 1) + persisted_problem = persisted_chapter.get_children()[0] + self.assertEqual(persisted_problem.category, 'problem') + self.assertEqual(persisted_problem.data, test_def_content) + + def test_delete_course(self): + test_course = persistent_factories.PersistentCourseFactory.create( + org='testx', + prettyid='edu.harvard.history.doomed', + display_name='doomed test course', + user_id='testbot') + persistent_factories.ItemFactory.create(display_name='chapter 1', + parent_location=test_course.location) + + id_locator = CourseLocator(course_id=test_course.location.course_id, revision='draft') + guid_locator = CourseLocator(version_guid=test_course.location.version_guid) + # verify it can be retireved by id + self.assertIsInstance(modulestore('split').get_course(id_locator), CourseDescriptor) + # and by guid + self.assertIsInstance(modulestore('split').get_course(guid_locator), CourseDescriptor) + modulestore('split').delete_course(id_locator.course_id) + # test can no longer retrieve by id + self.assertRaises(ItemNotFoundError, modulestore('split').get_course, id_locator) + # but can by guid + self.assertIsInstance(modulestore('split').get_course(guid_locator), CourseDescriptor) + + def test_block_generations(self): + """ + Test get_block_generations + """ + test_course = persistent_factories.PersistentCourseFactory.create( + org='testx', + prettyid='edu.harvard.history.hist101', + display_name='history test course', + user_id='testbot') + chapter = persistent_factories.ItemFactory.create(display_name='chapter 1', + parent_location=test_course.location, user_id='testbot') + sub = persistent_factories.ItemFactory.create(display_name='subsection 1', + parent_location=chapter.location, user_id='testbot', category='vertical') + first_problem = persistent_factories.ItemFactory.create(display_name='problem 1', + parent_location=sub.location, user_id='testbot', category='problem', data="") + first_problem.max_attempts = 3 + updated_problem = modulestore('split').update_item(first_problem, 'testbot') + updated_loc = modulestore('split').delete_item(updated_problem.location, 'testbot') + + second_problem = persistent_factories.ItemFactory.create(display_name='problem 2', + parent_location=BlockUsageLocator(updated_loc, usage_id=sub.location.usage_id), + user_id='testbot', category='problem', data="") + + # course root only updated 2x + version_history = modulestore('split').get_block_generations(test_course.location) + self.assertEqual(version_history.locator.version_guid, test_course.location.version_guid) + self.assertEqual(len(version_history.children), 1) + self.assertEqual(version_history.children[0].children, []) + self.assertEqual(version_history.children[0].locator.version_guid, chapter.location.version_guid) + + # sub changed on add, add problem, delete problem, add problem in strict linear seq + version_history = modulestore('split').get_block_generations(sub.location) + self.assertEqual(len(version_history.children), 1) + self.assertEqual(len(version_history.children[0].children), 1) + self.assertEqual(len(version_history.children[0].children[0].children), 1) + self.assertEqual(len(version_history.children[0].children[0].children[0].children), 0) + + # first and second problem may show as same usage_id; so, need to ensure their histories are right + version_history = modulestore('split').get_block_generations(updated_problem.location) + self.assertEqual(version_history.locator.version_guid, first_problem.location.version_guid) + self.assertEqual(len(version_history.children), 1) # updated max_attempts + self.assertEqual(len(version_history.children[0].children), 0) + + version_history = modulestore('split').get_block_generations(second_problem.location) + self.assertNotEqual(version_history.locator.version_guid, first_problem.location.version_guid) diff --git a/cms/djangoapps/contentstore/views/assets.py b/cms/djangoapps/contentstore/views/assets.py index d0b202da19..0bb9551ac9 100644 --- a/cms/djangoapps/contentstore/views/assets.py +++ b/cms/djangoapps/contentstore/views/assets.py @@ -13,7 +13,7 @@ from django_future.csrf import ensure_csrf_cookie from django.core.urlresolvers import reverse from django.core.servers.basehttp import FileWrapper from django.core.files.temp import NamedTemporaryFile -from django.views.decorators.http import require_POST +from django.views.decorators.http import require_POST, require_http_methods from mitxmako.shortcuts import render_to_response from cache_toolbox.core import del_cached_content @@ -249,6 +249,7 @@ def remove_asset(request, org, course, name): @ensure_csrf_cookie +@require_http_methods(("GET", "POST", "PUT")) @login_required def import_course(request, org, course, name): """ @@ -256,7 +257,7 @@ def import_course(request, org, course, name): """ location = get_location_and_verify_access(request, org, course, name) - if request.method == 'POST': + if request.method in ('POST', 'PUT'): filename = request.FILES['course-data'].name if not filename.endswith('.tar.gz'): diff --git a/cms/djangoapps/contentstore/views/checklist.py b/cms/djangoapps/contentstore/views/checklist.py index fdb5857ba7..bcf4a1a5b9 100644 --- a/cms/djangoapps/contentstore/views/checklist.py +++ b/cms/djangoapps/contentstore/views/checklist.py @@ -37,6 +37,7 @@ def get_checklists(request, org, course, name): checklists, modified = expand_checklist_action_urls(course_module) if copied or modified: + course_module.save() modulestore.update_metadata(location, own_metadata(course_module)) return render_to_response('checklists.html', { @@ -69,6 +70,7 @@ def update_checklist(request, org, course, name, checklist_index=None): # seeming noop which triggers kvs to record that the metadata is not default course_module.checklists = course_module.checklists checklists, _ = expand_checklist_action_urls(course_module) + course_module.save() modulestore.update_metadata(location, own_metadata(course_module)) return JsonResponse(checklists[index]) else: @@ -79,6 +81,7 @@ def update_checklist(request, org, course, name, checklist_index=None): # In the JavaScript view initialize method, we do a fetch to get all the checklists. checklists, modified = expand_checklist_action_urls(course_module) if modified: + course_module.save() modulestore.update_metadata(location, own_metadata(course_module)) return JsonResponse(checklists) diff --git a/cms/djangoapps/contentstore/views/component.py b/cms/djangoapps/contentstore/views/component.py index 505a93903a..1be6ac2822 100644 --- a/cms/djangoapps/contentstore/views/component.py +++ b/cms/djangoapps/contentstore/views/component.py @@ -245,6 +245,7 @@ def edit_unit(request, location): @expect_json @login_required +@require_http_methods(("GET", "POST", "PUT")) @ensure_csrf_cookie def assignment_type_update(request, org, course, category, name): ''' @@ -256,7 +257,7 @@ def assignment_type_update(request, org, course, category, name): if request.method == 'GET': return JsonResponse(CourseGradingModel.get_section_grader_type(location)) - elif request.method == 'POST': # post or put, doesn't matter. + elif request.method in ('POST', 'PUT'): # post or put, doesn't matter. return JsonResponse(CourseGradingModel.update_section_grader_type(location, request.POST)) diff --git a/cms/djangoapps/contentstore/views/course.py b/cms/djangoapps/contentstore/views/course.py index 3791e6779a..02eb4c65b8 100644 --- a/cms/djangoapps/contentstore/views/course.py +++ b/cms/djangoapps/contentstore/views/course.py @@ -42,8 +42,7 @@ from .component import ( ADVANCED_COMPONENT_POLICY_KEY) from django_comment_common.utils import seed_permissions_roles -import datetime -from django.utils.timezone import UTC + from xmodule.html_module import AboutDescriptor __all__ = ['course_index', 'create_new_course', 'course_info', 'course_info_updates', 'get_course_settings', @@ -176,6 +175,7 @@ def course_info(request, org, course, name, provided_id=None): @expect_json +@require_http_methods(("GET", "POST", "PUT", "DELETE")) @login_required @ensure_csrf_cookie def course_info_updates(request, org, course, provided_id=None): @@ -206,7 +206,7 @@ def course_info_updates(request, org, course, provided_id=None): except: return HttpResponseBadRequest("Failed to delete", content_type="text/plain") - elif request.method == 'POST': + elif request.method in ('POST', 'PUT'): # can be either and sometimes django is rewriting one to the other try: return JsonResponse(update_course_updates(location, request.POST, provided_id)) except: @@ -300,7 +300,7 @@ def course_settings_updates(request, org, course, name, section): if request.method == 'GET': # Cannot just do a get w/o knowing the course name :-( return JsonResponse(manager.fetch(Location(['i4x', org, course, 'course', name])), encoder=CourseSettingsEncoder) - elif request.method == 'POST': # post or put, doesn't matter. + elif request.method in ('POST', 'PUT'): # post or put, doesn't matter. return JsonResponse(manager.update_from_json(request.POST), encoder=CourseSettingsEncoder) @@ -479,7 +479,7 @@ def textbook_index(request, org, course, name): if request.is_ajax(): if request.method == 'GET': return JsonResponse(course_module.pdf_textbooks) - elif request.method == 'POST': + elif request.method in ('POST', 'PUT'): # can be either and sometimes django is rewriting one to the other try: textbooks = validate_textbooks_json(request.body) except TextbookValidationError as err: @@ -580,7 +580,7 @@ def textbook_by_id(request, org, course, name, tid): if not textbook: return JsonResponse(status=404) return JsonResponse(textbook) - elif request.method in ('POST', 'PUT'): + elif request.method in ('POST', 'PUT'): # can be either and sometimes django is rewriting one to the other try: new_textbook = validate_textbook_json(request.body) except TextbookValidationError as err: diff --git a/cms/djangoapps/contentstore/views/user.py b/cms/djangoapps/contentstore/views/user.py index ee6b0bf84d..020e4b5cb9 100644 --- a/cms/djangoapps/contentstore/views/user.py +++ b/cms/djangoapps/contentstore/views/user.py @@ -83,6 +83,9 @@ def add_user(request, location): } return JsonResponse(msg, 400) + # remove leading/trailing whitespace if necessary + email = email.strip() + # check that logged in user has admin permissions to this course if not has_access(request.user, location, role=INSTRUCTOR_ROLE_NAME): raise PermissionDenied() diff --git a/cms/envs/aws.py b/cms/envs/aws.py index 20217deaff..339425fee5 100644 --- a/cms/envs/aws.py +++ b/cms/envs/aws.py @@ -92,6 +92,7 @@ LOG_DIR = ENV_TOKENS['LOG_DIR'] CACHES = ENV_TOKENS['CACHES'] SESSION_COOKIE_DOMAIN = ENV_TOKENS.get('SESSION_COOKIE_DOMAIN') +SESSION_ENGINE = ENV_TOKENS.get('SESSION_ENGINE', SESSION_ENGINE) # allow for environments to specify what cookie name our login subsystem should use # this is to fix a bug regarding simultaneous logins between edx.org and edge.edx.org which can @@ -122,6 +123,10 @@ LOGGING = get_logger_config(LOG_DIR, debug=False, service_variant=SERVICE_VARIANT) +#theming start: +PLATFORM_NAME = ENV_TOKENS.get('PLATFORM_NAME', 'edX') + + ################ SECURE AUTH ITEMS ############################### # Secret things: passwords, access keys, etc. with open(ENV_ROOT / CONFIG_PREFIX + "auth.json") as auth_file: diff --git a/cms/envs/dev.py b/cms/envs/dev.py index 4f8174ac2b..0b0a62f05d 100644 --- a/cms/envs/dev.py +++ b/cms/envs/dev.py @@ -33,6 +33,10 @@ MODULESTORE = { 'direct': { 'ENGINE': 'xmodule.modulestore.mongo.MongoModuleStore', 'OPTIONS': modulestore_options + }, + 'split': { + 'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore', + 'OPTIONS': modulestore_options } } diff --git a/cms/envs/test.py b/cms/envs/test.py index 431a2c4184..efc7c5a7ef 100644 --- a/cms/envs/test.py +++ b/cms/envs/test.py @@ -63,6 +63,10 @@ MODULESTORE = { 'draft': { 'ENGINE': 'xmodule.modulestore.draft.DraftModuleStore', 'OPTIONS': MODULESTORE_OPTIONS + }, + 'split': { + 'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore', + 'OPTIONS': MODULESTORE_OPTIONS } } diff --git a/cms/manage.py b/cms/manage.py deleted file mode 100755 index 723fa59da1..0000000000 --- a/cms/manage.py +++ /dev/null @@ -1,16 +0,0 @@ -#!/usr/bin/env python -from django.core.management import execute_manager -import imp -try: - imp.find_module('settings') # Assumed to be in the same directory. -except ImportError: - import sys - sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. " - "It appears you've customized things.\nYou'll have to run django-admin.py, " - "passing it your settings module.\n" % __file__) - sys.exit(1) - -import settings - -if __name__ == "__main__": - execute_manager(settings) diff --git a/cms/static/coffee/spec/views/overview_spec.coffee b/cms/static/coffee/spec/views/overview_spec.coffee index ba9b5d8918..d900e4bfb1 100644 --- a/cms/static/coffee/spec/views/overview_spec.coffee +++ b/cms/static/coffee/spec/views/overview_spec.coffee @@ -40,17 +40,30 @@ describe "Course Overview", -> """#" + appendSetFixtures """ +

+ +
+ """#" + spyOn(window, 'saveSetSectionScheduleDate').andCallThrough() # Have to do this here, as it normally gets bound in document.ready() $('a.save-button').click(saveSetSectionScheduleDate) + $('a.delete-section-button').click(deleteSection) + @notificationSpy = spyOn(CMS.Views.Notification.Mini.prototype, 'show').andCallThrough() window.analytics = jasmine.createSpyObj('analytics', ['track']) window.course_location_analytics = jasmine.createSpy() - sinon.useFakeXMLHttpRequest() + @xhr = sinon.useFakeXMLHttpRequest() + requests = @requests = [] + @xhr.onCreate = (req) -> requests.push(req) afterEach -> delete window.analytics delete window.course_location_analytics + @notificationSpy.reset() it "should save model when save is clicked", -> $('a.edit-button').click() @@ -61,3 +74,21 @@ describe "Course Overview", -> $('a.edit-button').click() $('a.save-button').click() expect(@notificationSpy).toHaveBeenCalled() + + it "should delete model when delete is clicked", -> + deleteSpy = spyOn(window, '_deleteItem').andCallThrough() + $('a.delete-section-button').click() + $('a.action-primary').click() + expect(deleteSpy).toHaveBeenCalled() + expect(@requests[0].url).toEqual('/delete_item') + + it "should not delete model when cancel is clicked", -> + deleteSpy = spyOn(window, '_deleteItem').andCallThrough() + $('a.delete-section-button').click() + $('a.action-secondary').click() + expect(@requests.length).toEqual(0) + + it "should show a confirmation on delete", -> + $('a.delete-section-button').click() + $('a.action-primary').click() + expect(@notificationSpy).toHaveBeenCalled() diff --git a/cms/static/coffee/src/views/module_edit.coffee b/cms/static/coffee/src/views/module_edit.coffee index 62083fa26d..c45feecd41 100644 --- a/cms/static/coffee/src/views/module_edit.coffee +++ b/cms/static/coffee/src/views/module_edit.coffee @@ -84,11 +84,15 @@ class CMS.Views.ModuleEdit extends Backbone.View data.metadata = _.extend(data.metadata || {}, @changedMetadata()) @hideModal() + saving = new CMS.Views.Notification.Mini + title: gettext('Saving') + '…' + saving.show() @model.save(data).done( => # # showToastMessage("Your changes have been saved.", null, 3) @module = null @render() @$el.removeClass('editing') + saving.hide() ).fail( -> showToastMessage(gettext("There was an error saving your changes. Please try again."), null, 3) ) diff --git a/cms/static/coffee/src/views/unit.coffee b/cms/static/coffee/src/views/unit.coffee index 774ef04f6d..14b64b7b9e 100644 --- a/cms/static/coffee/src/views/unit.coffee +++ b/cms/static/coffee/src/views/unit.coffee @@ -67,8 +67,8 @@ class CMS.Views.UnitEdit extends Backbone.View type = $(event.currentTarget).data('type') @$newComponentTypePicker.slideUp(250) @$(".new-component-#{type}").slideDown(250) - $('html, body').animate({ - scrollTop: @$(".new-component-#{type}").offset().top + $('html, body').animate({ + scrollTop: @$(".new-component-#{type}").offset().top }, 500) closeNewComponent: (event) => @@ -115,27 +115,43 @@ class CMS.Views.UnitEdit extends Backbone.View @model.save() deleteComponent: (event) => - if not confirm 'Are you sure you want to delete this component? This action cannot be undone.' - return - $component = $(event.currentTarget).parents('.component') - $.post('/delete_item', { - id: $component.data('id') - }, => - analytics.track "Deleted a Component", - course: course_location_analytics - unit_id: unit_location_analytics - id: $component.data('id') + msg = new CMS.Views.Prompt.Warning( + title: gettext('Delete this component?'), + message: gettext('Deleting this component is permanent and cannot be undone.'), + actions: + primary: + text: gettext('Yes, delete this component'), + click: (view) => + view.hide() + deleting = new CMS.Views.Notification.Mini + title: gettext('Deleting') + '…', + deleting.show() + $component = $(event.currentTarget).parents('.component') + $.post('/delete_item', { + id: $component.data('id') + }, => + deleting.hide() + analytics.track "Deleted a Component", + course: course_location_analytics + unit_id: unit_location_analytics + id: $component.data('id') - $component.remove() - # b/c we don't vigilantly keep children up to date - # get rid of it before it hurts someone - # sorry for the js, i couldn't figure out the coffee equivalent - `_this.model.save({children: _this.components()}, - {success: function(model) { - model.unset('children'); - }} - );` + $component.remove() + # b/c we don't vigilantly keep children up to date + # get rid of it before it hurts someone + # sorry for the js, i couldn't figure out the coffee equivalent + `_this.model.save({children: _this.components()}, + {success: function(model) { + model.unset('children'); + }} + );` + ) + secondary: + text: gettext('Cancel'), + click: (view) -> + view.hide() ) + msg.show() deleteDraft: (event) -> @wait(true) @@ -236,7 +252,7 @@ class CMS.Views.UnitEdit.NameEdit extends Backbone.View class CMS.Views.UnitEdit.LocationState extends Backbone.View initialize: => @model.on('change:state', @render) - + render: => @$el.toggleClass("#{@model.previous('state')}-item #{@model.get('state')}-item") diff --git a/cms/static/js/base.js b/cms/static/js/base.js index 329624ef46..3a30e0bc81 100644 --- a/cms/static/js/base.js +++ b/cms/static/js/base.js @@ -356,39 +356,61 @@ function createNewUnit(e) { function deleteUnit(e) { e.preventDefault(); - _deleteItem($(this).parents('li.leaf')); + _deleteItem($(this).parents('li.leaf'), 'Unit'); } function deleteSubsection(e) { e.preventDefault(); - _deleteItem($(this).parents('li.branch')); + _deleteItem($(this).parents('li.branch'), 'Subsection'); } function deleteSection(e) { e.preventDefault(); - _deleteItem($(this).parents('section.branch')); + _deleteItem($(this).parents('section.branch'), 'Section'); } -function _deleteItem($el) { - if (!confirm(gettext('Are you sure you wish to delete this item. It cannot be reversed!'))) return; +function _deleteItem($el, type) { + var confirm = new CMS.Views.Prompt.Warning({ + title: gettext('Delete this ' + type + '?'), + message: gettext('Deleting this ' + type + ' is permanent and cannot be undone.'), + actions: { + primary: { + text: gettext('Yes, delete this ' + type), + click: function(view) { + view.hide(); - var id = $el.data('id'); + var id = $el.data('id'); - analytics.track('Deleted an Item', { - 'course': course_location_analytics, - 'id': id - }); - - - $.post('/delete_item', { - 'id': id, - 'delete_children': true, - 'delete_all_versions': true - }, - - function(data) { - $el.remove(); + analytics.track('Deleted an Item', { + 'course': course_location_analytics, + 'id': id + }); + + var deleting = new CMS.Views.Notification.Mini({ + title: gettext('Deleting') + '…' + }); + deleting.show(); + + $.post('/delete_item', + {'id': id, + 'delete_children': true, + 'delete_all_versions': true}, + function(data) { + $el.remove(); + deleting.hide(); + } + ); + } + }, + secondary: { + text: gettext('Cancel'), + click: function(view) { + view.hide(); + } + } + } }); + confirm.show(); } function markAsLoaded() { @@ -728,7 +750,7 @@ function saveSetSectionScheduleDate(e) { var $thisSection = $('.courseware-section[data-id="' + id + '"]'); var html = _.template( '' + - '' + gettext("Will Release: ") + '' + + '' + gettext("Will Release:") + ' ' + gettext("<%= date %> at <%= time %> UTC") + '' + '' + diff --git a/cms/static/js/views/assets.js b/cms/static/js/views/assets.js index 224ec928fb..282aeab69c 100644 --- a/cms/static/js/views/assets.js +++ b/cms/static/js/views/assets.js @@ -9,7 +9,7 @@ function removeAsset(e){ e.preventDefault(); var that = this; - var msg = new CMS.Views.Prompt.Confirmation({ + var msg = new CMS.Views.Prompt.Warning({ title: gettext("Delete File Confirmation"), message: gettext("Are you sure you wish to delete this item. It cannot be reversed!\n\nAlso any content that links/refers to this item will no longer work (e.g. broken images and/or links)"), actions: { diff --git a/cms/static/js/views/grader-select-view.js b/cms/static/js/views/grader-select-view.js index b22e763710..a16f5fa1b7 100644 --- a/cms/static/js/views/grader-select-view.js +++ b/cms/static/js/views/grader-select-view.js @@ -81,9 +81,18 @@ CMS.Views.OverviewAssignmentGrader = Backbone.View.extend({ this.removeMenu(e); + var saving = new CMS.Views.Notification.Mini({ + title: gettext('Saving') + '…' + }); + saving.show(); + // TODO I'm not happy with this string fetch via the html for what should be an id. I'd rather use the id attr // of the CourseGradingPolicy model or null for Not Graded (NOTE, change template's if check for is-selected accordingly) - this.assignmentGrade.save('graderType', $(e.target).text()); + this.assignmentGrade.save( + 'graderType', + $(e.target).text(), + {success: function () { saving.hide(); }} + ); this.render(); } diff --git a/cms/static/js/views/settings/main_settings_view.js b/cms/static/js/views/settings/main_settings_view.js index 5550c550c0..0cbf573ba9 100644 --- a/cms/static/js/views/settings/main_settings_view.js +++ b/cms/static/js/views/settings/main_settings_view.js @@ -107,6 +107,7 @@ CMS.Views.Settings.Details = CMS.Views.ValidatingView.extend({ // to pick up when the date is typed directly in the field. datefield.change(setfield); timefield.on('changeTime', setfield); + timefield.on('input', setfield); datefield.datepicker('setDate', this.model.get(fieldName)); // timepicker doesn't let us set null, so check that we have a time diff --git a/cms/static/js/views/textbook.js b/cms/static/js/views/textbook.js index 74eaae8601..f666143c0a 100644 --- a/cms/static/js/views/textbook.js +++ b/cms/static/js/views/textbook.js @@ -241,7 +241,7 @@ CMS.Views.EditChapter = Backbone.View.extend({ asset_path: this.$("input.chapter-asset-path").val() }); var msg = new CMS.Models.FileUpload({ - title: _.template(gettext("Upload a new asset to “<%= name %>”"), + title: _.template(gettext("Upload a new PDF to “<%= name %>”"), {name: section.escape('name')}), message: "Files must be in PDF format." }); diff --git a/cms/static/sass/views/_index.scss b/cms/static/sass/views/_index.scss index ddb2ecce89..d571949cba 100644 --- a/cms/static/sass/views/_index.scss +++ b/cms/static/sass/views/_index.scss @@ -71,8 +71,13 @@ body.index { color: $white; } + .wrapper-text-welcome, .logo { + display: inline-block; + } + .logo { font-weight: 600; + margin-left: ($baseline/2); } .tagline { diff --git a/cms/static/sass/views/_unit.scss b/cms/static/sass/views/_unit.scss index 45e9823c2e..06685ad96b 100644 --- a/cms/static/sass/views/_unit.scss +++ b/cms/static/sass/views/_unit.scss @@ -747,6 +747,7 @@ body.unit { // Unit Page Sidebar .unit-settings { + .window-contents { padding: $baseline/2 $baseline; } @@ -854,6 +855,24 @@ body.unit { } .unit-location { + + // unit id + .wrapper-unit-id { + + .unit-id { + + .label { + @extend .t-title7; + margin-bottom: ($baseline/4); + color: $gray-d1; + } + + .value { + margin-bottom: 0; + } + } + } + .url { box-shadow: none; width: 100%; diff --git a/cms/templates/edit-static-page.html b/cms/templates/edit-static-page.html deleted file mode 100644 index f1b2374b46..0000000000 --- a/cms/templates/edit-static-page.html +++ /dev/null @@ -1,41 +0,0 @@ -<%inherit file="base.html" /> -<%! from django.core.urlresolvers import reverse %> -<%block name="title">Editing Static Page -<%block name="bodyclass">is-signedin course pages edit-static-page - -<%block name="content"> -
-
-
-
-
- - -
-
- - -
-
-
-
-
-

Page Settings

-
-
- - -
-
- Save - Preview -
-
-
-
-
-
- \ No newline at end of file diff --git a/cms/templates/editable_preview.html b/cms/templates/editable_preview.html deleted file mode 100644 index 731fd9b1c8..0000000000 --- a/cms/templates/editable_preview.html +++ /dev/null @@ -1,13 +0,0 @@ -
-${content} -
- Edit - Delete -
- -
-
Edit Video Component
- - SaveCancel -
-
diff --git a/cms/templates/howitworks.html b/cms/templates/howitworks.html index 8de260a69b..e3a92aa345 100644 --- a/cms/templates/howitworks.html +++ b/cms/templates/howitworks.html @@ -11,7 +11,7 @@
## "edX Studio" should not be translated -

${_('Welcome to')}

+

${_('Welcome to')}

${_("Studio helps manage your courses online, so you can focus on teaching them")}

diff --git a/cms/templates/js/edit-chapter.underscore b/cms/templates/js/edit-chapter.underscore index 0f9a91e3e0..34f42daa64 100644 --- a/cms/templates/js/edit-chapter.underscore +++ b/cms/templates/js/edit-chapter.underscore @@ -9,6 +9,6 @@ " value="<%= asset_path %>" type="text"> <%= gettext("upload a PDF file or provide the path to a Studio asset file") %> - + <%= gettext("delete chapter") %> diff --git a/cms/templates/overview.html b/cms/templates/overview.html index 56836b00ad..3795e9d09b 100644 --- a/cms/templates/overview.html +++ b/cms/templates/overview.html @@ -1,4 +1,4 @@ -/<%! from django.utils.translation import ugettext as _ %> +<%! from django.utils.translation import ugettext as _ %> <%inherit file="base.html" /> <%! import logging diff --git a/cms/templates/unit.html b/cms/templates/unit.html index 300d631421..fdff92e389 100644 --- a/cms/templates/unit.html +++ b/cms/templates/unit.html @@ -171,10 +171,15 @@

${_("Unit Location")}

-
+
+

+ ${_("Unit Identifier:")} + +

+
  1. - ${section.display_name_with_default} + ${section.display_name_with_default}
    1. diff --git a/common/djangoapps/external_auth/models.py b/common/djangoapps/external_auth/models.py index 6c2f38d8b3..b40e427434 100644 --- a/common/djangoapps/external_auth/models.py +++ b/common/djangoapps/external_auth/models.py @@ -4,9 +4,9 @@ WE'RE USING MIGRATIONS! If you make changes to this model, be sure to create an appropriate migration file and check it in at the same time as your model changes. To do that, -1. Go to the mitx dir -2. django-admin.py schemamigration student --auto --settings=lms.envs.dev --pythonpath=. description_of_your_change -3. Add the migration file created in mitx/common/djangoapps/external_auth/migrations/ +1. Go to the edx-platform dir +2. ./manage.py lms schemamigration student --auto description_of_your_change +3. Add the migration file created in edx-platform/common/djangoapps/external_auth/migrations/ """ from django.db import models diff --git a/common/djangoapps/heartbeat/views.py b/common/djangoapps/heartbeat/views.py index d7c3a32192..0cee7116b4 100644 --- a/common/djangoapps/heartbeat/views.py +++ b/common/djangoapps/heartbeat/views.py @@ -1,16 +1,18 @@ import json from datetime import datetime +from pytz import UTC from django.http import HttpResponse from xmodule.modulestore.django import modulestore from dogapi import dog_stats_api + @dog_stats_api.timed('edxapp.heartbeat') def heartbeat(request): """ Simple view that a loadbalancer can check to verify that the app is up """ output = { - 'date': datetime.now().isoformat(), + 'date': datetime.now(UTC).isoformat(), 'courses': [course.location.url() for course in modulestore().get_courses()], } return HttpResponse(json.dumps(output, indent=4)) diff --git a/common/djangoapps/static_replace/__init__.py b/common/djangoapps/static_replace/__init__.py index b73a658c5f..9e50d73b26 100644 --- a/common/djangoapps/static_replace/__init__.py +++ b/common/djangoapps/static_replace/__init__.py @@ -43,6 +43,35 @@ def try_staticfiles_lookup(path): return url +def replace_jump_to_id_urls(text, course_id, jump_to_id_base_url): + """ + This will replace a link to another piece of courseware to a 'jump_to' + URL that will redirect to the right place in the courseware + + NOTE: This is similar to replace_course_urls in terms of functionality + but it is intended to be used when we only have a 'id' that the + course author provides. This is much more helpful when using + Studio authored courses since they don't need to know the path. This + is also durable with respect to item moves. + + text: The content over which to perform the subtitutions + course_id: The course_id in which this rewrite happens + jump_to_id_base_url: + A app-tier (e.g. LMS) absolute path to the base of the handler that will perform the + redirect. e.g. /courses////jump_to_id. NOTE the will be appended to + the end of this URL at re-write time + + output: after the link rewriting rules are applied + """ + + def replace_jump_to_id_url(match): + quote = match.group('quote') + rest = match.group('rest') + return "".join([quote, jump_to_id_base_url + rest, quote]) + + return re.sub(_url_replace_regex('/jump_to_id/'), replace_jump_to_id_url, text) + + def replace_course_urls(text, course_id): """ Replace /course/$stuff urls with /courses/$course_id/$stuff urls @@ -53,7 +82,6 @@ def replace_course_urls(text, course_id): returns: text with the links replaced """ - def replace_course_url(match): quote = match.group('quote') rest = match.group('rest') diff --git a/common/djangoapps/student/management/commands/pearson_transfer.py b/common/djangoapps/student/management/commands/pearson_transfer.py index 75716c7443..39e1e7468e 100644 --- a/common/djangoapps/student/management/commands/pearson_transfer.py +++ b/common/djangoapps/student/management/commands/pearson_transfer.py @@ -20,7 +20,7 @@ class Command(BaseCommand): files and then uploads over SFTP to Pearson and stuffs the entry in an S3 bucket for archive purposes. - Usage: django-admin.py pearson-transfer --mode [import|export|both] + Usage: ./manage.py pearson-transfer --mode [import|export|both] """ option_list = BaseCommand.option_list + ( diff --git a/common/djangoapps/student/models.py b/common/djangoapps/student/models.py index af93c34317..97d7e8b028 100644 --- a/common/djangoapps/student/models.py +++ b/common/djangoapps/student/models.py @@ -6,9 +6,9 @@ Migration Notes If you make changes to this model, be sure to create an appropriate migration file and check it in at the same time as your model changes. To do that, -1. Go to the mitx dir -2. django-admin.py schemamigration student --auto --settings=lms.envs.dev --pythonpath=. description_of_your_change -3. Add the migration file created in mitx/common/djangoapps/student/migrations/ +1. Go to the edx-platform dir +2. ./manage.py lms schemamigration student --auto description_of_your_change +3. Add the migration file created in edx-platform/common/djangoapps/student/migrations/ """ from datetime import datetime import hashlib @@ -69,30 +69,33 @@ class UserProfile(models.Model): location = models.CharField(blank=True, max_length=255, db_index=True) # Optional demographic data we started capturing from Fall 2012 - this_year = datetime.now().year + this_year = datetime.now(UTC).year VALID_YEARS = range(this_year, this_year - 120, -1) year_of_birth = models.IntegerField(blank=True, null=True, db_index=True) GENDER_CHOICES = (('m', 'Male'), ('f', 'Female'), ('o', 'Other')) - gender = models.CharField(blank=True, null=True, max_length=6, db_index=True, - choices=GENDER_CHOICES) + gender = models.CharField( + blank=True, null=True, max_length=6, db_index=True, choices=GENDER_CHOICES + ) # [03/21/2013] removed these, but leaving comment since there'll still be # p_se and p_oth in the existing data in db. # ('p_se', 'Doctorate in science or engineering'), # ('p_oth', 'Doctorate in another field'), - LEVEL_OF_EDUCATION_CHOICES = (('p', 'Doctorate'), - ('m', "Master's or professional degree"), - ('b', "Bachelor's degree"), - ('a', "Associate's degree"), - ('hs', "Secondary/high school"), - ('jhs', "Junior secondary/junior high/middle school"), - ('el', "Elementary/primary school"), - ('none', "None"), - ('other', "Other")) + LEVEL_OF_EDUCATION_CHOICES = ( + ('p', 'Doctorate'), + ('m', "Master's or professional degree"), + ('b', "Bachelor's degree"), + ('a', "Associate's degree"), + ('hs', "Secondary/high school"), + ('jhs', "Junior secondary/junior high/middle school"), + ('el', "Elementary/primary school"), + ('none', "None"), + ('other', "Other") + ) level_of_education = models.CharField( - blank=True, null=True, max_length=6, db_index=True, - choices=LEVEL_OF_EDUCATION_CHOICES - ) + blank=True, null=True, max_length=6, db_index=True, + choices=LEVEL_OF_EDUCATION_CHOICES + ) mailing_address = models.TextField(blank=True, null=True) goals = models.TextField(blank=True, null=True) allow_certificate = models.BooleanField(default=1) @@ -307,18 +310,18 @@ class TestCenterUserForm(ModelForm): ACCOMMODATION_REJECTED_CODE = 'NONE' ACCOMMODATION_CODES = ( - (ACCOMMODATION_REJECTED_CODE, 'No Accommodation Granted'), - ('EQPMNT', 'Equipment'), - ('ET12ET', 'Extra Time - 1/2 Exam Time'), - ('ET30MN', 'Extra Time - 30 Minutes'), - ('ETDBTM', 'Extra Time - Double Time'), - ('SEPRMM', 'Separate Room'), - ('SRREAD', 'Separate Room and Reader'), - ('SRRERC', 'Separate Room and Reader/Recorder'), - ('SRRECR', 'Separate Room and Recorder'), - ('SRSEAN', 'Separate Room and Service Animal'), - ('SRSGNR', 'Separate Room and Sign Language Interpreter'), - ) + (ACCOMMODATION_REJECTED_CODE, 'No Accommodation Granted'), + ('EQPMNT', 'Equipment'), + ('ET12ET', 'Extra Time - 1/2 Exam Time'), + ('ET30MN', 'Extra Time - 30 Minutes'), + ('ETDBTM', 'Extra Time - Double Time'), + ('SEPRMM', 'Separate Room'), + ('SRREAD', 'Separate Room and Reader'), + ('SRRERC', 'Separate Room and Reader/Recorder'), + ('SRRECR', 'Separate Room and Recorder'), + ('SRSEAN', 'Separate Room and Service Animal'), + ('SRSGNR', 'Separate Room and Sign Language Interpreter'), +) ACCOMMODATION_CODE_DICT = {code: name for (code, name) in ACCOMMODATION_CODES} @@ -572,7 +575,6 @@ class TestCenterRegistrationForm(ModelForm): return code - def get_testcenter_registration(user, course_id, exam_series_code): try: tcu = TestCenterUser.objects.get(user=user) diff --git a/common/djangoapps/student/views.py b/common/djangoapps/student/views.py index 398a3f6efc..553643bde7 100644 --- a/common/djangoapps/student/views.py +++ b/common/djangoapps/student/views.py @@ -111,9 +111,9 @@ def get_date_for_press(publish_date): # strip off extra months, and just use the first: date = re.sub(multimonth_pattern, ", ", publish_date) if re.search(day_pattern, date): - date = datetime.datetime.strptime(date, "%B %d, %Y") + date = datetime.datetime.strptime(date, "%B %d, %Y").replace(tzinfo=UTC) else: - date = datetime.datetime.strptime(date, "%B, %Y") + date = datetime.datetime.strptime(date, "%B, %Y").replace(tzinfo=UTC) return date @@ -1100,7 +1100,7 @@ def confirm_email_change(request, key): meta = up.get_meta() if 'old_emails' not in meta: meta['old_emails'] = [] - meta['old_emails'].append([user.email, datetime.datetime.now().isoformat()]) + meta['old_emails'].append([user.email, datetime.datetime.now(UTC).isoformat()]) up.set_meta(meta) up.save() # Send it to the old email... @@ -1198,7 +1198,7 @@ def accept_name_change_by_id(id): meta = up.get_meta() if 'old_names' not in meta: meta['old_names'] = [] - meta['old_names'].append([up.name, pnc.rationale, datetime.datetime.now().isoformat()]) + meta['old_names'].append([up.name, pnc.rationale, datetime.datetime.now(UTC).isoformat()]) up.set_meta(meta) up.name = pnc.new_name diff --git a/common/djangoapps/terrain/steps.py b/common/djangoapps/terrain/steps.py index bd0b448e11..cfa8de15b7 100644 --- a/common/djangoapps/terrain/steps.py +++ b/common/djangoapps/terrain/steps.py @@ -129,6 +129,13 @@ def should_have_link_with_id_and_text(step, link_id, text): assert_equals(link.text, text) +@step(r'should see a link to "([^"]*)" with the text "([^"]*)"$') +def should_have_link_with_path_and_text(step, path, text): + link = world.browser.find_link_by_text(text) + assert len(link) > 0 + assert_equals(link.first["href"], django_url(path)) + + @step(r'should( not)? see "(.*)" (?:somewhere|anywhere) (?:in|on) (?:the|this) page') def should_see_in_the_page(step, doesnt_appear, text): if doesnt_appear: diff --git a/common/djangoapps/xmodule_modifiers.py b/common/djangoapps/xmodule_modifiers.py index 3efc04789e..dd40b5139d 100644 --- a/common/djangoapps/xmodule_modifiers.py +++ b/common/djangoapps/xmodule_modifiers.py @@ -42,6 +42,28 @@ def wrap_xmodule(get_html, module, template, context=None): return _get_html +def replace_jump_to_id_urls(get_html, course_id, jump_to_id_base_url): + """ + This will replace a link between courseware in the format + /jump_to/ with a URL for a page that will correctly redirect + This is similar to replace_course_urls, but much more flexible and + durable for Studio authored courses. See more comments in static_replace.replace_jump_to_urls + + course_id: The course_id in which this rewrite happens + jump_to_id_base_url: + A app-tier (e.g. LMS) absolute path to the base of the handler that will perform the + redirect. e.g. /courses////jump_to_id. NOTE the will be appended to + the end of this URL at re-write time + + output: a wrapped get_html() function pointer, which, when called, will apply the + rewrite rules + """ + @wraps(get_html) + def _get_html(): + return static_replace.replace_jump_to_id_urls(get_html(), course_id, jump_to_id_base_url) + return _get_html + + def replace_course_urls(get_html, course_id): """ Updates the supplied module with a new get_html function that wraps diff --git a/common/lib/capa/capa/capa_problem.py b/common/lib/capa/capa/capa_problem.py index 2c813f49d5..c2bdeadc21 100644 --- a/common/lib/capa/capa/capa_problem.py +++ b/common/lib/capa/capa/capa_problem.py @@ -32,6 +32,8 @@ import capa.xqueue_interface as xqueue_interface import capa.responsetypes as responsetypes from capa.safe_exec import safe_exec +from pytz import UTC + # dict of tagname, Response Class -- this should come from auto-registering response_tag_dict = dict([(x.response_tag, x) for x in responsetypes.__all__]) @@ -42,13 +44,22 @@ solution_tags = ['solution'] response_properties = ["codeparam", "responseparam", "answer", "openendedparam"] # special problem tags which should be turned into innocuous HTML -html_transforms = {'problem': {'tag': 'div'}, - 'text': {'tag': 'span'}, - 'math': {'tag': 'span'}, - } +html_transforms = { + 'problem': {'tag': 'div'}, + 'text': {'tag': 'span'}, + 'math': {'tag': 'span'}, +} # These should be removed from HTML output, including all subelements -html_problem_semantics = ["codeparam", "responseparam", "answer", "script", "hintgroup", "openendedparam", "openendedrubric"] +html_problem_semantics = [ + "codeparam", + "responseparam", + "answer", + "script", + "hintgroup", + "openendedparam", + "openendedrubric" +] log = logging.getLogger(__name__) @@ -242,11 +253,15 @@ class LoncapaProblem(object): return None # Get a list of timestamps of all queueing requests, then convert it to a DateTime object - queuetime_strs = [self.correct_map.get_queuetime_str(answer_id) - for answer_id in self.correct_map - if self.correct_map.is_queued(answer_id)] - queuetimes = [datetime.strptime(qt_str, xqueue_interface.dateformat) - for qt_str in queuetime_strs] + queuetime_strs = [ + self.correct_map.get_queuetime_str(answer_id) + for answer_id in self.correct_map + if self.correct_map.is_queued(answer_id) + ] + queuetimes = [ + datetime.strptime(qt_str, xqueue_interface.dateformat).replace(tzinfo=UTC) + for qt_str in queuetime_strs + ] return max(queuetimes) @@ -404,10 +419,16 @@ class LoncapaProblem(object): # open using ModuleSystem OSFS filestore ifp = self.system.filestore.open(filename) except Exception as err: - log.warning('Error %s in problem xml include: %s' % ( - err, etree.tostring(inc, pretty_print=True))) - log.warning('Cannot find file %s in %s' % ( - filename, self.system.filestore)) + log.warning( + 'Error %s in problem xml include: %s' % ( + err, etree.tostring(inc, pretty_print=True) + ) + ) + log.warning( + 'Cannot find file %s in %s' % ( + filename, self.system.filestore + ) + ) # if debugging, don't fail - just log error # TODO (vshnayder): need real error handling, display to users if not self.system.get('DEBUG'): @@ -418,8 +439,11 @@ class LoncapaProblem(object): # read in and convert to XML incxml = etree.XML(ifp.read()) except Exception as err: - log.warning('Error %s in problem xml include: %s' % ( - err, etree.tostring(inc, pretty_print=True))) + log.warning( + 'Error %s in problem xml include: %s' % ( + err, etree.tostring(inc, pretty_print=True) + ) + ) log.warning('Cannot parse XML in %s' % (filename)) # if debugging, don't fail - just log error # TODO (vshnayder): same as above @@ -579,8 +603,9 @@ class LoncapaProblem(object): # let each Response render itself if problemtree in self.responders: overall_msg = self.correct_map.get_overall_message() - return self.responders[problemtree].render_html(self._extract_html, - response_msg=overall_msg) + return self.responders[problemtree].render_html( + self._extract_html, response_msg=overall_msg + ) # let each custom renderer render itself: if problemtree.tag in customrender.registry.registered_tags(): @@ -628,9 +653,10 @@ class LoncapaProblem(object): answer_id = 1 input_tags = inputtypes.registry.registered_tags() - inputfields = tree.xpath("|".join(['//' + response.tag + '[@id=$id]//' + x - for x in (input_tags + solution_tags)]), - id=response_id_str) + inputfields = tree.xpath( + "|".join(['//' + response.tag + '[@id=$id]//' + x for x in (input_tags + solution_tags)]), + id=response_id_str + ) # assign one answer_id for each input type or solution type for entry in inputfields: diff --git a/common/lib/capa/capa/correctmap.py b/common/lib/capa/capa/correctmap.py index 950cd199fc..e50be92152 100644 --- a/common/lib/capa/capa/correctmap.py +++ b/common/lib/capa/capa/correctmap.py @@ -37,23 +37,27 @@ class CorrectMap(object): return self.cmap.__iter__() # See the documentation for 'set_dict' for the use of kwargs - def set(self, - answer_id=None, - correctness=None, - npoints=None, - msg='', - hint='', - hintmode=None, - queuestate=None, **kwargs): + def set( + self, + answer_id=None, + correctness=None, + npoints=None, + msg='', + hint='', + hintmode=None, + queuestate=None, + **kwargs + ): if answer_id is not None: - self.cmap[str(answer_id)] = {'correctness': correctness, - 'npoints': npoints, - 'msg': msg, - 'hint': hint, - 'hintmode': hintmode, - 'queuestate': queuestate, - } + self.cmap[str(answer_id)] = { + 'correctness': correctness, + 'npoints': npoints, + 'msg': msg, + 'hint': hint, + 'hintmode': hintmode, + 'queuestate': queuestate, + } def __repr__(self): return repr(self.cmap) diff --git a/common/lib/capa/capa/responsetypes.py b/common/lib/capa/capa/responsetypes.py index f99518c8ce..03c82ea218 100644 --- a/common/lib/capa/capa/responsetypes.py +++ b/common/lib/capa/capa/responsetypes.py @@ -33,6 +33,7 @@ from shapely.geometry import Point, MultiPoint from calc import evaluator, UndefinedVariable from . import correctmap from datetime import datetime +from pytz import UTC from .util import * from lxml import etree from lxml.html.soupparser import fromstring as fromstring_bs # uses Beautiful Soup!!! FIXME? @@ -1365,9 +1366,11 @@ class CodeResponse(LoncapaResponse): # Note that submission can be a file submission = student_answers[self.answer_id] except Exception as err: - log.error('Error in CodeResponse %s: cannot get student answer for %s;' - ' student_answers=%s' % - (err, self.answer_id, convert_files_to_filenames(student_answers))) + log.error( + 'Error in CodeResponse %s: cannot get student answer for %s;' + ' student_answers=%s' % + (err, self.answer_id, convert_files_to_filenames(student_answers)) + ) raise Exception(err) # We do not support xqueue within Studio. @@ -1381,19 +1384,20 @@ class CodeResponse(LoncapaResponse): #------------------------------------------------------------ qinterface = self.system.xqueue['interface'] - qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat) + qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat) anonymous_student_id = self.system.anonymous_student_id # Generate header - queuekey = xqueue_interface.make_hashkey(str(self.system.seed) + qtime + - anonymous_student_id + - self.answer_id) + queuekey = xqueue_interface.make_hashkey( + str(self.system.seed) + qtime + anonymous_student_id + self.answer_id + ) callback_url = self.system.xqueue['construct_callback']() xheader = xqueue_interface.make_xheader( lms_callback_url=callback_url, lms_key=queuekey, - queue_name=self.queue_name) + queue_name=self.queue_name + ) # Generate body if is_list_of_files(submission): @@ -1406,9 +1410,10 @@ class CodeResponse(LoncapaResponse): # Metadata related to the student submission revealed to the external # grader - student_info = {'anonymous_student_id': anonymous_student_id, - 'submission_time': qtime, - } + student_info = { + 'anonymous_student_id': anonymous_student_id, + 'submission_time': qtime, + } contents.update({'student_info': json.dumps(student_info)}) # Submit request. When successful, 'msg' is the prior length of the diff --git a/common/lib/capa/capa/tests/test_responsetypes.py b/common/lib/capa/capa/tests/test_responsetypes.py index 2a15145579..a756dc640e 100644 --- a/common/lib/capa/capa/tests/test_responsetypes.py +++ b/common/lib/capa/capa/tests/test_responsetypes.py @@ -18,6 +18,8 @@ from capa.correctmap import CorrectMap from capa.util import convert_files_to_filenames from capa.xqueue_interface import dateformat +from pytz import UTC + class ResponseTest(unittest.TestCase): """ Base class for tests of capa responses.""" @@ -333,8 +335,9 @@ class SymbolicResponseTest(ResponseTest): correct_map = problem.grade_answers(input_dict) - self.assertEqual(correct_map.get_correctness('1_2_1'), - expected_correctness) + self.assertEqual( + correct_map.get_correctness('1_2_1'), expected_correctness + ) class OptionResponseTest(ResponseTest): @@ -702,7 +705,7 @@ class CodeResponseTest(ResponseTest): # Now we queue the LCP cmap = CorrectMap() for i, answer_id in enumerate(answer_ids): - queuestate = CodeResponseTest.make_queuestate(i, datetime.now()) + queuestate = CodeResponseTest.make_queuestate(i, datetime.now(UTC)) cmap.update(CorrectMap(answer_id=answer_ids[i], queuestate=queuestate)) self.problem.correct_map.update(cmap) @@ -718,7 +721,7 @@ class CodeResponseTest(ResponseTest): old_cmap = CorrectMap() for i, answer_id in enumerate(answer_ids): queuekey = 1000 + i - queuestate = CodeResponseTest.make_queuestate(queuekey, datetime.now()) + queuestate = CodeResponseTest.make_queuestate(queuekey, datetime.now(UTC)) old_cmap.update(CorrectMap(answer_id=answer_ids[i], queuestate=queuestate)) # Message format common to external graders @@ -778,13 +781,15 @@ class CodeResponseTest(ResponseTest): cmap = CorrectMap() for i, answer_id in enumerate(answer_ids): queuekey = 1000 + i - latest_timestamp = datetime.now() + latest_timestamp = datetime.now(UTC) queuestate = CodeResponseTest.make_queuestate(queuekey, latest_timestamp) cmap.update(CorrectMap(answer_id=answer_id, queuestate=queuestate)) self.problem.correct_map.update(cmap) # Queue state only tracks up to second - latest_timestamp = datetime.strptime(datetime.strftime(latest_timestamp, dateformat), dateformat) + latest_timestamp = datetime.strptime( + datetime.strftime(latest_timestamp, dateformat), dateformat + ).replace(tzinfo=UTC) self.assertEquals(self.problem.get_recentmost_queuetime(), latest_timestamp) diff --git a/common/lib/capa/capa/xqueue_interface.py b/common/lib/capa/capa/xqueue_interface.py index 5cf2488af0..4da8e11d53 100644 --- a/common/lib/capa/capa/xqueue_interface.py +++ b/common/lib/capa/capa/xqueue_interface.py @@ -30,9 +30,11 @@ def make_xheader(lms_callback_url, lms_key, queue_name): 'queue_name': designate a specific queue within xqueue server, e.g. 'MITx-6.00x' (string) } """ - return json.dumps({'lms_callback_url': lms_callback_url, - 'lms_key': lms_key, - 'queue_name': queue_name}) + return json.dumps({ + 'lms_callback_url': lms_callback_url, + 'lms_key': lms_key, + 'queue_name': queue_name + }) def parse_xreply(xreply): @@ -60,7 +62,7 @@ class XQueueInterface(object): ''' def __init__(self, url, django_auth, requests_auth=None): - self.url = url + self.url = url self.auth = django_auth self.session = requests.session(auth=requests_auth) @@ -95,13 +97,13 @@ class XQueueInterface(object): return (error, msg) - def _login(self): - payload = {'username': self.auth['username'], - 'password': self.auth['password']} + payload = { + 'username': self.auth['username'], + 'password': self.auth['password'] + } return self._http_post(self.url + '/xqueue/login/', payload) - def _send_to_queue(self, header, body, files_to_upload): payload = {'xqueue_header': header, 'xqueue_body': body} @@ -112,7 +114,6 @@ class XQueueInterface(object): return self._http_post(self.url + '/xqueue/submit/', payload, files=files) - def _http_post(self, url, data, files=None): try: r = self.session.post(url, data=data, files=files) diff --git a/common/lib/xmodule/xmodule/capa_module.py b/common/lib/xmodule/xmodule/capa_module.py index d76b62dc06..8489b5f986 100644 --- a/common/lib/xmodule/xmodule/capa_module.py +++ b/common/lib/xmodule/xmodule/capa_module.py @@ -309,7 +309,13 @@ class CapaModule(CapaFields, XModule): d = self.get_score() score = d['score'] total = d['total'] + if total > 0: + if self.weight is not None: + # scale score and total by weight/total: + score = score * self.weight / total + total = self.weight + try: return Progress(score, total) except (TypeError, ValueError): @@ -321,11 +327,13 @@ class CapaModule(CapaFields, XModule): """ Return some html with data about the module """ + progress = self.get_progress() return self.system.render_template('problem_ajax.html', { 'element_id': self.location.html_id(), 'id': self.id, 'ajax_url': self.system.ajax_url, - 'progress': Progress.to_js_status_str(self.get_progress()) + 'progress_status': Progress.to_js_status_str(progress), + 'progress_detail': Progress.to_js_detail_str(progress), }) def check_button_name(self): @@ -485,8 +493,7 @@ class CapaModule(CapaFields, XModule): """ Return html for the problem. - Adds check, reset, save buttons as necessary based on the problem config - and state. + Adds check, reset, save buttons as necessary based on the problem config and state. """ try: @@ -516,13 +523,12 @@ class CapaModule(CapaFields, XModule): 'reset_button': self.should_show_reset_button(), 'save_button': self.should_show_save_button(), 'answer_available': self.answer_available(), - 'ajax_url': self.system.ajax_url, 'attempts_used': self.attempts, 'attempts_allowed': self.max_attempts, - 'progress': self.get_progress(), } html = self.system.render_template('problem.html', context) + if encapsulate: html = u'
      '.format( id=self.location.html_id(), ajax_url=self.system.ajax_url @@ -584,6 +590,7 @@ class CapaModule(CapaFields, XModule): result.update({ 'progress_changed': after != before, 'progress_status': Progress.to_js_status_str(after), + 'progress_detail': Progress.to_js_detail_str(after), }) return json.dumps(result, cls=ComplexEncoder) @@ -614,6 +621,7 @@ class CapaModule(CapaFields, XModule): Problem can be completely wrong. Pressing RESET button makes this function to return False. """ + # used by conditional module return self.lcp.done def is_attempted(self): @@ -757,6 +765,7 @@ class CapaModule(CapaFields, XModule): """ return {'html': self.get_problem_html(encapsulate=False)} + @staticmethod def make_dict_of_responses(data): """ @@ -1117,8 +1126,12 @@ class CapaDescriptor(CapaFields, RawDescriptor): mako_template = "widgets/problem-edit.html" js = {'coffee': [resource_string(__name__, 'js/src/problem/edit.coffee')]} js_module_name = "MarkdownEditingDescriptor" - css = {'scss': [resource_string(__name__, 'css/editor/edit.scss'), - resource_string(__name__, 'css/problem/edit.scss')]} + css = { + 'scss': [ + resource_string(__name__, 'css/editor/edit.scss'), + resource_string(__name__, 'css/problem/edit.scss') + ] + } # Capa modules have some additional metadata: # TODO (vshnayder): do problems have any other metadata? Do they @@ -1146,19 +1159,6 @@ class CapaDescriptor(CapaFields, RawDescriptor): path[8:], ] - @classmethod - def from_xml(cls, xml_data, system, org=None, course=None): - """ - Augment regular translation w/ setting the pre-Studio defaults. - """ - problem = super(CapaDescriptor, cls).from_xml(xml_data, system, org, course) - # pylint: disable=W0212 - if 'showanswer' not in problem._model_data: - problem.showanswer = "closed" - if 'rerandomize' not in problem._model_data: - problem.rerandomize = "always" - return problem - @property def non_editable_metadata_fields(self): non_editable_fields = super(CapaDescriptor, self).non_editable_metadata_fields diff --git a/common/lib/xmodule/xmodule/course_module.py b/common/lib/xmodule/xmodule/course_module.py index d4aac5b0ae..3c9f5c0683 100644 --- a/common/lib/xmodule/xmodule/course_module.py +++ b/common/lib/xmodule/xmodule/course_module.py @@ -15,6 +15,7 @@ import json from xblock.core import Scope, List, String, Dict, Boolean from .fields import Date +from xmodule.modulestore.locator import CourseLocator from django.utils.timezone import UTC from xmodule.util import date_utils @@ -192,9 +193,8 @@ class CourseFields(object): }}, scope=Scope.content) show_calculator = Boolean(help="Whether to show the calculator in this course", default=False, scope=Scope.settings) - display_name = String( - help="Display name for this module", default="Empty", - display_name="Display Name", scope=Scope.settings) + display_name = String(help="Display name for this module", default="Empty", display_name="Display Name", scope=Scope.settings) + show_chat = Boolean(help="Whether to show the chat widget in this course", default=False, scope=Scope.settings) tabs = List(help="List of tabs to enable in this course", scope=Scope.settings) end_of_course_survey_url = String(help="Url for the end-of-course survey", scope=Scope.settings) discussion_blackouts = List(help="List of pairs of start/end dates for discussion blackouts", scope=Scope.settings) @@ -373,7 +373,10 @@ class CourseDescriptor(CourseFields, SequenceDescriptor): super(CourseDescriptor, self).__init__(*args, **kwargs) if self.wiki_slug is None: - self.wiki_slug = self.location.course + if isinstance(self.location, Location): + self.wiki_slug = self.location.course + elif isinstance(self.location, CourseLocator): + self.wiki_slug = self.location.course_id or self.display_name msg = None @@ -407,7 +410,7 @@ class CourseDescriptor(CourseFields, SequenceDescriptor): continue # TODO check that this is still needed here and can't be by defaults. - if self.tabs is None: + if not self.tabs: # When calling the various _tab methods, can omit the 'type':'blah' from the # first arg, since that's only used for dispatch tabs = [] diff --git a/common/lib/xmodule/xmodule/css/capa/display.scss b/common/lib/xmodule/xmodule/css/capa/display.scss index 9e6826242f..48912795f0 100644 --- a/common/lib/xmodule/xmodule/css/capa/display.scss +++ b/common/lib/xmodule/xmodule/css/capa/display.scss @@ -3,6 +3,7 @@ h2 { margin-bottom: 15px; &.problem-header { + display: inline-block; section.staff { margin-top: 30px; font-size: 80%; @@ -28,6 +29,13 @@ iframe[seamless]{ color: darken($error-red, 11%); } +section.problem-progress { + display: inline-block; + color: #999; + font-size: em(16); + font-weight: 100; + padding-left: 5px; +} section.problem { @media print { diff --git a/common/lib/xmodule/xmodule/css/sequence/display.scss b/common/lib/xmodule/xmodule/css/sequence/display.scss index 4baf60a8fe..1e82b31f10 100644 --- a/common/lib/xmodule/xmodule/css/sequence/display.scss +++ b/common/lib/xmodule/xmodule/css/sequence/display.scss @@ -211,6 +211,8 @@ nav.sequence-nav { @include transition(all .1s $ease-in-out-quart 0s); white-space: pre; z-index: 99; + visibility: hidden; + pointer-events: none; &:empty { background: none; @@ -238,6 +240,7 @@ nav.sequence-nav { display: block; margin-top: 4px; opacity: 1.0; + visibility: visible; } } } diff --git a/common/lib/xmodule/xmodule/discussion_module.py b/common/lib/xmodule/xmodule/discussion_module.py index fac6a498e5..2c6cc9f4dd 100644 --- a/common/lib/xmodule/xmodule/discussion_module.py +++ b/common/lib/xmodule/xmodule/discussion_module.py @@ -12,10 +12,14 @@ class DiscussionFields(object): display_name = String( display_name="Display Name", help="Display name for this module", - default="Discussion Tag", - scope=Scope.settings) - data = String(help="XML data for the problem", scope=Scope.content, - default="") + default="Discussion", + scope=Scope.settings + ) + data = String( + help="XML data for the problem", + scope=Scope.content, + default="" + ) discussion_category = String( display_name="Category", default="Week 1", diff --git a/common/lib/xmodule/xmodule/error_module.py b/common/lib/xmodule/xmodule/error_module.py index db1f998187..e7483f485a 100644 --- a/common/lib/xmodule/xmodule/error_module.py +++ b/common/lib/xmodule/xmodule/error_module.py @@ -79,8 +79,10 @@ class ErrorDescriptor(ErrorFields, JSONEditingDescriptor): @classmethod def _construct(cls, system, contents, error_msg, location): - if location.name is None: - location = location._replace( + if isinstance(location, dict) and 'course' in location: + location = Location(location) + if isinstance(location, Location) and location.name is None: + location = location.replace( category='error', # Pick a unique url_name -- the sha1 hash of the contents. # NOTE: We could try to pull out the url_name of the errored descriptor, @@ -94,7 +96,7 @@ class ErrorDescriptor(ErrorFields, JSONEditingDescriptor): model_data = { 'error_msg': str(error_msg), 'contents': contents, - 'display_name': 'Error: ' + location.name, + 'display_name': 'Error: ' + location.url(), 'location': location, 'category': 'error' } diff --git a/common/lib/xmodule/xmodule/fields.py b/common/lib/xmodule/xmodule/fields.py index 465993a51f..dc2f000286 100644 --- a/common/lib/xmodule/xmodule/fields.py +++ b/common/lib/xmodule/xmodule/fields.py @@ -80,6 +80,7 @@ class Date(ModelType): TIMEDELTA_REGEX = re.compile(r'^((?P\d+?) day(?:s?))?(\s)?((?P\d+?) hour(?:s?))?(\s)?((?P\d+?) minute(?:s)?)?(\s)?((?P\d+?) second(?:s)?)?$') + class Timedelta(ModelType): def from_json(self, time_str): """ diff --git a/common/lib/xmodule/xmodule/foldit_module.py b/common/lib/xmodule/xmodule/foldit_module.py index c4e9e2d35c..cadf6cef0b 100644 --- a/common/lib/xmodule/xmodule/foldit_module.py +++ b/common/lib/xmodule/xmodule/foldit_module.py @@ -91,15 +91,18 @@ class FolditModule(FolditFields, XModule): PuzzleComplete.completed_puzzles(self.system.anonymous_student_id), key=lambda d: (d['set'], d['subset'])) - def puzzle_leaders(self, n=10): + def puzzle_leaders(self, n=10, courses=None): """ Returns a list of n pairs (user, score) corresponding to the top scores; the pairs are in descending order of score. """ from foldit.models import Score - leaders = [(e['username'], e['score']) for e in Score.get_tops_n(10)] - leaders.sort(key=lambda x:-x[1]) + if courses is None: + courses = [self.location.course_id] + + leaders = [(leader['username'], leader['score']) for leader in Score.get_tops_n(10, course_list=courses)] + leaders.sort(key=lambda x: -x[1]) return leaders diff --git a/common/lib/xmodule/xmodule/gst_module.py b/common/lib/xmodule/xmodule/gst_module.py index 5a4930ff95..5c902f48c2 100644 --- a/common/lib/xmodule/xmodule/gst_module.py +++ b/common/lib/xmodule/xmodule/gst_module.py @@ -19,10 +19,45 @@ from xblock.core import String, Scope log = logging.getLogger(__name__) +DEFAULT_RENDER=""" +

      Graphic slider tool: Dynamic range and implicit functions.

      + +

      You can make the range of the x axis (but not ticks of x axis) of + functions depend on a parameter value. This can be useful when the + function domain needs to be variable.

      +

      Implicit functions like a circle can be plotted as 2 separate + functions of the same color.

      +
      + + +
      + +""" +DEFAULT_CONFIGURATION=""" + + + + + Math.sqrt(r * r - x * x) + -Math.sqrt(r * r - x * x) + + + + + -r + r + + 1000 + -30, 6, 30 + -30, 6, 30 + +""" + + class GraphicalSliderToolFields(object): - render = String(scope=Scope.content) - configuration = String(scope=Scope.content) + render = String(scope=Scope.content, default=DEFAULT_RENDER) + configuration = String(scope=Scope.content, default=DEFAULT_CONFIGURATION) class GraphicalSliderToolModule(GraphicalSliderToolFields, XModule): diff --git a/common/lib/xmodule/xmodule/html_module.py b/common/lib/xmodule/xmodule/html_module.py index 0536f5ef5e..1cc7bed948 100644 --- a/common/lib/xmodule/xmodule/html_module.py +++ b/common/lib/xmodule/xmodule/html_module.py @@ -25,9 +25,9 @@ class HtmlFields(object): scope=Scope.settings, # it'd be nice to have a useful default but it screws up other things; so, # use display_name_with_default for those - default="Blank HTML Page" + default="Text" ) - data = String(help="Html contents to display for this module", default="", scope=Scope.content) + data = String(help="Html contents to display for this module", default=u"", scope=Scope.content) source_code = String(help="Source code for LaTeX documents. This feature is not well-supported.", scope=Scope.settings) diff --git a/common/lib/xmodule/xmodule/js/fixtures/problem.html b/common/lib/xmodule/xmodule/js/fixtures/problem.html index 525b4323b7..07e147a9e7 100644 --- a/common/lib/xmodule/xmodule/js/fixtures/problem.html +++ b/common/lib/xmodule/xmodule/js/fixtures/problem.html @@ -1,6 +1,6 @@
      diff --git a/common/lib/xmodule/xmodule/js/fixtures/problem_content.html b/common/lib/xmodule/xmodule/js/fixtures/problem_content.html index 3e580be722..5ccce952e7 100644 --- a/common/lib/xmodule/xmodule/js/fixtures/problem_content.html +++ b/common/lib/xmodule/xmodule/js/fixtures/problem_content.html @@ -1,5 +1,8 @@

      Problem Header

      +
      +
      +

      Problem Content

      diff --git a/common/lib/xmodule/xmodule/js/spec/capa/display_spec.coffee b/common/lib/xmodule/xmodule/js/spec/capa/display_spec.coffee index bca89b0dea..33d74e2335 100644 --- a/common/lib/xmodule/xmodule/js/spec/capa/display_spec.coffee +++ b/common/lib/xmodule/xmodule/js/spec/capa/display_spec.coffee @@ -77,6 +77,25 @@ describe 'Problem', -> [@problem.updateMathML, @stubbedJax, $('#input_example_1').get(0)] ] + describe 'renderProgressState', -> + beforeEach -> + @problem = new Problem($('.xmodule_display')) + #@renderProgressState = @problem.renderProgressState + + describe 'with a status of "none"', -> + it 'reports the number of points possible', -> + @problem.el.data('progress_status', 'none') + @problem.el.data('progress_detail', '0/1') + @problem.renderProgressState() + expect(@problem.$('.problem-progress').html()).toEqual "(1 point possible)" + + describe 'with any other valid status', -> + it 'reports the current score', -> + @problem.el.data('progress_status', 'foo') + @problem.el.data('progress_detail', '1/1') + @problem.renderProgressState() + expect(@problem.$('.problem-progress').html()).toEqual "(1/1 points)" + describe 'render', -> beforeEach -> @problem = new Problem($('.xmodule_display')) diff --git a/common/lib/xmodule/xmodule/js/src/capa/display.coffee b/common/lib/xmodule/xmodule/js/src/capa/display.coffee index 601fb749ac..09a398dd7a 100644 --- a/common/lib/xmodule/xmodule/js/src/capa/display.coffee +++ b/common/lib/xmodule/xmodule/js/src/capa/display.coffee @@ -35,15 +35,34 @@ class @Problem @$('input.math').each (index, element) => MathJax.Hub.Queue [@refreshMath, null, element] + renderProgressState: => + detail = @el.data('progress_detail') + status = @el.data('progress_status') + # i18n + progress = "(#{detail} points)" + if status == 'none' and detail? and detail.indexOf('/') > 0 + a = detail.split('/') + possible = parseInt(a[1]) + if possible == 1 + # i18n + progress = "(#{possible} point possible)" + else + # i18n + progress = "(#{possible} points possible)" + @$('.problem-progress').html(progress) + updateProgress: (response) => if response.progress_changed - @el.attr progress: response.progress_status + @el.data('progress_status', response.progress_status) + @el.data('progress_detail', response.progress_detail) @el.trigger('progressChanged') + @renderProgressState() forceUpdate: (response) => - @el.attr progress: response.progress_status + @el.data('progress_status', response.progress_status) + @el.data('progress_detail', response.progress_detail) @el.trigger('progressChanged') - + @renderProgressState() queueing: => @queued_items = @$(".xqueue") @@ -113,7 +132,7 @@ class @Problem @setupInputTypes() @bind() @queueing() - + @forceUpdate response # TODO add hooks for problem types here by inspecting response.html and doing # stuff if a div w a class is found diff --git a/common/lib/xmodule/xmodule/js/src/sequence/display.coffee b/common/lib/xmodule/xmodule/js/src/sequence/display.coffee index 495b734785..bae1b89bab 100644 --- a/common/lib/xmodule/xmodule/js/src/sequence/display.coffee +++ b/common/lib/xmodule/xmodule/js/src/sequence/display.coffee @@ -45,7 +45,7 @@ class @Sequence new_progress = "NA" _this = this $('.problems-wrapper').each (index) -> - progress = $(this).attr 'progress' + progress = $(this).data 'progress_status' new_progress = _this.mergeProgress progress, new_progress @progressTable[@position] = new_progress diff --git a/common/lib/xmodule/xmodule/modulestore/exceptions.py b/common/lib/xmodule/xmodule/modulestore/exceptions.py index a63efc3e43..508599b677 100644 --- a/common/lib/xmodule/xmodule/modulestore/exceptions.py +++ b/common/lib/xmodule/xmodule/modulestore/exceptions.py @@ -7,10 +7,18 @@ class ItemNotFoundError(Exception): pass +class ItemWriteConflictError(Exception): + pass + + class InsufficientSpecificationError(Exception): pass +class OverSpecificationError(Exception): + pass + + class InvalidLocationError(Exception): pass @@ -21,3 +29,13 @@ class NoPathToItem(Exception): class DuplicateItemError(Exception): pass + + +class VersionConflictError(Exception): + """ + The caller asked for either draft or published head and gave a version which conflicted with it. + """ + def __init__(self, requestedLocation, currentHead): + super(VersionConflictError, self).__init__() + self.requestedLocation = requestedLocation + self.currentHead = currentHead diff --git a/common/lib/xmodule/xmodule/modulestore/inheritance.py b/common/lib/xmodule/xmodule/modulestore/inheritance.py index a816aa9776..1314c72094 100644 --- a/common/lib/xmodule/xmodule/modulestore/inheritance.py +++ b/common/lib/xmodule/xmodule/modulestore/inheritance.py @@ -50,6 +50,8 @@ def inherit_metadata(descriptor, model_data): def own_metadata(module): + # IN SPLIT MONGO this is just ['metadata'] as it keeps ['_inherited_metadata'] separate! + # FIXME move into kvs? will that work for xml mongo? """ Return a dictionary that contains only non-inherited field keys, mapped to their values diff --git a/common/lib/xmodule/xmodule/modulestore/locator.py b/common/lib/xmodule/xmodule/modulestore/locator.py new file mode 100644 index 0000000000..928bc9f133 --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/locator.py @@ -0,0 +1,465 @@ +""" +Created on Mar 13, 2013 + +@author: dmitchell +""" +from __future__ import absolute_import +import logging +import inspect +from abc import ABCMeta, abstractmethod +from urllib import quote + +from bson.objectid import ObjectId +from bson.errors import InvalidId + +from xmodule.modulestore.exceptions import InsufficientSpecificationError, OverSpecificationError + +from .parsers import parse_url, parse_course_id, parse_block_ref + +log = logging.getLogger(__name__) + + +class Locator(object): + """ + A locator is like a URL, it refers to a course resource. + + Locator is an abstract base class: do not instantiate + """ + + __metaclass__ = ABCMeta + + @abstractmethod + def url(self): + """ + Return a string containing the URL for this location. Raises + InsufficientSpecificationError if the instance doesn't have a + complete enough specification to generate a url + """ + raise InsufficientSpecificationError() + + def quoted_url(self): + return quote(self.url(), '@;#') + + def __eq__(self, other): + return self.__dict__ == other.__dict__ + + def __repr__(self): + ''' + repr(self) returns something like this: CourseLocator("edu.mit.eecs.6002x") + ''' + classname = self.__class__.__name__ + if classname.find('.') != -1: + classname = classname.split['.'][-1] + return '%s("%s")' % (classname, unicode(self)) + + def __str__(self): + ''' + str(self) returns something like this: "edu.mit.eecs.6002x" + ''' + return unicode(self).encode('utf8') + + def __unicode__(self): + ''' + unicode(self) returns something like this: "edu.mit.eecs.6002x" + ''' + return self.url() + + @abstractmethod + def version(self): + """ + Returns the ObjectId referencing this specific location. + Raises InsufficientSpecificationError if the instance + doesn't have a complete enough specification. + """ + raise InsufficientSpecificationError() + + def set_property(self, property_name, new): + """ + Initialize property to new value. + If property has already been initialized to a different value, raise an exception. + """ + current = getattr(self, property_name) + if current and current != new: + raise OverSpecificationError('%s cannot be both %s and %s' % + (property_name, current, new)) + setattr(self, property_name, new) + + +class CourseLocator(Locator): + """ + Examples of valid CourseLocator specifications: + CourseLocator(version_guid=ObjectId('519665f6223ebd6980884f2b')) + CourseLocator(course_id='edu.mit.eecs.6002x') + CourseLocator(course_id='edu.mit.eecs.6002x;published') + CourseLocator(course_id='edu.mit.eecs.6002x', revision='published') + CourseLocator(url='edx://@519665f6223ebd6980884f2b') + CourseLocator(url='edx://edu.mit.eecs.6002x') + CourseLocator(url='edx://edu.mit.eecs.6002x;published') + + Should have at lease a specific course_id (id for the course as if it were a project w/ + versions) with optional 'revision' (must be 'draft', 'published', or None), + or version_guid (which points to a specific version). Can contain both in which case + the persistence layer may raise exceptions if the given version != the current such version + of the course. + """ + + # Default values + version_guid = None + course_id = None + revision = None + + def __unicode__(self): + """ + Return a string representing this location. + """ + if self.course_id: + result = self.course_id + if self.revision: + result += ';' + self.revision + return result + elif self.version_guid: + return '@' + str(self.version_guid) + else: + # raise InsufficientSpecificationError("missing course_id or version_guid") + return '' + + def url(self): + """ + Return a string containing the URL for this location. + """ + return 'edx://' + unicode(self) + + # -- unused args which are used via inspect + # pylint: disable= W0613 + def validate_args(self, url, version_guid, course_id, revision): + """ + Validate provided arguments. + """ + need_oneof = set(('url', 'version_guid', 'course_id')) + args, _, _, values = inspect.getargvalues(inspect.currentframe()) + provided_args = [a for a in args if a != 'self' and values[a] is not None] + if len(need_oneof.intersection(provided_args)) == 0: + raise InsufficientSpecificationError("Must provide one of these args: %s " % + list(need_oneof)) + + def is_fully_specified(self): + """ + Returns True if either version_guid is specified, or course_id+revision + are specified. + This should always return True, since this should be validated in the constructor. + """ + return self.version_guid is not None \ + or (self.course_id is not None and self.revision is not None) + + def set_course_id(self, new): + """ + Initialize course_id to new value. + If course_id has already been initialized to a different value, raise an exception. + """ + self.set_property('course_id', new) + + def set_revision(self, new): + """ + Initialize revision to new value. + If revision has already been initialized to a different value, raise an exception. + """ + self.set_property('revision', new) + + def set_version_guid(self, new): + """ + Initialize version_guid to new value. + If version_guid has already been initialized to a different value, raise an exception. + """ + self.set_property('version_guid', new) + + def as_course_locator(self): + """ + Returns a copy of itself (downcasting) as a CourseLocator. + The copy has the same CourseLocator fields as the original. + The copy does not include subclass information, such as + a usage_id (a property of BlockUsageLocator). + """ + return CourseLocator(course_id=self.course_id, + version_guid=self.version_guid, + revision=self.revision) + + def __init__(self, url=None, version_guid=None, course_id=None, revision=None): + """ + Construct a CourseLocator + Caller may provide url (but no other parameters). + Caller may provide version_guid (but no other parameters). + Caller may provide course_id (optionally provide revision). + + Resulting CourseLocator will have either a version_guid property + or a course_id (with optional revision) property, or both. + + version_guid must be an instance of bson.objectid.ObjectId or None + url, course_id, and revision must be strings or None + + """ + self.validate_args(url, version_guid, course_id, revision) + if url: + self.init_from_url(url) + if version_guid: + self.init_from_version_guid(version_guid) + if course_id or revision: + self.init_from_course_id(course_id, revision) + assert self.version_guid or self.course_id, \ + "Either version_guid or course_id should be set." + + @classmethod + def as_object_id(cls, value): + """ + Attempts to cast value as a bson.objectid.ObjectId. + If cast fails, raises ValueError + """ + if isinstance(value, ObjectId): + return value + try: + return ObjectId(value) + except InvalidId: + raise ValueError('"%s" is not a valid version_guid' % value) + + def init_from_url(self, url): + """ + url must be a string beginning with 'edx://' and containing + either a valid version_guid or course_id (with optional revision) + If a block ('#HW3') is present, it is ignored. + """ + if isinstance(url, Locator): + url = url.url() + assert isinstance(url, basestring), \ + '%s is not an instance of basestring' % url + parse = parse_url(url) + assert parse, 'Could not parse "%s" as a url' % url + if 'version_guid' in parse: + new_guid = parse['version_guid'] + self.set_version_guid(self.as_object_id(new_guid)) + else: + self.set_course_id(parse['id']) + self.set_revision(parse['revision']) + + def init_from_version_guid(self, version_guid): + """ + version_guid must be an instance of bson.objectid.ObjectId, + or able to be cast as one. + If it's a string, attempt to cast it as an ObjectId first. + """ + version_guid = self.as_object_id(version_guid) + + assert isinstance(version_guid, ObjectId), \ + '%s is not an instance of ObjectId' % version_guid + self.set_version_guid(version_guid) + + def init_from_course_id(self, course_id, explicit_revision=None): + """ + Course_id is a string like 'edu.mit.eecs.6002x' or 'edu.mit.eecs.6002x;published'. + + Revision (optional) is a string like 'published'. + It may be provided explicitly (explicit_revision) or embedded into course_id. + If revision is part of course_id ("...;published"), parse it out separately. + If revision is provided both ways, that's ok as long as they are the same value. + + If a block ('#HW3') is a part of course_id, it is ignored. + + """ + + if course_id: + if isinstance(course_id, CourseLocator): + course_id = course_id.course_id + assert course_id, "%s does not have a valid course_id" + + parse = parse_course_id(course_id) + assert parse, 'Could not parse "%s" as a course_id' % course_id + self.set_course_id(parse['id']) + rev = parse['revision'] + if rev: + self.set_revision(rev) + if explicit_revision: + self.set_revision(explicit_revision) + + def version(self): + """ + Returns the ObjectId referencing this specific location. + """ + return self.version_guid + + def html_id(self): + """ + Generate a discussion group id based on course + + To make compatible with old Location object functionality. I don't believe this behavior fits at this + place, but I have no way to override. If this is really needed, it should probably use the pretty_id to seed + the name although that's mutable. We should also clearly define the purpose and restrictions of this + (e.g., I'm assuming periods are fine). + """ + return self.course_id + + +class BlockUsageLocator(CourseLocator): + """ + Encodes a location. + + Locations address modules (aka blocks) which are definitions situated in a + course instance. Thus, a Location must identify the course and the occurrence of + the defined element in the course. Courses can be a version of an offering, the + current draft head, or the current production version. + + Locators can contain both a version and a course_id w/ revision. The split mongo functions + may raise errors if these conflict w/ the current db state (i.e., the course's revision != + the version_guid) + + Locations can express as urls as well as dictionaries. They consist of + course_identifier: course_guid | version_guid + block : guid + revision : 'draft' | 'published' (optional) + """ + + # Default value + usage_id = None + + def __init__(self, url=None, version_guid=None, course_id=None, + revision=None, usage_id=None): + """ + Construct a BlockUsageLocator + Caller may provide url, version_guid, or course_id, and optionally provide revision. + + The usage_id may be specified, either explictly or as part of + the url or course_id. If omitted, the locator is created but it + has not yet been initialized. + + Resulting BlockUsageLocator will have a usage_id property. + It will have either a version_guid property or a course_id (with optional revision) property, or both. + + version_guid must be an instance of bson.objectid.ObjectId or None + url, course_id, revision, and usage_id must be strings or None + + """ + self.validate_args(url, version_guid, course_id, revision) + if url: + self.init_block_ref_from_url(url) + if course_id: + self.init_block_ref_from_course_id(course_id) + if usage_id: + self.init_block_ref(usage_id) + CourseLocator.__init__(self, + url=url, + version_guid=version_guid, + course_id=course_id, + revision=revision) + + def is_initialized(self): + """ + Returns True if usage_id has been initialized, else returns False + """ + return self.usage_id is not None + + def version_agnostic(self): + """ + Returns a copy of itself. + If both version_guid and course_id are known, use a blank course_id in the copy. + + We don't care if the locator's version is not the current head; so, avoid version conflict + by reducing info. + + :param block_locator: + """ + if self.course_id and self.version_guid: + return BlockUsageLocator(version_guid=self.version_guid, + revision=self.revision, + usage_id=self.usage_id) + else: + return BlockUsageLocator(course_id=self.course_id, + revision=self.revision, + usage_id=self.usage_id) + + def set_usage_id(self, new): + """ + Initialize usage_id to new value. + If usage_id has already been initialized to a different value, raise an exception. + """ + self.set_property('usage_id', new) + + def init_block_ref(self, block_ref): + parse = parse_block_ref(block_ref) + assert parse, 'Could not parse "%s" as a block_ref' % block_ref + self.set_usage_id(parse['block']) + + def init_block_ref_from_url(self, url): + if isinstance(url, Locator): + url = url.url() + parse = parse_url(url) + assert parse, 'Could not parse "%s" as a url' % url + block = parse.get('block', None) + if block: + self.set_usage_id(block) + + def init_block_ref_from_course_id(self, course_id): + if isinstance(course_id, CourseLocator): + course_id = course_id.course_id + assert course_id, "%s does not have a valid course_id" + parse = parse_course_id(course_id) + assert parse, 'Could not parse "%s" as a course_id' % course_id + block = parse.get('block', None) + if block: + self.set_usage_id(block) + + def __unicode__(self): + """ + Return a string representing this location. + """ + rep = CourseLocator.__unicode__(self) + if self.usage_id is None: + # usage_id has not been initialized + return rep + '#NONE' + else: + return rep + '#' + self.usage_id + + +class DescriptionLocator(Locator): + """ + Container for how to locate a description + """ + + def __init__(self, definition_id): + self.definition_id = definition_id + + def __unicode__(self): + ''' + Return a string representing this location. + unicode(self) returns something like this: "@519665f6223ebd6980884f2b" + ''' + return '@' + str(self.definition_guid) + + def url(self): + """ + Return a string containing the URL for this location. + url(self) returns something like this: 'edx://@519665f6223ebd6980884f2b' + """ + return 'edx://' + unicode(self) + + def version(self): + """ + Returns the ObjectId referencing this specific location. + """ + return self.definition_guid + + +class VersionTree(object): + """ + Holds trees of Locators to represent version histories. + """ + def __init__(self, locator, tree_dict=None): + """ + :param locator: must be version specific (Course has version_guid or definition had id) + """ + assert isinstance(locator, Locator) and not inspect.isabstract(locator), \ + "locator must be a concrete subclass of Locator" + assert locator.version(), \ + "locator must be version specific (Course has version_guid or definition had id)" + self.locator = locator + if tree_dict is None: + self.children = [] + else: + self.children = [VersionTree(child, tree_dict) + for child in tree_dict.get(locator.version(), [])] diff --git a/common/lib/xmodule/xmodule/modulestore/mongo/base.py b/common/lib/xmodule/xmodule/modulestore/mongo/base.py index ae879ba3e8..bedba5d65d 100644 --- a/common/lib/xmodule/xmodule/modulestore/mongo/base.py +++ b/common/lib/xmodule/xmodule/modulestore/mongo/base.py @@ -105,15 +105,6 @@ class MongoKeyValueStore(KeyValueStore): else: raise InvalidScopeError(key.scope) - def set_many(self, update_dict): - """set_many method. Implementations should accept an `update_dict` of - key-value pairs, and set all the `keys` to the given `value`s.""" - # `set` simply updates an in-memory db, rather than calling down to a real db, - # as mongo bulk save is handled elsewhere. A future improvement would be to pull - # the mongo-specific bulk save logic into this method. - for key, value in update_dict.iteritems(): - self.set(key, value) - def delete(self, key): if key.scope == Scope.children: self._children = [] diff --git a/common/lib/xmodule/xmodule/modulestore/parsers.py b/common/lib/xmodule/xmodule/modulestore/parsers.py new file mode 100644 index 0000000000..9aac3073ae --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/parsers.py @@ -0,0 +1,115 @@ +import re + +URL_RE = re.compile(r'^edx://(.+)$', re.IGNORECASE) + + +def parse_url(string): + """ + A url must begin with 'edx://' (case-insensitive match), + followed by either a version_guid or a course_id. + + Examples: + 'edx://@0123FFFF' + 'edx://edu.mit.eecs.6002x' + 'edx://edu.mit.eecs.6002x;published' + 'edx://edu.mit.eecs.6002x;published#HW3' + + This returns None if string cannot be parsed. + + If it can be parsed as a version_guid, returns a dict + with key 'version_guid' and the value, + + If it can be parsed as a course_id, returns a dict + with keys 'id' and 'revision' (value of 'revision' may be None), + + """ + match = URL_RE.match(string) + if not match: + return None + path = match.group(1) + if path[0] == '@': + return parse_guid(path[1:]) + return parse_course_id(path) + + +BLOCK_RE = re.compile(r'^\w+$', re.IGNORECASE) + + +def parse_block_ref(string): + r""" + A block_ref is a string of word_chars. + + matches one or more Unicode word characters; this includes most + characters that can be part of a word in any language, as well as numbers + and the underscore. (see definition of \w in python regular expressions, + at http://docs.python.org/dev/library/re.html) + + If string is a block_ref, returns a dict with key 'block_ref' and the value, + otherwise returns None. + """ + if len(string) > 0 and BLOCK_RE.match(string): + return {'block': string} + return None + + +GUID_RE = re.compile(r'^(?P[A-F0-9]+)(#(?P\w+))?$', re.IGNORECASE) + + +def parse_guid(string): + """ + A version_guid is a string of hex digits (0-F). + + If string is a version_guid, returns a dict with key 'version_guid' and the value, + otherwise returns None. + """ + m = GUID_RE.match(string) + if m is not None: + return m.groupdict() + else: + return None + + +COURSE_ID_RE = re.compile(r'^(?P(\w+)(\.\w+\w*)*)(;(?P\w+))?(#(?P\w+))?$', re.IGNORECASE) + + +def parse_course_id(string): + r""" + + A course_id has a main id component. + There may also be an optional revision (;published or ;draft). + There may also be an optional block (#HW3 or #Quiz2). + + Examples of valid course_ids: + + 'edu.mit.eecs.6002x' + 'edu.mit.eecs.6002x;published' + 'edu.mit.eecs.6002x#HW3' + 'edu.mit.eecs.6002x;published#HW3' + + + Syntax: + + course_id = main_id [; revision] [# block] + + main_id = name [. name]* + + revision = name + + block = name + + name = + + matches one or more Unicode word characters; this includes most + characters that can be part of a word in any language, as well as numbers + and the underscore. (see definition of \w in python regular expressions, + at http://docs.python.org/dev/library/re.html) + + If string is a course_id, returns a dict with keys 'id', 'revision', and 'block'. + Revision is optional: if missing returned_dict['revision'] is None. + Block is optional: if missing returned_dict['block'] is None. + Else returns None. + """ + match = COURSE_ID_RE.match(string) + if not match: + return None + return match.groupdict() diff --git a/common/lib/xmodule/xmodule/modulestore/split_mongo/__init__.py b/common/lib/xmodule/xmodule/modulestore/split_mongo/__init__.py new file mode 100644 index 0000000000..65a4d723dd --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/split_mongo/__init__.py @@ -0,0 +1 @@ +from split import SplitMongoModuleStore diff --git a/common/lib/xmodule/xmodule/modulestore/split_mongo/caching_descriptor_system.py b/common/lib/xmodule/xmodule/modulestore/split_mongo/caching_descriptor_system.py new file mode 100644 index 0000000000..1591757490 --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/split_mongo/caching_descriptor_system.py @@ -0,0 +1,119 @@ +import sys +import logging +from xmodule.mako_module import MakoDescriptorSystem +from xmodule.x_module import XModuleDescriptor +from xmodule.modulestore.locator import BlockUsageLocator +from xmodule.error_module import ErrorDescriptor +from xmodule.errortracker import exc_info_to_str +from xblock.runtime import DbModel +from ..exceptions import ItemNotFoundError +from .split_mongo_kvs import SplitMongoKVS, SplitMongoKVSid + +log = logging.getLogger(__name__) + +# TODO should this be here or w/ x_module or ??? +class CachingDescriptorSystem(MakoDescriptorSystem): + """ + A system that has a cache of a course version's json that it will use to load modules + from, with a backup of calling to the underlying modulestore for more data. + + Computes the metadata inheritance upon creation. + """ + def __init__(self, modulestore, course_entry, module_data, lazy, + default_class, error_tracker, render_template): + """ + Computes the metadata inheritance and sets up the cache. + + modulestore: the module store that can be used to retrieve additional + modules + + module_data: a dict mapping Location -> json that was cached from the + underlying modulestore + + default_class: The default_class to use when loading an + XModuleDescriptor from the module_data + + resources_fs: a filesystem, as per MakoDescriptorSystem + + error_tracker: a function that logs errors for later display to users + + render_template: a function for rendering templates, as per + MakoDescriptorSystem + """ + # TODO find all references to resources_fs and make handle None + super(CachingDescriptorSystem, self).__init__( + self._load_item, None, error_tracker, render_template) + self.modulestore = modulestore + self.course_entry = course_entry + self.lazy = lazy + self.module_data = module_data + self.default_class = default_class + # TODO see if self.course_id is needed: is already in course_entry but could be > 1 value + # Compute inheritance + modulestore.inherit_metadata(course_entry.get('blocks', {}), + course_entry.get('blocks', {}) + .get(course_entry.get('root'))) + + def _load_item(self, usage_id, course_entry_override=None): + # TODO ensure all callers of system.load_item pass just the id + json_data = self.module_data.get(usage_id) + if json_data is None: + # deeper than initial descendant fetch or doesn't exist + self.modulestore.cache_items(self, [usage_id], lazy=self.lazy) + json_data = self.module_data.get(usage_id) + if json_data is None: + raise ItemNotFoundError + + class_ = XModuleDescriptor.load_class( + json_data.get('category'), + self.default_class + ) + return self.xblock_from_json(class_, usage_id, json_data, course_entry_override) + + def xblock_from_json(self, class_, usage_id, json_data, course_entry_override=None): + if course_entry_override is None: + course_entry_override = self.course_entry + # most likely a lazy loader but not the id directly + definition = json_data.get('definition', {}) + metadata = json_data.get('metadata', {}) + + block_locator = BlockUsageLocator( + version_guid=course_entry_override['_id'], + usage_id=usage_id, + course_id=course_entry_override.get('course_id'), + revision=course_entry_override.get('revision') + ) + + kvs = SplitMongoKVS( + definition, + json_data.get('children', []), + metadata, + json_data.get('_inherited_metadata'), + block_locator, + json_data.get('category')) + model_data = DbModel(kvs, class_, None, + SplitMongoKVSid( + # DbModel req's that these support .url() + block_locator, + self.modulestore.definition_locator(definition))) + + try: + module = class_(self, model_data) + except Exception: + log.warning("Failed to load descriptor", exc_info=True) + if usage_id is None: + usage_id = "MISSING" + return ErrorDescriptor.from_json( + json_data, + self, + BlockUsageLocator(version_guid=course_entry_override['_id'], + usage_id=usage_id), + error_msg=exc_info_to_str(sys.exc_info()) + ) + + module.edited_by = json_data.get('edited_by') + module.edited_on = json_data.get('edited_on') + module.previous_version = json_data.get('previous_version') + module.update_version = json_data.get('update_version') + module.definition_locator = self.modulestore.definition_locator(definition) + return module diff --git a/common/lib/xmodule/xmodule/modulestore/split_mongo/definition_lazy_loader.py b/common/lib/xmodule/xmodule/modulestore/split_mongo/definition_lazy_loader.py new file mode 100644 index 0000000000..5ccaaa7ed3 --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/split_mongo/definition_lazy_loader.py @@ -0,0 +1,26 @@ +from xmodule.modulestore.locator import DescriptionLocator + + +class DefinitionLazyLoader(object): + """ + A placeholder to put into an xblock in place of its definition which + when accessed knows how to get its content. Only useful if the containing + object doesn't force access during init but waits until client wants the + definition. Only works if the modulestore is a split mongo store. + """ + def __init__(self, modulestore, definition_id): + """ + Simple placeholder for yet-to-be-fetched data + :param modulestore: the pymongo db connection with the definitions + :param definition_locator: the id of the record in the above to fetch + """ + self.modulestore = modulestore + self.definition_locator = DescriptionLocator(definition_id) + + def fetch(self): + """ + Fetch the definition. Note, the caller should replace this lazy + loader pointer with the result so as not to fetch more than once + """ + return self.modulestore.definitions.find_one( + {'_id': self.definition_locator.definition_id}) diff --git a/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py b/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py new file mode 100644 index 0000000000..6dd6fb480f --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py @@ -0,0 +1,1240 @@ +import threading +import datetime +import logging +import pymongo +import re +from importlib import import_module +from path import path + +from xmodule.errortracker import null_error_tracker +from xmodule.x_module import XModuleDescriptor +from xmodule.modulestore.locator import BlockUsageLocator, DescriptionLocator, CourseLocator, VersionTree +from xmodule.modulestore.exceptions import InsufficientSpecificationError, VersionConflictError +from xmodule.modulestore import inheritance + +from .. import ModuleStoreBase +from ..exceptions import ItemNotFoundError +from .definition_lazy_loader import DefinitionLazyLoader +from .caching_descriptor_system import CachingDescriptorSystem + +log = logging.getLogger(__name__) +#============================================================================== +# Documentation is at +# https://edx-wiki.atlassian.net/wiki/display/ENG/Mongostore+Data+Structure +# +# Known issue: +# Inheritance for cached kvs doesn't work on edits. Use case. +# 1) attribute foo is inheritable +# 2) g.children = [p], p.children = [a] +# 3) g.foo = 1 on load +# 4) if g.foo > 0, if p.foo > 0, if a.foo > 0 all eval True +# 5) p.foo = -1 +# 6) g.foo > 0, p.foo <= 0 all eval True BUT +# 7) BUG: a.foo > 0 still evals True but should be False +# 8) reread and everything works right +# 9) p.del(foo), p.foo > 0 is True! works +# 10) BUG: a.foo < 0! +# Local fix wont' permanently work b/c xblock may cache a.foo... +# +#============================================================================== + + +class SplitMongoModuleStore(ModuleStoreBase): + """ + A Mongodb backed ModuleStore supporting versions, inheritance, + and sharing. + """ + def __init__(self, host, db, collection, fs_root, render_template, + port=27017, default_class=None, + error_tracker=null_error_tracker, + user=None, password=None, + **kwargs): + + ModuleStoreBase.__init__(self) + + self.db = pymongo.database.Database(pymongo.MongoClient( + host=host, + port=port, + tz_aware=True, + **kwargs + ), db) + + # TODO add caching of structures to thread_cache to prevent repeated fetches (but not index b/c + # it changes w/o having a change in id) + self.course_index = self.db[collection + '.active_versions'] + self.structures = self.db[collection + '.structures'] + self.definitions = self.db[collection + '.definitions'] + + # ??? Code review question: those familiar w/ python threading. Should I instead + # use django cache? How should I expire entries? + # _add_cache could use a lru mechanism to control the cache size? + self.thread_cache = threading.local() + + if user is not None and password is not None: + self.db.authenticate(user, password) + + # every app has write access to the db (v having a flag to indicate r/o v write) + # Force mongo to report errors, at the expense of performance + # pymongo docs suck but explanation: + # http://api.mongodb.org/java/2.10.1/com/mongodb/WriteConcern.html + self.course_index.write_concern = {'w': 1} + self.structures.write_concern = {'w': 1} + self.definitions.write_concern = {'w': 1} + + if default_class is not None: + module_path, _, class_name = default_class.rpartition('.') + class_ = getattr(import_module(module_path), class_name) + self.default_class = class_ + else: + self.default_class = None + self.fs_root = path(fs_root) + self.error_tracker = error_tracker + self.render_template = render_template + + def cache_items(self, system, base_usage_ids, depth=0, lazy=True): + ''' + Handles caching of items once inheritance and any other one time + per course per fetch operations are done. + :param system: a CachingDescriptorSystem + :param base_usage_ids: list of usage_ids to fetch + :param depth: how deep below these to prefetch + :param lazy: whether to fetch definitions or use placeholders + ''' + new_module_data = {} + for usage_id in base_usage_ids: + new_module_data = self.descendants(system.course_entry['blocks'], + usage_id, + depth, + new_module_data) + + # remove any which were already in module_data (not sure if there's a better way) + for newkey in new_module_data.iterkeys(): + if newkey in system.module_data: + del new_module_data[newkey] + + if lazy: + for block in new_module_data.itervalues(): + block['definition'] = DefinitionLazyLoader(self, + block['definition']) + else: + # Load all descendants by id + descendent_definitions = self.definitions.find({ + '_id': {'$in': [block['definition'] + for block in new_module_data.itervalues()]}}) + # turn into a map + definitions = {definition['_id']: definition + for definition in descendent_definitions} + + for block in new_module_data.itervalues(): + if block['definition'] in definitions: + block['definition'] = definitions[block['definition']] + + system.module_data.update(new_module_data) + return system.module_data + + def _load_items(self, course_entry, usage_ids, depth=0, lazy=True): + ''' + Load & cache the given blocks from the course. Prefetch down to the + given depth. Load the definitions into each block if lazy is False; + otherwise, use the lazy definition placeholder. + ''' + system = self._get_cache(course_entry['_id']) + if system is None: + system = CachingDescriptorSystem( + self, + course_entry, + {}, + lazy, + self.default_class, + self.error_tracker, + self.render_template + ) + self._add_cache(course_entry['_id'], system) + self.cache_items(system, usage_ids, depth, lazy) + return [system.load_item(usage_id, course_entry) for usage_id in usage_ids] + + def _get_cache(self, course_version_guid): + """ + Find the descriptor cache for this course if it exists + :param course_version_guid: + """ + if not hasattr(self.thread_cache, 'course_cache'): + self.thread_cache.course_cache = {} + system = self.thread_cache.course_cache + return system.get(course_version_guid) + + def _add_cache(self, course_version_guid, system): + """ + Save this cache for subsequent access + :param course_version_guid: + :param system: + """ + if not hasattr(self.thread_cache, 'course_cache'): + self.thread_cache.course_cache = {} + self.thread_cache.course_cache[course_version_guid] = system + return system + + def _clear_cache(self): + """ + Should only be used by testing or something which implements transactional boundary semantics + """ + self.thread_cache.course_cache = {} + + def _lookup_course(self, course_locator): + ''' + Decode the locator into the right series of db access. Does not + return the CourseDescriptor! It returns the actual db json from + structures. + + Semantics: if course_id and revision given, then it will get that revision. If + also give a version_guid, it will see if the current head of that revision == that guid. If not + it raises VersionConflictError (the version now differs from what it was when you got your + reference) + + :param course_locator: any subclass of CourseLocator + ''' + # NOTE: if and when this uses cache, the update if changed logic will break if the cache + # holds the same objects as the descriptors! + if not course_locator.is_fully_specified(): + raise InsufficientSpecificationError('Not fully specified: %s' % course_locator) + + if course_locator.course_id is not None and course_locator.revision is not None: + # use the course_id + index = self.course_index.find_one({'_id': course_locator.course_id}) + if index is None: + raise ItemNotFoundError(course_locator) + if course_locator.revision not in index['versions']: + raise ItemNotFoundError(course_locator) + version_guid = index['versions'][course_locator.revision] + if course_locator.version_guid is not None and version_guid != course_locator.version_guid: + # This may be a bit too touchy but it's hard to infer intent + raise VersionConflictError(course_locator, CourseLocator(course_locator, version_guid=version_guid)) + else: + # TODO should this raise an exception if revision was provided? + version_guid = course_locator.version_guid + + # cast string to ObjectId if necessary + version_guid = course_locator.as_object_id(version_guid) + entry = self.structures.find_one({'_id': version_guid}) + + # b/c more than one course can use same structure, the 'course_id' is not intrinsic to structure + # and the one assoc'd w/ it by another fetch may not be the one relevant to this fetch; so, + # fake it by explicitly setting it in the in memory structure. + + if course_locator.course_id: + entry['course_id'] = course_locator.course_id + entry['revision'] = course_locator.revision + return entry + + def get_courses(self, revision, qualifiers=None): + ''' + Returns a list of course descriptors matching any given qualifiers. + + qualifiers should be a dict of keywords matching the db fields or any + legal query for mongo to use against the active_versions collection. + + Note, this is to find the current head of the named revision type + (e.g., 'draft'). To get specific versions via guid use get_course. + ''' + if qualifiers is None: + qualifiers = {} + qualifiers.update({"versions.{}".format(revision): {"$exists": True}}) + matching = self.course_index.find(qualifiers) + + # collect ids and then query for those + version_guids = [] + id_version_map = {} + for course_entry in matching: + version_guid = course_entry['versions'][revision] + version_guids.append(version_guid) + id_version_map[version_guid] = course_entry['_id'] + + course_entries = self.structures.find({'_id': {'$in': version_guids}}) + + # get the block for the course element (s/b the root) + result = [] + for entry in course_entries: + # structures are course agnostic but the caller wants to know course, so add it in here + entry['course_id'] = id_version_map[entry['_id']] + root = entry['root'] + result.extend(self._load_items(entry, [root], 0, lazy=True)) + return result + + def get_course(self, course_locator): + ''' + Gets the course descriptor for the course identified by the locator + which may or may not be a blockLocator. + + raises InsufficientSpecificationError + ''' + course_entry = self._lookup_course(course_locator) + root = course_entry['root'] + result = self._load_items(course_entry, [root], 0, lazy=True) + return result[0] + + def get_course_for_item(self, location): + ''' + Provided for backward compatibility. Is equivalent to calling get_course + :param location: + ''' + return self.get_course(location) + + def has_item(self, block_location): + """ + Returns True if location exists in its course. Returns false if + the course or the block w/in the course do not exist for the given version. + raises InsufficientSpecificationError if the locator does not id a block + """ + if block_location.usage_id is None: + raise InsufficientSpecificationError(block_location) + try: + course_structure = self._lookup_course(block_location) + except ItemNotFoundError: + # this error only occurs if the course does not exist + return False + + return course_structure['blocks'].get(block_location.usage_id) is not None + + def get_item(self, location, depth=0): + """ + depth (int): An argument that some module stores may use to prefetch + descendants of the queried modules for more efficient results later + in the request. The depth is counted in the number of + calls to get_children() to cache. None indicates to cache all + descendants. + raises InsufficientSpecificationError or ItemNotFoundError + """ + assert isinstance(location, BlockUsageLocator) + if not location.is_initialized(): + raise InsufficientSpecificationError("Not yet initialized: %s" % location) + course = self._lookup_course(location) + items = self._load_items(course, [location.usage_id], depth, lazy=True) + if len(items) == 0: + raise ItemNotFoundError(location) + return items[0] + + # TODO refactor this and get_courses to use a constructed query + def get_items(self, locator, qualifiers): + ''' + Get all of the modules in the given course matching the qualifiers. The + qualifiers should only be fields in the structures collection (sorry). + There will be a separate search method for searching through + definitions. + + Common qualifiers are category, definition (provide definition id), + metadata: {display_name ..}, children (return + block if its children includes the one given value). If you want + substring matching use {$regex: /acme.*corp/i} type syntax. + + Although these + look like mongo queries, it is all done in memory; so, you cannot + try arbitrary queries. + + :param locator: CourseLocator or BlockUsageLocator restricting search scope + :param qualifiers: a dict restricting which elements should match + ''' + # TODO extend to only search a subdag of the course? + course = self._lookup_course(locator) + items = [] + for usage_id, value in course['blocks'].iteritems(): + if self._block_matches(value, qualifiers): + items.append(usage_id) + + if len(items) > 0: + return self._load_items(course, items, 0, lazy=True) + else: + return [] + + # What's the use case for usage_id being separate? + def get_parent_locations(self, locator, usage_id=None): + ''' + Return the locations (Locators w/ usage_ids) for the parents of this location in this + course. Could use get_items(location, {'children': usage_id}) but this is slightly faster. + NOTE: does not actually ensure usage_id exists + If usage_id is None, then the locator must specify the usage_id + ''' + if usage_id is None: + usage_id = locator.usage_id + course = self._lookup_course(locator) + items = [] + for parent_id, value in course['blocks'].iteritems(): + for child_id in value['children']: + if usage_id == child_id: + locator = locator.as_course_locator() + items.append(BlockUsageLocator(url=locator, usage_id=parent_id)) + return items + + def get_course_index_info(self, course_locator): + """ + The index records the initial creation of the indexed course and tracks the current version + heads. This function is primarily for test verification but may serve some + more general purpose. + :param course_locator: must have a course_id set + :return {'org': , 'prettyid': , + versions: {'draft': the head draft version id, + 'published': the head published version id if any, + }, + 'edited_by': who created the course originally (named edited for consistency), + 'edited_on': when the course was originally created + } + """ + if course_locator.course_id is None: + return None + index = self.course_index.find_one({'_id': course_locator.course_id}) + return index + + # TODO figure out a way to make this info accessible from the course descriptor + def get_course_history_info(self, course_locator): + """ + Because xblocks doesn't give a means to separate the course structure's meta information from + the course xblock's, this method will get that info for the structure as a whole. + :param course_locator: + :return {'original_version': the version guid of the original version of this course, + 'previous_version': the version guid of the previous version, + 'edited_by': who made the last change, + 'edited_on': when the change was made + } + """ + course = self._lookup_course(course_locator) + return {'original_version': course['original_version'], + 'previous_version': course['previous_version'], + 'edited_by': course['edited_by'], + 'edited_on': course['edited_on'] + } + + def get_definition_history_info(self, definition_locator): + """ + Because xblocks doesn't give a means to separate the definition's meta information from + the usage xblock's, this method will get that info for the definition + :return {'original_version': the version guid of the original version of this course, + 'previous_version': the version guid of the previous version, + 'edited_by': who made the last change, + 'edited_on': when the change was made + } + """ + definition = self.definitions.find_one({'_id': definition_locator.definition_id}) + if definition is None: + return None + return {'original_version': definition['original_version'], + 'previous_version': definition['previous_version'], + 'edited_by': definition['edited_by'], + 'edited_on': definition['edited_on'] + } + + def get_course_successors(self, course_locator, version_history_depth=1): + ''' + Find the version_history_depth next versions of this course. Return as a VersionTree + Mostly makes sense when course_locator uses a version_guid, but because it finds all relevant + next versions, these do include those created for other courses. + :param course_locator: + ''' + if version_history_depth < 1: + return None + if course_locator.version_guid is None: + course = self._lookup_course(course_locator) + version_guid = course.version_guid + else: + version_guid = course_locator.version_guid + + # TODO if depth is significant, it may make sense to get all that have the same original_version + # and reconstruct the subtree from version_guid + next_entries = self.structures.find({'previous_version' : version_guid}) + # must only scan cursor's once + next_versions = [struct for struct in next_entries] + result = {version_guid: [CourseLocator(version_guid=struct['_id']) for struct in next_versions]} + depth = 1 + while depth < version_history_depth and len(next_versions) > 0: + depth += 1 + next_entries = self.structures.find({'previous_version': + {'$in': [struct['_id'] for struct in next_versions]}}) + next_versions = [struct for struct in next_entries] + for course_structure in next_versions: + result.setdefault(course_structure['previous_version'], []).append( + CourseLocator(version_guid=struct['_id'])) + return VersionTree(CourseLocator(course_locator, version_guid=version_guid), result) + + + def get_block_generations(self, block_locator): + ''' + Find the history of this block. Return as a VersionTree of each place the block changed (except + deletion). + + The block's history tracks its explicit changes; so, changes in descendants won't be reflected + as new iterations. + ''' + block_locator = block_locator.version_agnostic() + course_struct = self._lookup_course(block_locator) + usage_id = block_locator.usage_id + update_version_field = 'blocks.{}.update_version'.format(usage_id) + all_versions_with_block = self.structures.find({'original_version': course_struct['original_version'], + update_version_field: {'$exists': True}}) + # find (all) root versions and build map previous: [successors] + possible_roots = [] + result = {} + for version in all_versions_with_block: + if version['_id'] == version['blocks'][usage_id]['update_version']: + if version['blocks'][usage_id].get('previous_version') is None: + possible_roots.append(version['blocks'][usage_id]['update_version']) + else: + result.setdefault(version['blocks'][usage_id]['previous_version'], set()).add( + version['blocks'][usage_id]['update_version']) + # more than one possible_root means usage was added and deleted > 1x. + if len(possible_roots) > 1: + # find the history segment including block_locator's version + element_to_find = course_struct['blocks'][usage_id]['update_version'] + if element_to_find in possible_roots: + possible_roots = [element_to_find] + for possibility in possible_roots: + if self._find_local_root(element_to_find, possibility, result): + possible_roots = [possibility] + break + elif len(possible_roots) == 0: + return None + # convert the results value sets to locators + for k, versions in result.iteritems(): + result[k] = [BlockUsageLocator(version_guid=version, usage_id=usage_id) + for version in versions] + return VersionTree(BlockUsageLocator(version_guid=possible_roots[0], usage_id=usage_id), result) + + def get_definition_successors(self, definition_locator, version_history_depth=1): + ''' + Find the version_history_depth next versions of this definition. Return as a VersionTree + ''' + # TODO implement + pass + + def create_definition_from_data(self, new_def_data, category, user_id): + """ + Pull the definition fields out of descriptor and save to the db as a new definition + w/o a predecessor and return the new id. + + :param user_id: request.user object + """ + document = {"category" : category, + "data": new_def_data, + "edited_by": user_id, + "edited_on": datetime.datetime.utcnow(), + "previous_version": None, + "original_version": None} + new_id = self.definitions.insert(document) + definition_locator = DescriptionLocator(new_id) + document['original_version'] = new_id + self.definitions.update({'_id': new_id}, {'$set': {"original_version": new_id}}) + return definition_locator + + def update_definition_from_data(self, definition_locator, new_def_data, user_id): + """ + See if new_def_data differs from the persisted version. If so, update + the persisted version and return the new id. + + :param user_id: request.user + """ + def needs_saved(): + if isinstance(new_def_data, dict): + for key, value in new_def_data.iteritems(): + if key not in old_definition['data'] or value != old_definition['data'][key]: + return True + for key, value in old_definition['data'].iteritems(): + if key not in new_def_data: + return True + else: + return new_def_data != old_definition['data'] + + # if this looks in cache rather than fresh fetches, then it will probably not detect + # actual change b/c the descriptor and cache probably point to the same objects + old_definition = self.definitions.find_one({'_id': definition_locator.definition_id}) + if old_definition is None: + raise ItemNotFoundError(definition_locator.url()) + del old_definition['_id'] + + if needs_saved(): + old_definition['data'] = new_def_data + old_definition['edited_by'] = user_id + old_definition['edited_on'] = datetime.datetime.utcnow() + old_definition['previous_version'] = definition_locator.definition_id + new_id = self.definitions.insert(old_definition) + return DescriptionLocator(new_id), True + else: + return definition_locator, False + + def _generate_usage_id(self, course_blocks, category): + """ + Generate a somewhat readable block id unique w/in this course using the category + :param course_blocks: the current list of blocks. + :param category: + """ + # NOTE: a potential bug is that a block is deleted and another created which gets the old + # block's id. a possible fix is to cache the last serial in a dict in the structure + # {category: last_serial...} + # A potential confusion is if the name incorporates the parent's name, then if the child + # moves, its id won't change and will be confusing + serial = 1 + while category + str(serial) in course_blocks: + serial += 1 + return category + str(serial) + + def _generate_course_id(self, id_root): + """ + Generate a somewhat readable course id unique w/in this db using the id_root + :param course_blocks: the current list of blocks. + :param category: + """ + existing_uses = self.course_index.find({"_id": {"$regex": id_root}}) + if existing_uses.count() > 0: + max_found = 0 + matcher = re.compile(id_root + r'(\d+)') + for entry in existing_uses: + serial = re.search(matcher, entry['_id']) + if serial is not None and serial.groups > 0: + value = int(serial.group(1)) + if value > max_found: + max_found = value + return id_root + str(max_found + 1) + else: + return id_root + + # TODO I would love to write this to take a real descriptor and persist it BUT descriptors, kvs, and dbmodel + # all assume locators are set and unique! Having this take the model contents piecemeal breaks the separation + # of model from persistence layer + def create_item(self, course_or_parent_locator, category, user_id, definition_locator=None, new_def_data=None, + metadata=None, force=False): + """ + Add a descriptor to persistence as the last child of the optional parent_location or just as an element + of the course (if no parent provided). Return the resulting post saved version with populated locators. + + If the locator is a BlockUsageLocator, then it's assumed to be the parent. If it's a CourseLocator, then it's + merely the containing course. + + raises InsufficientSpecificationError if there is no course locator. + raises VersionConflictError if course_id and version_guid given and the current version head != version_guid + and force is not True. + force: fork the structure and don't update the course draftVersion if the above + + The incoming definition_locator should either be None to indicate this is a brand new definition or + a pointer to the existing definition to which this block should point or from which this was derived. + If new_def_data is None, then definition_locator must have a value meaning that this block points + to the existing definition. If new_def_data is not None and definition_location is not None, then + new_def_data is assumed to be a new payload for definition_location. + + Creates a new version of the course structure, creates and inserts the new block, makes the block point + to the definition which may be new or a new version of an existing or an existing. + Rules for course locator: + + * If the course locator specifies a course_id and either it doesn't + specify version_guid or the one it specifies == the current draft, it progresses the course to point + to the new draft and sets the active version to point to the new draft + * If the locator has a course_id but its version_guid != current draft, it raises VersionConflictError. + + NOTE: using a version_guid will end up creating a new version of the course. Your new item won't be in + the course id'd by version_guid but instead in one w/ a new version_guid. Ensure in this case that you get + the new version_guid from the locator in the returned object! + """ + # find course_index entry if applicable and structures entry + index_entry = self._get_index_if_valid(course_or_parent_locator, force) + structure = self._lookup_course(course_or_parent_locator) + + # persist the definition if persisted != passed + if (definition_locator is None or definition_locator.definition_id is None): + definition_locator = self.create_definition_from_data(new_def_data, category, user_id) + elif new_def_data is not None: + definition_locator, _ = self.update_definition_from_data(definition_locator, new_def_data, user_id) + + # copy the structure and modify the new one + new_structure = self._version_structure(structure, user_id) + # generate an id + new_usage_id = self._generate_usage_id(new_structure['blocks'], category) + update_version_keys = ['blocks.{}.update_version'.format(new_usage_id)] + if isinstance(course_or_parent_locator, BlockUsageLocator) and course_or_parent_locator.usage_id is not None: + parent = new_structure['blocks'][course_or_parent_locator.usage_id] + parent['children'].append(new_usage_id) + parent['edited_on'] = datetime.datetime.utcnow() + parent['edited_by'] = user_id + parent['previous_version'] = parent['update_version'] + update_version_keys.append('blocks.{}.update_version'.format(course_or_parent_locator.usage_id)) + new_structure['blocks'][new_usage_id] = { + "children": [], + "category": category, + "definition": definition_locator.definition_id, + "metadata": metadata if metadata else {}, + 'edited_on': datetime.datetime.utcnow(), + 'edited_by': user_id, + 'previous_version': None + } + new_id = self.structures.insert(new_structure) + update_version_payload = {key: new_id for key in update_version_keys} + self.structures.update({'_id': new_id}, + {'$set': update_version_payload}) + + # update the index entry if appropriate + if index_entry is not None: + self._update_head(index_entry, course_or_parent_locator.revision, new_id) + course_parent = course_or_parent_locator.as_course_locator() + else: + course_parent = None + + # fetch and return the new item--fetching is unnecessary but a good qc step + return self.get_item(BlockUsageLocator(course_id=course_parent, + usage_id=new_usage_id, + version_guid=new_id)) + + def create_course(self, org, prettyid, user_id, id_root=None, metadata=None, course_data=None, + master_version='draft', versions_dict=None, root_category='course'): + """ + Create a new entry in the active courses index which points to an existing or new structure. Returns + the course root of the resulting entry (the location has the course id) + + id_root: allows the caller to specify the course_id. It's a root in that, if it's already taken, + this method will append things to the root to make it unique. (defaults to org) + + metadata: if provided, will set the metadata of the root course object in the new draft course. If both + metadata and a starting version are provided, it will generate a successor version to the given version, + and update the metadata with any provided values (via update not setting). + + course_data: if provided, will update the data of the new course xblock definition to this. Like metadata, + if provided, this will cause a new version of any given version as well as a new version of the + definition (which will point to the existing one if given a version). If not provided and given + a draft_version, it will reuse the same definition as the draft course (obvious since it's reusing the draft + course). If not provided and no draft is given, it will be empty and get the field defaults (hopefully) when + loaded. + + master_version: the tag (key) for the version name in the dict which is the 'draft' version. Not the actual + version guid, but what to call it. + + versions_dict: the starting version ids where the keys are the tags such as 'draft' and 'published' + and the values are structure guids. If provided, the new course will reuse this version (unless you also + provide any overrides such as metadata, see above). if not provided, will create a mostly empty course + structure with just a category course root xblock. + """ + if metadata is None: + metadata = {} + # build from inside out: definition, structure, index entry + # if building a wholly new structure + if versions_dict is None or master_version not in versions_dict: + # create new definition and structure + if course_data is None: + course_data = {} + definition_entry = { + 'category': root_category, + 'data': course_data, + 'edited_by': user_id, + 'edited_on': datetime.datetime.utcnow(), + 'previous_version': None, + } + definition_id = self.definitions.insert(definition_entry) + definition_entry['original_version'] = definition_id + self.definitions.update({'_id': definition_id}, {'$set': {"original_version": definition_id}}) + + draft_structure = { + 'root': 'course', + 'previous_version': None, + 'edited_by': user_id, + 'edited_on': datetime.datetime.utcnow(), + 'blocks': { + 'course': { + 'children':[], + 'category': 'course', + 'definition': definition_id, + 'metadata': metadata, + 'edited_on': datetime.datetime.utcnow(), + 'edited_by': user_id, + 'previous_version': None}}} + new_id = self.structures.insert(draft_structure) + draft_structure['original_version'] = new_id + self.structures.update({'_id': new_id}, + {'$set': {"original_version": new_id, + 'blocks.course.update_version': new_id}}) + if versions_dict is None: + versions_dict = {master_version: new_id} + else: + versions_dict[master_version] = new_id + + else: + # just get the draft_version structure + draft_version = CourseLocator(version_guid=versions_dict[master_version]) + draft_structure = self._lookup_course(draft_version) + if course_data is not None or metadata: + draft_structure = self._version_structure(draft_structure, user_id) + root_block = draft_structure['blocks'][draft_structure['root']] + if metadata is not None: + root_block['metadata'].update(metadata) + if course_data is not None: + definition = self.definitions.find_one({'_id': root_block['definition']}) + definition['data'].update(course_data) + definition['previous_version'] = definition['_id'] + definition['edited_by'] = user_id + definition['edited_on'] = datetime.datetime.utcnow() + del definition['_id'] + root_block['definition'] = self.definitions.insert(definition) + root_block['edited_on'] = datetime.datetime.utcnow() + root_block['edited_by'] = user_id + root_block['previous_version'] = root_block.get('update_version') + # insert updates the '_id' in draft_structure + new_id = self.structures.insert(draft_structure) + versions_dict[master_version] = new_id + self.structures.update({'_id': new_id}, + {'$set': {'blocks.{}.update_version'.format(draft_structure['root']): new_id}}) + # create the index entry + if id_root is None: + id_root = org + new_id = self._generate_course_id(id_root) + + index_entry = { + '_id': new_id, + 'org': org, + 'prettyid': prettyid, + 'edited_by': user_id, + 'edited_on': datetime.datetime.utcnow(), + 'versions': versions_dict} + new_id = self.course_index.insert(index_entry) + return self.get_course(CourseLocator(course_id=new_id, revision=master_version)) + + def update_item(self, descriptor, user_id, force=False): + """ + Save the descriptor's definition, metadata, & children references (i.e., it doesn't descend the tree). + Return the new descriptor (updated location). + + raises ItemNotFoundError if the location does not exist. + + Creates a new course version. If the descriptor's location has a course_id, it moves the course head + pointer. If the version_guid of the descriptor points to a non-head version and there's been an intervening + change to this item, it raises a VersionConflictError unless force is True. In the force case, it forks + the course but leaves the head pointer where it is (this change will not be in the course head). + + The implementation tries to detect which, if any changes, actually need to be saved and thus won't version + the definition, structure, nor course if they didn't change. + """ + original_structure = self._lookup_course(descriptor.location) + index_entry = self._get_index_if_valid(descriptor.location, force) + + descriptor.definition_locator, is_updated = self.update_definition_from_data( + descriptor.definition_locator, descriptor.xblock_kvs.get_data(), user_id) + # check children + original_entry = original_structure['blocks'][descriptor.location.usage_id] + if (not is_updated and descriptor.has_children + and not self._xblock_lists_equal(original_entry['children'], descriptor.children)): + is_updated = True + # check metadata + if not is_updated: + is_updated = self._compare_metadata(descriptor.xblock_kvs.get_own_metadata(), original_entry['metadata']) + + # if updated, rev the structure + if is_updated: + new_structure = self._version_structure(original_structure, user_id) + block_data = new_structure['blocks'][descriptor.location.usage_id] + if descriptor.has_children: + block_data["children"] = [self._usage_id(child) for child in descriptor.children] + + block_data["definition"] = descriptor.definition_locator.definition_id + block_data["metadata"] = descriptor.xblock_kvs.get_own_metadata() + block_data['edited_on'] = datetime.datetime.utcnow() + block_data['edited_by'] = user_id + block_data['previous_version'] = block_data['update_version'] + new_id = self.structures.insert(new_structure) + self.structures.update({'_id': new_id}, + {'$set': {'blocks.{}.update_version'.format(descriptor.location.usage_id): new_id}}) + + # update the index entry if appropriate + if index_entry is not None: + self._update_head(index_entry, descriptor.location.revision, new_id) + + # fetch and return the new item--fetching is unnecessary but a good qc step + return self.get_item(BlockUsageLocator(descriptor.location, version_guid=new_id)) + else: + # nothing changed, just return the one sent in + return descriptor + + def persist_xblock_dag(self, xblock, user_id, force=False): + """ + create or update the xblock and all of its children. The xblock's location must specify a course. + If it doesn't specify a usage_id, then it's presumed to be new and need creation. This function + descends the children performing the same operation for any that are xblocks. Any children which + are usage_ids just update the children pointer. + + All updates go into the same course version (bulk updater). + + Updates the objects which came in w/ updated location and definition_location info. + + returns the post-persisted version of the incoming xblock. Note that its children will be ids not + objects. + + :param xblock: + :param user_id: + """ + # find course_index entry if applicable and structures entry + index_entry = self._get_index_if_valid(xblock.location, force) + structure = self._lookup_course(xblock.location) + new_structure = self._version_structure(structure, user_id) + + changed_blocks = self._persist_subdag(xblock, user_id, new_structure['blocks']) + + if changed_blocks: + new_id = self.structures.insert(new_structure) + update_command = {} + for usage_id in changed_blocks: + update_command['blocks.{}.update_version'.format(usage_id)] = new_id + self.structures.update({'_id': new_id}, {'$set': update_command}) + + # update the index entry if appropriate + if index_entry is not None: + self._update_head(index_entry, xblock.location.revision, new_id) + + # fetch and return the new item--fetching is unnecessary but a good qc step + return self.get_item(BlockUsageLocator(xblock.location, version_guid=new_id)) + else: + return xblock + + def _persist_subdag(self, xblock, user_id, structure_blocks): + # persist the definition if persisted != passed + new_def_data = xblock.xblock_kvs.get_data() + if (xblock.definition_locator is None or xblock.definition_locator.definition_id is None): + xblock.definition_locator = self.create_definition_from_data(new_def_data, + xblock.category, user_id) + is_updated = True + elif new_def_data is not None: + xblock.definition_locator, is_updated = self.update_definition_from_data(xblock.definition_locator, + new_def_data, user_id) + + if xblock.location.usage_id is None: + # generate an id + is_new = True + is_updated = True + usage_id = self._generate_usage_id(structure_blocks, xblock.category) + xblock.location.usage_id = usage_id + else: + is_new = False + usage_id = xblock.location.usage_id + if (not is_updated and xblock.has_children + and not self._xblock_lists_equal(structure_blocks[usage_id]['children'], xblock.children)): + is_updated = True + + children = [] + updated_blocks = [] + if xblock.has_children: + for child in xblock.children: + if isinstance(child, XModuleDescriptor): + updated_blocks += self._persist_subdag(child, user_id, structure_blocks) + children.append(child.location.usage_id) + else: + children.append(child) + + is_updated = is_updated or updated_blocks + metadata = xblock.xblock_kvs.get_own_metadata() + if not is_new and not is_updated: + is_updated = self._compare_metadata(metadata, structure_blocks[usage_id]['metadata']) + + if is_updated: + structure_blocks[usage_id] = { + "children": children, + "category": xblock.category, + "definition": xblock.definition_locator.definition_id, + "metadata": metadata if metadata else {}, + 'previous_version': structure_blocks.get(usage_id, {}).get('update_version'), + 'edited_by': user_id, + 'edited_on': datetime.datetime.utcnow() + } + updated_blocks.append(usage_id) + + return updated_blocks + + def _compare_metadata(self, metadata, original_metadata): + original_keys = original_metadata.keys() + if len(metadata) != len(original_keys): + return True + else: + new_keys = metadata.keys() + for key in original_keys: + if key not in new_keys or original_metadata[key] != metadata[key]: + return True + + # TODO change all callers to update_item + def update_children(self, course_id, location, children): + raise NotImplementedError() + + # TODO change all callers to update_item + def update_metadata(self, course_id, location, metadata): + raise NotImplementedError() + + def update_course_index(self, course_locator, new_values_dict, update_versions=False): + """ + Change the given course's index entry for the given fields. new_values_dict + should be a subset of the dict returned by get_course_index_info. + It cannot include '_id' (will raise IllegalArgument). + Provide update_versions=True if you intend this to replace the versions hash. + Note, this operation can be dangerous and break running courses. + + If the dict includes versions and not update_versions, it will raise an exception. + + If the dict includes edited_on or edited_by, it will raise an exception + + Does not return anything useful. + """ + # TODO how should this log the change? edited_on and edited_by for this entry + # has the semantic of who created the course and when; so, changing those will lose + # that information. + if '_id' in new_values_dict: + raise ValueError("Cannot override _id") + if 'edited_on' in new_values_dict or 'edited_by' in new_values_dict: + raise ValueError("Cannot set edited_on or edited_by") + if not update_versions and 'versions' in new_values_dict: + raise ValueError("Cannot override versions without setting update_versions") + self.course_index.update({'_id': course_locator.course_id}, + {'$set': new_values_dict}) + + def delete_item(self, usage_locator, user_id, force=False): + """ + Delete the tree rooted at block and any references w/in the course to the block + from a new version of the course structure. + + returns CourseLocator for new version + + raises ItemNotFoundError if the location does not exist. + raises ValueError if usage_locator points to the structure root + + Creates a new course version. If the descriptor's location has a course_id, it moves the course head + pointer. If the version_guid of the descriptor points to a non-head version and there's been an intervening + change to this item, it raises a VersionConflictError unless force is True. In the force case, it forks + the course but leaves the head pointer where it is (this change will not be in the course head). + """ + assert isinstance(usage_locator, BlockUsageLocator) and usage_locator.is_initialized() + original_structure = self._lookup_course(usage_locator) + if original_structure['root'] == usage_locator.usage_id: + raise ValueError("Cannot delete the root of a course") + index_entry = self._get_index_if_valid(usage_locator, force) + new_structure = self._version_structure(original_structure, user_id) + new_blocks = new_structure['blocks'] + parents = self.get_parent_locations(usage_locator) + update_version_keys = [] + for parent in parents: + parent_block = new_blocks[parent.usage_id] + parent_block['children'].remove(usage_locator.usage_id) + parent_block['edited_on'] = datetime.datetime.utcnow() + parent_block['edited_by'] = user_id + parent_block['previous_version'] = parent_block['update_version'] + update_version_keys.append('blocks.{}.update_version'.format(parent.usage_id)) + # remove subtree + def remove_subtree(usage_id): + for child in new_blocks[usage_id]['children']: + remove_subtree(child) + del new_blocks[usage_id] + remove_subtree(usage_locator.usage_id) + + # update index if appropriate and structures + new_id = self.structures.insert(new_structure) + if update_version_keys: + update_version_payload = {key: new_id for key in update_version_keys} + self.structures.update({'_id': new_id}, {'$set': update_version_payload}) + + result = CourseLocator(version_guid=new_id) + + # update the index entry if appropriate + if index_entry is not None: + self._update_head(index_entry, usage_locator.revision, new_id) + result.course_id = usage_locator.course_id + result.revision = usage_locator.revision + + return result + + def delete_course(self, course_id): + """ + Remove the given course from the course index. + + Only removes the course from the index. The data remains. You can use create_course + with a versions hash to restore the course; however, the edited_on and + edited_by won't reflect the originals, of course. + + :param course_id: uses course_id rather than locator to emphasize its global effect + """ + index = self.course_index.find_one({'_id': course_id}) + if index is None: + raise ItemNotFoundError(course_id) + # this is the only real delete in the system. should it do something else? + self.course_index.remove(index['_id']) + + # TODO remove all callers and then this + def get_errored_courses(self): + """ + This function doesn't make sense for the mongo modulestore, as structures + are loaded on demand, rather than up front + """ + return {} + + def inherit_metadata(self, block_map, block, inheriting_metadata=None): + """ + Updates block with any value + that exist in inheriting_metadata and don't appear in block['metadata'], + and then inherits block['metadata'] to all of the children in + block['children']. Filters by inheritance.INHERITABLE_METADATA + """ + if block is None: + return + + if inheriting_metadata is None: + inheriting_metadata = {} + + # the currently passed down values take precedence over any previously cached ones + # NOTE: this should show the values which all fields would have if inherited: i.e., + # not set to the locally defined value but to value set by nearest ancestor who sets it + block.setdefault('_inherited_metadata', {}).update(inheriting_metadata) + + # update the inheriting w/ what should pass to children + inheriting_metadata = block['_inherited_metadata'].copy() + for field in inheritance.INHERITABLE_METADATA: + if field in block['metadata']: + inheriting_metadata[field] = block['metadata'][field] + + for child in block.get('children', []): + self.inherit_metadata(block_map, block_map[child], inheriting_metadata) + + def descendants(self, block_map, usage_id, depth, descendent_map): + """ + adds block and its descendants out to depth to descendent_map + Depth specifies the number of levels of descendants to return + (0 => this usage only, 1 => this usage and its children, etc...) + A depth of None returns all descendants + """ + if usage_id not in block_map: + return descendent_map + + if usage_id not in descendent_map: + descendent_map[usage_id] = block_map[usage_id] + + if depth is None or depth > 0: + depth = depth - 1 if depth is not None else None + for child in block_map[usage_id].get('children', []): + descendent_map = self.descendants(block_map, child, depth, + descendent_map) + + return descendent_map + + def definition_locator(self, definition): + ''' + Pull the id out of the definition w/ correct semantics for its + representation + ''' + if isinstance(definition, DefinitionLazyLoader): + return definition.definition_locator + elif '_id' not in definition: + return None + else: + return DescriptionLocator(definition['_id']) + + def _block_matches(self, value, qualifiers): + ''' + Return True or False depending on whether the value (block contents) + matches the qualifiers as per get_items + :param value: + :param qualifiers: + ''' + for key, criteria in qualifiers.iteritems(): + if key in value: + target = value[key] + if not self._value_matches(target, criteria): + return False + elif criteria is not None: + return False + return True + + def _value_matches(self, target, criteria): + ''' helper for _block_matches ''' + if isinstance(target, list): + return any(self._value_matches(ele, criteria) + for ele in target) + elif isinstance(criteria, dict): + if '$regex' in criteria: + return re.search(criteria['$regex'], target) is not None + elif not isinstance(target, dict): + return False + else: + return (isinstance(target, dict) and + self._block_matches(target, criteria)) + else: + return criteria == target + + def _xblock_lists_equal(self, lista, listb): + """ + Do the 2 lists refer to the same xblocks in the same order (presumes they're from the + same course) + + :param lista: + :param listb: + """ + if len(lista) != len(listb): + return False + for idx in enumerate(lista): + if lista[idx] != listb[idx]: + itema = self._usage_id(lista[idx]) + if itema != self._usage_id(listb[idx]): + return False + return True + + def _usage_id(self, xblock_or_id): + """ + arg is either an xblock or an id. If an xblock, get the usage_id from its location. Otherwise, return itself. + :param xblock_or_id: + """ + if isinstance(xblock_or_id, XModuleDescriptor): + return xblock_or_id.location.usage_id + else: + return xblock_or_id + + def _get_index_if_valid(self, locator, force=False): + """ + If the locator identifies a course and points to its draft (or plausibly its draft), + then return the index entry. + + raises VersionConflictError if not the right version + + :param locator: + """ + if locator.course_id is None or locator.revision is None: + return None + else: + index_entry = self.course_index.find_one({'_id': locator.course_id}) + if (locator.version_guid is not None + and index_entry['versions'][locator.revision] != locator.version_guid + and not force): + raise VersionConflictError( + locator, + CourseLocator( + course_id=index_entry['_id'], + version_guid=index_entry['versions'][locator.revision], + revision=locator.revision)) + else: + return index_entry + + def _version_structure(self, structure, user_id): + """ + Copy the structure and update the history info (edited_by, edited_on, previous_version) + :param structure: + :param user_id: + """ + new_structure = structure.copy() + new_structure['blocks'] = new_structure['blocks'].copy() + del new_structure['_id'] + new_structure['previous_version'] = structure['_id'] + new_structure['edited_by'] = user_id + new_structure['edited_on'] = datetime.datetime.utcnow() + return new_structure + + def _find_local_root(self, element_to_find, possibility, tree): + if possibility not in tree: + return False + if element_to_find in tree[possibility]: + return True + for subtree in tree[possibility]: + if self._find_local_root(element_to_find, subtree, tree): + return True + return False + + + def _update_head(self, index_entry, revision, new_id): + """ + Update the active index for the given course's revision to point to new_id + + :param index_entry: + :param course_locator: + :param new_id: + """ + self.course_index.update( + {"_id": index_entry["_id"]}, + {"$set": {"versions.{}".format(revision): new_id}}) diff --git a/common/lib/xmodule/xmodule/modulestore/split_mongo/split_mongo_kvs.py b/common/lib/xmodule/xmodule/modulestore/split_mongo/split_mongo_kvs.py new file mode 100644 index 0000000000..843c1ce364 --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/split_mongo/split_mongo_kvs.py @@ -0,0 +1,163 @@ +import copy +from xblock.core import Scope +from collections import namedtuple +from xblock.runtime import KeyValueStore, InvalidScopeError +from .definition_lazy_loader import DefinitionLazyLoader + +# id is a BlockUsageLocator, def_id is the definition's guid +SplitMongoKVSid = namedtuple('SplitMongoKVSid', 'id, def_id') + + +# TODO should this be here or w/ x_module or ??? +class SplitMongoKVS(KeyValueStore): + """ + A KeyValueStore that maps keyed data access to one of the 3 data areas + known to the MongoModuleStore (data, children, and metadata) + """ + def __init__(self, definition, children, metadata, _inherited_metadata, location, category): + """ + + :param definition: + :param children: + :param metadata: the locally defined value for each metadata field + :param _inherited_metadata: the value of each inheritable field from above this. + Note, metadata may override and disagree w/ this b/c this says what the value + should be if metadata is undefined for this field. + """ + # ensure kvs's don't share objects w/ others so that changes can't appear in separate ones + # the particular use case was that changes to kvs's were polluting caches. My thinking was + # that kvs's should be independent thus responsible for the isolation. + if isinstance(definition, DefinitionLazyLoader): + self._definition = definition + else: + self._definition = copy.copy(definition) + self._children = copy.copy(children) + self._metadata = copy.copy(metadata) + self._inherited_metadata = _inherited_metadata + self._location = location + self._category = category + + def get(self, key): + if key.scope == Scope.children: + return self._children + elif key.scope == Scope.parent: + return None + elif key.scope == Scope.settings: + if key.field_name in self._metadata: + return self._metadata[key.field_name] + elif key.field_name in self._inherited_metadata: + return self._inherited_metadata[key.field_name] + else: + raise KeyError() + elif key.scope == Scope.content: + if key.field_name == 'location': + return self._location + elif key.field_name == 'category': + return self._category + else: + if isinstance(self._definition, DefinitionLazyLoader): + self._definition = self._definition.fetch() + if (key.field_name == 'data' and + not isinstance(self._definition.get('data'), dict)): + return self._definition.get('data') + elif 'data' not in self._definition or key.field_name not in self._definition['data']: + raise KeyError() + else: + return self._definition['data'][key.field_name] + else: + raise InvalidScopeError(key.scope) + + def set(self, key, value): + # TODO cache db update implications & add method to invoke + if key.scope == Scope.children: + self._children = value + # TODO remove inheritance from any orphaned exchildren + # TODO add inheritance to any new children + elif key.scope == Scope.settings: + # TODO if inheritable, push down to children who don't override + self._metadata[key.field_name] = value + elif key.scope == Scope.content: + if key.field_name == 'location': + self._location = value + elif key.field_name == 'category': + self._category = value + else: + if isinstance(self._definition, DefinitionLazyLoader): + self._definition = self._definition.fetch() + if (key.field_name == 'data' and + not isinstance(self._definition.get('data'), dict)): + self._definition.get('data') + else: + self._definition.setdefault('data', {})[key.field_name] = value + else: + raise InvalidScopeError(key.scope) + + def delete(self, key): + # TODO cache db update implications & add method to invoke + if key.scope == Scope.children: + self._children = [] + elif key.scope == Scope.settings: + # TODO if inheritable, ensure _inherited_metadata has value from above and + # revert children to that value + if key.field_name in self._metadata: + del self._metadata[key.field_name] + elif key.scope == Scope.content: + # don't allow deletion of location nor category + if key.field_name == 'location': + pass + elif key.field_name == 'category': + pass + else: + if isinstance(self._definition, DefinitionLazyLoader): + self._definition = self._definition.fetch() + if (key.field_name == 'data' and + not isinstance(self._definition.get('data'), dict)): + self._definition.setdefault('data', None) + else: + try: + del self._definition['data'][key.field_name] + except KeyError: + pass + else: + raise InvalidScopeError(key.scope) + + def has(self, key): + if key.scope in (Scope.children, Scope.parent): + return True + elif key.scope == Scope.settings: + return key.field_name in self._metadata or key.field_name in self._inherited_metadata + elif key.scope == Scope.content: + if key.field_name == 'location': + return True + elif key.field_name == 'category': + return self._category is not None + else: + if isinstance(self._definition, DefinitionLazyLoader): + self._definition = self._definition.fetch() + if (key.field_name == 'data' and + not isinstance(self._definition.get('data'), dict)): + return self._definition.get('data') is not None + else: + return key.field_name in self._definition.get('data', {}) + else: + return False + + def get_data(self): + """ + Intended only for use by persistence layer to get the native definition['data'] rep + """ + if isinstance(self._definition, DefinitionLazyLoader): + self._definition = self._definition.fetch() + return self._definition.get('data') + + def get_own_metadata(self): + """ + Get the metadata explicitly set on this element. + """ + return self._metadata + + def get_inherited_metadata(self): + """ + Get the metadata set by the ancestors (which own metadata may override or not) + """ + return self._inherited_metadata diff --git a/common/lib/xmodule/xmodule/modulestore/tests/factories.py b/common/lib/xmodule/xmodule/modulestore/tests/factories.py index 8c3b5d59dd..f2e4017114 100644 --- a/common/lib/xmodule/xmodule/modulestore/tests/factories.py +++ b/common/lib/xmodule/xmodule/modulestore/tests/factories.py @@ -38,16 +38,6 @@ class XModuleCourseFactory(Factory): new_course.display_name = display_name new_course.lms.start = datetime.datetime.now(UTC).replace(microsecond=0) - new_course.tabs = kwargs.pop( - 'tabs', - [ - {"type": "courseware"}, - {"type": "course_info", "name": "Course Info"}, - {"type": "discussion", "name": "Discussion"}, - {"type": "wiki", "name": "Wiki"}, - {"type": "progress", "name": "Progress"} - ] - ) # The rest of kwargs become attributes on the course: for k, v in kwargs.iteritems(): diff --git a/common/lib/xmodule/xmodule/modulestore/tests/persistent_factories.py b/common/lib/xmodule/xmodule/modulestore/tests/persistent_factories.py new file mode 100644 index 0000000000..5e46f5a318 --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/tests/persistent_factories.py @@ -0,0 +1,96 @@ +from xmodule.modulestore.django import modulestore +from xmodule.course_module import CourseDescriptor +from xmodule.x_module import XModuleDescriptor +import factory + + +# [dhm] I'm not sure why we're using factory_boy if we're not following its pattern. If anyone +# assumes they can call build, it will completely fail, for example. +# pylint: disable=W0232 +class PersistentCourseFactory(factory.Factory): + """ + Create a new course (not a new version of a course, but a whole new index entry). + + keywords: + * org: defaults to textX + * prettyid: defaults to 999 + * display_name + * user_id + * data (optional) the data payload to save in the course item + * metadata (optional) the metadata payload. If display_name is in the metadata, that takes + precedence over any display_name provided directly. + """ + FACTORY_FOR = CourseDescriptor + + org = 'testX' + prettyid = '999' + display_name = 'Robot Super Course' + user_id = "test_user" + data = None + metadata = None + master_version = 'draft' + + # pylint: disable=W0613 + @classmethod + def _create(cls, target_class, *args, **kwargs): + + org = kwargs.get('org') + prettyid = kwargs.get('prettyid') + display_name = kwargs.get('display_name') + user_id = kwargs.get('user_id') + data = kwargs.get('data') + metadata = kwargs.get('metadata', {}) + if metadata is None: + metadata = {} + if 'display_name' not in metadata: + metadata['display_name'] = display_name + + # Write the data to the mongo datastore + new_course = modulestore('split').create_course( + org, prettyid, user_id, metadata=metadata, course_data=data, id_root=prettyid, + master_version=kwargs.get('master_version')) + + return new_course + + @classmethod + def _build(cls, target_class, *args, **kwargs): + raise NotImplementedError() + + +class ItemFactory(factory.Factory): + FACTORY_FOR = XModuleDescriptor + + category = 'chapter' + user_id = 'test_user' + display_name = factory.LazyAttributeSequence(lambda o, n: "{} {}".format(o.category, n)) + + # pylint: disable=W0613 + @classmethod + def _create(cls, target_class, *args, **kwargs): + """ + Uses *kwargs*: + + *parent_location* (required): the location of the course & possibly parent + + *category* (defaults to 'chapter') + + *data* (optional): the data for the item + + definition_locator (optional): the DescriptorLocator for the definition this uses or branches + + *display_name* (optional): the display name of the item + + *metadata* (optional): dictionary of metadata attributes (display_name here takes + precedence over the above attr) + """ + metadata = kwargs.get('metadata', {}) + if 'display_name' not in metadata and 'display_name' in kwargs: + metadata['display_name'] = kwargs['display_name'] + + return modulestore('split').create_item(kwargs['parent_location'], kwargs['category'], + kwargs['user_id'], definition_locator=kwargs.get('definition_locator'), + new_def_data=kwargs.get('data'), metadata=metadata) + + @classmethod + def _build(cls, target_class, *args, **kwargs): + raise NotImplementedError() diff --git a/common/lib/xmodule/xmodule/modulestore/tests/test_locators.py b/common/lib/xmodule/xmodule/modulestore/tests/test_locators.py new file mode 100644 index 0000000000..2626b6692d --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/tests/test_locators.py @@ -0,0 +1,539 @@ +''' +Created on Mar 14, 2013 + +@author: dmitchell +''' +from unittest import TestCase +from nose.plugins.skip import SkipTest + +from bson.objectid import ObjectId +from xmodule.modulestore.locator import Locator, CourseLocator, BlockUsageLocator +from xmodule.modulestore.exceptions import InvalidLocationError, \ + InsufficientSpecificationError, OverSpecificationError + + +class LocatorTest(TestCase): + + def test_cant_instantiate_abstract_class(self): + self.assertRaises(TypeError, Locator) + + def test_course_constructor_overspecified(self): + self.assertRaises( + OverSpecificationError, + CourseLocator, + url='edx://edu.mit.eecs.6002x', + course_id='edu.harvard.history', + revision='published', + version_guid=ObjectId()) + self.assertRaises( + OverSpecificationError, + CourseLocator, + url='edx://edu.mit.eecs.6002x', + course_id='edu.harvard.history', + version_guid=ObjectId()) + self.assertRaises( + OverSpecificationError, + CourseLocator, + url='edx://edu.mit.eecs.6002x;published', + revision='draft') + self.assertRaises( + OverSpecificationError, + CourseLocator, + course_id='edu.mit.eecs.6002x;published', + revision='draft') + + def test_course_constructor_underspecified(self): + self.assertRaises(InsufficientSpecificationError, CourseLocator) + self.assertRaises(InsufficientSpecificationError, CourseLocator, revision='published') + + def test_course_constructor_bad_version_guid(self): + self.assertRaises(ValueError, CourseLocator, version_guid="012345") + self.assertRaises(InsufficientSpecificationError, CourseLocator, version_guid=None) + + def test_course_constructor_version_guid(self): + # generate a random location + test_id_1 = ObjectId() + test_id_1_loc = str(test_id_1) + testobj_1 = CourseLocator(version_guid=test_id_1) + self.check_course_locn_fields(testobj_1, 'version_guid', version_guid=test_id_1) + self.assertEqual(str(testobj_1.version_guid), test_id_1_loc) + self.assertEqual(str(testobj_1), '@' + test_id_1_loc) + self.assertEqual(testobj_1.url(), 'edx://@' + test_id_1_loc) + + # Test using a given string + test_id_2_loc = '519665f6223ebd6980884f2b' + test_id_2 = ObjectId(test_id_2_loc) + testobj_2 = CourseLocator(version_guid=test_id_2) + self.check_course_locn_fields(testobj_2, 'version_guid', version_guid=test_id_2) + self.assertEqual(str(testobj_2.version_guid), test_id_2_loc) + self.assertEqual(str(testobj_2), '@' + test_id_2_loc) + self.assertEqual(testobj_2.url(), 'edx://@' + test_id_2_loc) + + def test_course_constructor_bad_course_id(self): + """ + Test all sorts of badly-formed course_ids (and urls with those course_ids) + """ + for bad_id in ('edu.mit.', + ' edu.mit.eecs', + 'edu.mit.eecs ', + '@edu.mit.eecs', + '#edu.mit.eecs', + 'edu.mit.ee cs', + 'edu.mit.ee,cs', + 'edu.mit.ee/cs', + 'edu.mit.ee$cs', + 'edu.mit.ee&cs', + 'edu.mit.ee()cs', + ';this', + 'edu.mit.eecs;', + 'edu.mit.eecs;this;that', + 'edu.mit.eecs;this;', + 'edu.mit.eecs;this ', + 'edu.mit.eecs;th%is ', + ): + self.assertRaises(AssertionError, CourseLocator, course_id=bad_id) + self.assertRaises(AssertionError, CourseLocator, url='edx://' + bad_id) + + def test_course_constructor_bad_url(self): + for bad_url in ('edx://', + 'edx:/edu.mit.eecs', + 'http://edu.mit.eecs', + 'edu.mit.eecs', + 'edx//edu.mit.eecs'): + self.assertRaises(AssertionError, CourseLocator, url=bad_url) + + def test_course_constructor_redundant_001(self): + testurn = 'edu.mit.eecs.6002x' + testobj = CourseLocator(course_id=testurn, url='edx://' + testurn) + self.check_course_locn_fields(testobj, 'course_id', course_id=testurn) + + def test_course_constructor_redundant_002(self): + testurn = 'edu.mit.eecs.6002x;published' + expected_urn = 'edu.mit.eecs.6002x' + expected_rev = 'published' + testobj = CourseLocator(course_id=testurn, url='edx://' + testurn) + self.check_course_locn_fields(testobj, 'course_id', + course_id=expected_urn, + revision=expected_rev) + + def test_course_constructor_course_id_no_revision(self): + testurn = 'edu.mit.eecs.6002x' + testobj = CourseLocator(course_id=testurn) + self.check_course_locn_fields(testobj, 'course_id', course_id=testurn) + self.assertEqual(testobj.course_id, testurn) + self.assertEqual(str(testobj), testurn) + self.assertEqual(testobj.url(), 'edx://' + testurn) + + def test_course_constructor_course_id_with_revision(self): + testurn = 'edu.mit.eecs.6002x;published' + expected_id = 'edu.mit.eecs.6002x' + expected_revision = 'published' + testobj = CourseLocator(course_id=testurn) + self.check_course_locn_fields(testobj, 'course_id with revision', + course_id=expected_id, + revision=expected_revision, + ) + self.assertEqual(testobj.course_id, expected_id) + self.assertEqual(testobj.revision, expected_revision) + self.assertEqual(str(testobj), testurn) + self.assertEqual(testobj.url(), 'edx://' + testurn) + + def test_course_constructor_course_id_separate_revision(self): + test_id = 'edu.mit.eecs.6002x' + test_revision = 'published' + expected_urn = 'edu.mit.eecs.6002x;published' + testobj = CourseLocator(course_id=test_id, revision=test_revision) + self.check_course_locn_fields(testobj, 'course_id with separate revision', + course_id=test_id, + revision=test_revision, + ) + self.assertEqual(testobj.course_id, test_id) + self.assertEqual(testobj.revision, test_revision) + self.assertEqual(str(testobj), expected_urn) + self.assertEqual(testobj.url(), 'edx://' + expected_urn) + + def test_course_constructor_course_id_repeated_revision(self): + """ + The same revision appears in the course_id and the revision field. + """ + test_id = 'edu.mit.eecs.6002x;published' + test_revision = 'published' + expected_id = 'edu.mit.eecs.6002x' + expected_urn = 'edu.mit.eecs.6002x;published' + testobj = CourseLocator(course_id=test_id, revision=test_revision) + self.check_course_locn_fields(testobj, 'course_id with repeated revision', + course_id=expected_id, + revision=test_revision, + ) + self.assertEqual(testobj.course_id, expected_id) + self.assertEqual(testobj.revision, test_revision) + self.assertEqual(str(testobj), expected_urn) + self.assertEqual(testobj.url(), 'edx://' + expected_urn) + + def test_block_constructor(self): + testurn = 'edu.mit.eecs.6002x;published#HW3' + expected_id = 'edu.mit.eecs.6002x' + expected_revision = 'published' + expected_block_ref = 'HW3' + testobj = BlockUsageLocator(course_id=testurn) + self.check_block_locn_fields(testobj, 'test_block constructor', + course_id=expected_id, + revision=expected_revision, + block=expected_block_ref) + self.assertEqual(str(testobj), testurn) + self.assertEqual(testobj.url(), 'edx://' + testurn) + + # ------------------------------------------------------------ + # Disabled tests + + def test_course_urls(self): + ''' + Test constructor and property accessors. + ''' + raise SkipTest() + self.assertRaises(TypeError, CourseLocator, 'empty constructor') + + # url inits + testurn = 'edx://org/course/category/name' + self.assertRaises(InvalidLocationError, CourseLocator, url=testurn) + testurn = 'unknown/versionid/blockid' + self.assertRaises(InvalidLocationError, CourseLocator, url=testurn) + + testurn = 'cvx/versionid' + testobj = CourseLocator(testurn) + self.check_course_locn_fields(testobj, testurn, 'versionid') + self.assertEqual(testobj, CourseLocator(testobj), + 'initialization from another instance') + + testurn = 'cvx/versionid/' + testobj = CourseLocator(testurn) + self.check_course_locn_fields(testobj, testurn, 'versionid') + + testurn = 'cvx/versionid/blockid' + testobj = CourseLocator(testurn) + self.check_course_locn_fields(testobj, testurn, 'versionid') + + testurn = 'cvx/versionid/blockid/extraneousstuff?including=args' + testobj = CourseLocator(testurn) + self.check_course_locn_fields(testobj, testurn, 'versionid') + + testurn = 'cvx://versionid/blockid' + testobj = CourseLocator(testurn) + self.check_course_locn_fields(testobj, testurn, 'versionid') + + testurn = 'crx/courseid/blockid' + testobj = CourseLocator(testurn) + self.check_course_locn_fields(testobj, testurn, course_id='courseid') + + testurn = 'crx/courseid@revision/blockid' + testobj = CourseLocator(testurn) + self.check_course_locn_fields(testobj, testurn, course_id='courseid', + revision='revision') + self.assertEqual(testobj, CourseLocator(testobj), + 'run initialization from another instance') + + def test_course_keyword_setters(self): + raise SkipTest() + # arg list inits + testobj = CourseLocator(version_guid='versionid') + self.check_course_locn_fields(testobj, 'versionid arg', 'versionid') + + testobj = CourseLocator(course_id='courseid') + self.check_course_locn_fields(testobj, 'courseid arg', + course_id='courseid') + + testobj = CourseLocator(course_id='courseid', revision='rev') + self.check_course_locn_fields(testobj, 'rev arg', + course_id='courseid', + revision='rev') + # ignores garbage + testobj = CourseLocator(course_id='courseid', revision='rev', + potato='spud') + self.check_course_locn_fields(testobj, 'extra keyword arg', + course_id='courseid', + revision='rev') + + # url w/ keyword override + testurn = 'crx/courseid@revision/blockid' + testobj = CourseLocator(testurn, revision='rev') + self.check_course_locn_fields(testobj, 'rev override', + course_id='courseid', + revision='rev') + + def test_course_dict(self): + raise SkipTest() + # dict init w/ keyword overwrites + testobj = CourseLocator({"version_guid": 'versionid'}) + self.check_course_locn_fields(testobj, 'versionid dict', 'versionid') + + testobj = CourseLocator({"course_id": 'courseid'}) + self.check_course_locn_fields(testobj, 'courseid dict', + course_id='courseid') + + testobj = CourseLocator({"course_id": 'courseid', "revision": 'rev'}) + self.check_course_locn_fields(testobj, 'rev dict', + course_id='courseid', + revision='rev') + # ignores garbage + testobj = CourseLocator({"course_id": 'courseid', "revision": 'rev', + "potato": 'spud'}) + self.check_course_locn_fields(testobj, 'extra keyword dict', + course_id='courseid', + revision='rev') + testobj = CourseLocator({"course_id": 'courseid', "revision": 'rev'}, + revision='alt') + self.check_course_locn_fields(testobj, 'rev dict', + course_id='courseid', + revision='alt') + + # urn init w/ dict & keyword overwrites + testobj = CourseLocator('crx/notcourse@notthis', + {"course_id": 'courseid'}, + revision='alt') + self.check_course_locn_fields(testobj, 'rev dict', + course_id='courseid', + revision='alt') + + def test_url(self): + ''' + Ensure CourseLocator generates expected urls. + ''' + raise SkipTest() + + testobj = CourseLocator(version_guid='versionid') + self.assertEqual(testobj.url(), 'cvx/versionid', 'versionid') + self.assertEqual(testobj, CourseLocator(testobj.url()), + 'versionid conversion through url') + + testobj = CourseLocator(course_id='courseid') + self.assertEqual(testobj.url(), 'crx/courseid', 'courseid') + self.assertEqual(testobj, CourseLocator(testobj.url()), + 'courseid conversion through url') + + testobj = CourseLocator(course_id='courseid', revision='rev') + self.assertEqual(testobj.url(), 'crx/courseid@rev', 'rev') + self.assertEqual(testobj, CourseLocator(testobj.url()), + 'rev conversion through url') + + def test_html(self): + ''' + Ensure CourseLocator generates expected urls. + ''' + raise SkipTest() + testobj = CourseLocator(version_guid='versionid') + self.assertEqual(testobj.html_id(), 'cvx/versionid', 'versionid') + self.assertEqual(testobj, CourseLocator(testobj.html_id()), + 'versionid conversion through html_id') + + testobj = CourseLocator(course_id='courseid') + self.assertEqual(testobj.html_id(), 'crx/courseid', 'courseid') + self.assertEqual(testobj, CourseLocator(testobj.html_id()), + 'courseid conversion through html_id') + + testobj = CourseLocator(course_id='courseid', revision='rev') + self.assertEqual(testobj.html_id(), 'crx/courseid%40rev', 'rev') + self.assertEqual(testobj, CourseLocator(testobj.html_id()), + 'rev conversion through html_id') + + def test_block_locator(self): + ''' + Test constructor and property accessors. + ''' + raise SkipTest() + self.assertIsInstance(BlockUsageLocator(), BlockUsageLocator, + 'empty constructor') + + # url inits + testurn = 'edx://org/course/category/name' + self.assertRaises(InvalidLocationError, BlockUsageLocator, testurn) + testurn = 'unknown/versionid/blockid' + self.assertRaises(InvalidLocationError, BlockUsageLocator, testurn) + + testurn = 'cvx/versionid' + testobj = BlockUsageLocator(testurn) + self.check_block_locn_fields(testobj, testurn, 'versionid') + self.assertEqual(testobj, BlockUsageLocator(testobj), + 'initialization from another instance') + + testurn = 'cvx/versionid/' + testobj = BlockUsageLocator(testurn) + self.check_block_locn_fields(testobj, testurn, 'versionid') + + testurn = 'cvx/versionid/blockid' + testobj = BlockUsageLocator(testurn) + self.check_block_locn_fields(testobj, testurn, 'versionid', + block='blockid') + + testurn = 'cvx/versionid/blockid/extraneousstuff?including=args' + testobj = BlockUsageLocator(testurn) + self.check_block_locn_fields(testobj, testurn, 'versionid', + block='blockid') + + testurn = 'cvx://versionid/blockid' + testobj = BlockUsageLocator(testurn) + self.check_block_locn_fields(testobj, testurn, 'versionid', + block='blockid') + + testurn = 'crx/courseid/blockid' + testobj = BlockUsageLocator(testurn) + self.check_block_locn_fields(testobj, testurn, course_id='courseid', + block='blockid') + + testurn = 'crx/courseid@revision/blockid' + testobj = BlockUsageLocator(testurn) + self.check_block_locn_fields(testobj, testurn, course_id='courseid', + revision='revision', block='blockid') + self.assertEqual(testobj, BlockUsageLocator(testobj), + 'run initialization from another instance') + + def test_block_keyword_init(self): + # arg list inits + raise SkipTest() + testobj = BlockUsageLocator(version_guid='versionid') + self.check_block_locn_fields(testobj, 'versionid arg', 'versionid') + + testobj = BlockUsageLocator(version_guid='versionid', usage_id='myblock') + self.check_block_locn_fields(testobj, 'versionid arg', 'versionid', + block='myblock') + + testobj = BlockUsageLocator(course_id='courseid') + self.check_block_locn_fields(testobj, 'courseid arg', + course_id='courseid') + + testobj = BlockUsageLocator(course_id='courseid', revision='rev') + self.check_block_locn_fields(testobj, 'rev arg', + course_id='courseid', + revision='rev') + # ignores garbage + testobj = BlockUsageLocator(course_id='courseid', revision='rev', + usage_id='this_block', potato='spud') + self.check_block_locn_fields(testobj, 'extra keyword arg', + course_id='courseid', block='this_block', revision='rev') + + # url w/ keyword override + testurn = 'crx/courseid@revision/blockid' + testobj = BlockUsageLocator(testurn, revision='rev') + self.check_block_locn_fields(testobj, 'rev override', + course_id='courseid', block='blockid', + revision='rev') + + def test_block_keywords(self): + # dict init w/ keyword overwrites + raise SkipTest() + testobj = BlockUsageLocator({"version_guid": 'versionid', + 'usage_id': 'dictblock'}) + self.check_block_locn_fields(testobj, 'versionid dict', 'versionid', + block='dictblock') + + testobj = BlockUsageLocator({"course_id": 'courseid', + 'usage_id': 'dictblock'}) + self.check_block_locn_fields(testobj, 'courseid dict', + block='dictblock', course_id='courseid') + + testobj = BlockUsageLocator({"course_id": 'courseid', "revision": 'rev', + 'usage_id': 'dictblock'}) + self.check_block_locn_fields(testobj, 'rev dict', + course_id='courseid', block='dictblock', + revision='rev') + # ignores garbage + testobj = BlockUsageLocator({"course_id": 'courseid', "revision": 'rev', + 'usage_id': 'dictblock', "potato": 'spud'}) + self.check_block_locn_fields(testobj, 'extra keyword dict', + course_id='courseid', block='dictblock', + revision='rev') + testobj = BlockUsageLocator({"course_id": 'courseid', "revision": 'rev', + 'usage_id': 'dictblock'}, revision='alt', usage_id='anotherblock') + self.check_block_locn_fields(testobj, 'rev dict', + course_id='courseid', block='anotherblock', + revision='alt') + + # urn init w/ dict & keyword overwrites + testobj = BlockUsageLocator('crx/notcourse@notthis/northis', + {"course_id": 'courseid'}, revision='alt', usage_id='anotherblock') + self.check_block_locn_fields(testobj, 'rev dict', + course_id='courseid', block='anotherblock', + revision='alt') + + def test_ensure_fully_specd(self): + ''' + Test constructor and property accessors. + ''' + raise SkipTest() + self.assertRaises(InsufficientSpecificationError, + BlockUsageLocator.ensure_fully_specified, BlockUsageLocator()) + + # url inits + testurn = 'edx://org/course/category/name' + self.assertRaises(InvalidLocationError, + BlockUsageLocator.ensure_fully_specified, testurn) + testurn = 'unknown/versionid/blockid' + self.assertRaises(InvalidLocationError, + BlockUsageLocator.ensure_fully_specified, testurn) + + testurn = 'cvx/versionid' + self.assertRaises(InsufficientSpecificationError, + BlockUsageLocator.ensure_fully_specified, testurn) + + testurn = 'cvx/versionid/' + self.assertRaises(InsufficientSpecificationError, + BlockUsageLocator.ensure_fully_specified, testurn) + + testurn = 'cvx/versionid/blockid' + self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn), + BlockUsageLocator, testurn) + + testurn = 'cvx/versionid/blockid/extraneousstuff?including=args' + self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn), + BlockUsageLocator, testurn) + + testurn = 'cvx://versionid/blockid' + self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn), + BlockUsageLocator, testurn) + + testurn = 'crx/courseid/blockid' + self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn), + BlockUsageLocator, testurn) + + testurn = 'crx/courseid@revision/blockid' + self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn), + BlockUsageLocator, testurn) + + def test_ensure_fully_via_keyword(self): + # arg list inits + raise SkipTest() + testobj = BlockUsageLocator(version_guid='versionid') + self.assertRaises(InsufficientSpecificationError, + BlockUsageLocator.ensure_fully_specified, testobj) + + testurn = 'crx/courseid@revision/blockid' + testobj = BlockUsageLocator(version_guid='versionid', usage_id='myblock') + self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn), + BlockUsageLocator, testurn) + + testobj = BlockUsageLocator(course_id='courseid') + self.assertRaises(InsufficientSpecificationError, + BlockUsageLocator.ensure_fully_specified, testobj) + + testobj = BlockUsageLocator(course_id='courseid', revision='rev') + self.assertRaises(InsufficientSpecificationError, + BlockUsageLocator.ensure_fully_specified, testobj) + + testobj = BlockUsageLocator(course_id='courseid', revision='rev', + usage_id='this_block') + self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn), + BlockUsageLocator, testurn) + + # ------------------------------------------------------------------ + # Utilities + + def check_course_locn_fields(self, testobj, msg, version_guid=None, + course_id=None, revision=None): + self.assertEqual(testobj.version_guid, version_guid, msg) + self.assertEqual(testobj.course_id, course_id, msg) + self.assertEqual(testobj.revision, revision, msg) + + def check_block_locn_fields(self, testobj, msg, version_guid=None, + course_id=None, revision=None, block=None): + self.check_course_locn_fields(testobj, msg, version_guid, course_id, + revision) + self.assertEqual(testobj.usage_id, block) diff --git a/common/lib/xmodule/xmodule/modulestore/tests/test_split_modulestore.py b/common/lib/xmodule/xmodule/modulestore/tests/test_split_modulestore.py new file mode 100644 index 0000000000..29f6cce919 --- /dev/null +++ b/common/lib/xmodule/xmodule/modulestore/tests/test_split_modulestore.py @@ -0,0 +1,992 @@ +''' +Created on Mar 25, 2013 + +@author: dmitchell +''' +import datetime +import subprocess +import unittest +import uuid +from importlib import import_module + +from xblock.core import Scope +from xmodule.course_module import CourseDescriptor +from xmodule.modulestore.exceptions import InsufficientSpecificationError, ItemNotFoundError, VersionConflictError +from xmodule.modulestore.locator import CourseLocator, BlockUsageLocator, VersionTree, DescriptionLocator +from pytz import UTC +from path import path +import re + + +class SplitModuleTest(unittest.TestCase): + ''' + The base set of tests manually populates a db w/ courses which have + versions. It creates unique collection names and removes them after all + tests finish. + ''' + # Snippet of what would be in the django settings envs file + modulestore_options = { + 'default_class': 'xmodule.raw_module.RawDescriptor', + 'host': 'localhost', + 'db': 'test_xmodule', + 'collection': 'modulestore{0}'.format(uuid.uuid4().hex), + 'fs_root': '', + } + + MODULESTORE = { + 'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore', + 'OPTIONS': modulestore_options + } + + # don't create django dependency; so, duplicates common.py in envs + match = re.search(r'(.*?/common)(?:$|/)', path(__file__)) + COMMON_ROOT = match.group(1) + + modulestore = None + + # These version_guids correspond to values hard-coded in fixture files + # used for these tests. The files live in mitx/fixtures/splitmongo_json/* + + GUID_D0 = "1d00000000000000dddd0000" # v12345d + GUID_D1 = "1d00000000000000dddd1111" # v12345d1 + GUID_D2 = "1d00000000000000dddd2222" # v23456d + GUID_D3 = "1d00000000000000dddd3333" # v12345d0 + GUID_D4 = "1d00000000000000dddd4444" # v23456d0 + GUID_D5 = "1d00000000000000dddd5555" # v345679d + GUID_P = "1d00000000000000eeee0000" # v23456p + + @staticmethod + def bootstrapDB(): + ''' + Loads the initial data into the db ensuring the collection name is + unique. + ''' + collection_prefix = SplitModuleTest.MODULESTORE['OPTIONS']['collection'] + '.' + dbname = SplitModuleTest.MODULESTORE['OPTIONS']['db'] + processes = [ + subprocess.Popen([ + 'mongoimport', '-d', dbname, '-c', + collection_prefix + collection, '--jsonArray', + '--file', + SplitModuleTest.COMMON_ROOT + '/test/data/splitmongo_json/' + collection + '.json' + ]) + for collection in ('active_versions', 'structures', 'definitions')] + for p in processes: + if p.wait() != 0: + raise Exception("DB did not init correctly") + + @classmethod + def tearDownClass(cls): + collection_prefix = SplitModuleTest.MODULESTORE['OPTIONS']['collection'] + '.' + if SplitModuleTest.modulestore: + for collection in ('active_versions', 'structures', 'definitions'): + modulestore().db.drop_collection(collection_prefix + collection) + # drop the modulestore to force re init + SplitModuleTest.modulestore = None + + def findByIdInResult(self, collection, _id): + """ + Result is a collection of descriptors. Find the one whose block id + matches the _id. + """ + for element in collection: + if element.location.usage_id == _id: + return element + + +class SplitModuleCourseTests(SplitModuleTest): + ''' + Course CRUD operation tests + ''' + + def test_get_courses(self): + courses = modulestore().get_courses('draft') + # should have gotten 3 draft courses + self.assertEqual(len(courses), 3, "Wrong number of courses") + # check metadata -- NOTE no promised order + course = self.findByIdInResult(courses, "head12345") + self.assertEqual(course.location.course_id, "GreekHero") + self.assertEqual( + str(course.location.version_guid), self.GUID_D0, + "course version mismatch" + ) + self.assertEqual(course.category, 'course', 'wrong category') + self.assertEqual(len(course.tabs), 6, "wrong number of tabs") + self.assertEqual( + course.display_name, "The Ancient Greek Hero", + "wrong display name" + ) + self.assertEqual( + course.advertised_start, "Fall 2013", + "advertised_start" + ) + self.assertEqual( + len(course.children), 3, + "children") + self.assertEqual(course.definition_locator.definition_id, "head12345_12") + # check dates and graders--forces loading of descriptor + self.assertEqual(course.edited_by, "testassist@edx.org") + self.assertEqual(str(course.previous_version), self.GUID_D1) + self.assertDictEqual(course.grade_cutoffs, {"Pass": 0.45}) + + def test_revision_requests(self): + # query w/ revision qualifier (both draft and published) + courses_published = modulestore().get_courses('published') + self.assertEqual(len(courses_published), 1, len(courses_published)) + course = self.findByIdInResult(courses_published, "head23456") + self.assertIsNotNone(course, "published courses") + self.assertEqual(course.location.course_id, "wonderful") + self.assertEqual(str(course.location.version_guid), self.GUID_P, + course.location.version_guid) + self.assertEqual(course.category, 'course', 'wrong category') + self.assertEqual(len(course.tabs), 4, "wrong number of tabs") + self.assertEqual(course.display_name, "The most wonderful course", + course.display_name) + self.assertIsNone(course.advertised_start) + self.assertEqual(len(course.children), 0, + "children") + + def test_search_qualifiers(self): + # query w/ search criteria + courses = modulestore().get_courses('draft', qualifiers={'org': 'testx'}) + self.assertEqual(len(courses), 2) + self.assertIsNotNone(self.findByIdInResult(courses, "head12345")) + self.assertIsNotNone(self.findByIdInResult(courses, "head23456")) + + courses = modulestore().get_courses( + 'draft', + qualifiers={'edited_on': {"$lt": datetime.datetime(2013, 3, 28, 15)}}) + self.assertEqual(len(courses), 2) + + courses = modulestore().get_courses( + 'draft', + qualifiers={'org': 'testx', "prettyid": "test_course"}) + self.assertEqual(len(courses), 1) + self.assertIsNotNone(self.findByIdInResult(courses, "head12345")) + + def test_get_course(self): + ''' + Test the various calling forms for get_course + ''' + locator = CourseLocator(version_guid=self.GUID_D1) + course = modulestore().get_course(locator) + self.assertIsNone(course.location.course_id) + self.assertEqual(str(course.location.version_guid), self.GUID_D1) + self.assertEqual(course.category, 'course') + self.assertEqual(len(course.tabs), 6) + self.assertEqual(course.display_name, "The Ancient Greek Hero") + self.assertIsNone(course.advertised_start) + self.assertEqual(len(course.children), 0) + self.assertEqual(course.definition_locator.definition_id, "head12345_11") + # check dates and graders--forces loading of descriptor + self.assertEqual(course.edited_by, "testassist@edx.org") + self.assertDictEqual(course.grade_cutoffs, {"Pass": 0.55}) + + locator = CourseLocator(course_id='GreekHero', revision='draft') + course = modulestore().get_course(locator) + self.assertEqual(course.location.course_id, "GreekHero") + self.assertEqual(str(course.location.version_guid), self.GUID_D0) + self.assertEqual(course.category, 'course') + self.assertEqual(len(course.tabs), 6) + self.assertEqual(course.display_name, "The Ancient Greek Hero") + self.assertEqual(course.advertised_start, "Fall 2013") + self.assertEqual(len(course.children), 3) + # check dates and graders--forces loading of descriptor + self.assertEqual(course.edited_by, "testassist@edx.org") + self.assertDictEqual(course.grade_cutoffs, {"Pass": 0.45}) + + locator = CourseLocator(course_id='wonderful', revision='published') + course = modulestore().get_course(locator) + self.assertEqual(course.location.course_id, "wonderful") + self.assertEqual(str(course.location.version_guid), self.GUID_P) + + locator = CourseLocator(course_id='wonderful', revision='draft') + course = modulestore().get_course(locator) + self.assertEqual(str(course.location.version_guid), self.GUID_D2) + + def test_get_course_negative(self): + # Now negative testing + self.assertRaises(InsufficientSpecificationError, + modulestore().get_course, CourseLocator(course_id='edu.meh.blah')) + self.assertRaises(ItemNotFoundError, + modulestore().get_course, CourseLocator(course_id='nosuchthing', revision='draft')) + self.assertRaises(ItemNotFoundError, + modulestore().get_course, + CourseLocator(course_id='GreekHero', revision='published')) + + def test_course_successors(self): + """ + get_course_successors(course_locator, version_history_depth=1) + """ + locator = CourseLocator(version_guid=self.GUID_D3) + result = modulestore().get_course_successors(locator) + self.assertIsInstance(result, VersionTree) + self.assertIsNone(result.locator.course_id) + self.assertEqual(str(result.locator.version_guid), self.GUID_D3) + self.assertEqual(len(result.children), 1) + self.assertEqual(str(result.children[0].locator.version_guid), self.GUID_D1) + self.assertEqual(len(result.children[0].children), 0, "descended more than one level") + result = modulestore().get_course_successors(locator, version_history_depth=2) + self.assertEqual(len(result.children), 1) + self.assertEqual(str(result.children[0].locator.version_guid), self.GUID_D1) + self.assertEqual(len(result.children[0].children), 1) + result = modulestore().get_course_successors(locator, version_history_depth=99) + self.assertEqual(len(result.children), 1) + self.assertEqual(str(result.children[0].locator.version_guid), self.GUID_D1) + self.assertEqual(len(result.children[0].children), 1) + + +class SplitModuleItemTests(SplitModuleTest): + ''' + Item read tests including inheritance + ''' + + def test_has_item(self): + ''' + has_item(BlockUsageLocator) + ''' + # positive tests of various forms + locator = BlockUsageLocator(version_guid=self.GUID_D1, usage_id='head12345') + self.assertTrue(modulestore().has_item(locator), + "couldn't find in %s" % self.GUID_D1) + + locator = BlockUsageLocator(course_id='GreekHero', usage_id='head12345', revision='draft') + self.assertTrue( + modulestore().has_item(locator), + "couldn't find in 12345" + ) + self.assertTrue( + modulestore().has_item(BlockUsageLocator( + course_id=locator.course_id, + revision='draft', + usage_id=locator.usage_id + )), + "couldn't find in draft 12345" + ) + self.assertFalse( + modulestore().has_item(BlockUsageLocator( + course_id=locator.course_id, + revision='published', + usage_id=locator.usage_id)), + "found in published 12345" + ) + locator.revision = 'draft' + self.assertTrue( + modulestore().has_item(locator), + "not found in draft 12345" + ) + + # not a course obj + locator = BlockUsageLocator(course_id='GreekHero', usage_id='chapter1', revision='draft') + self.assertTrue( + modulestore().has_item(locator), + "couldn't find chapter1" + ) + + # in published course + locator = BlockUsageLocator(course_id="wonderful", usage_id="head23456", revision='draft') + self.assertTrue(modulestore().has_item(BlockUsageLocator(course_id=locator.course_id, + usage_id=locator.usage_id, + revision='published')), + "couldn't find in 23456") + locator.revision = 'published' + self.assertTrue(modulestore().has_item(locator), "couldn't find in 23456") + + def test_negative_has_item(self): + # negative tests--not found + # no such course or block + locator = BlockUsageLocator(course_id="doesnotexist", usage_id="head23456", revision='draft') + self.assertFalse(modulestore().has_item(locator)) + locator = BlockUsageLocator(course_id="wonderful", usage_id="doesnotexist", revision='draft') + self.assertFalse(modulestore().has_item(locator)) + + # negative tests--insufficient specification + self.assertRaises(InsufficientSpecificationError, BlockUsageLocator) + self.assertRaises(InsufficientSpecificationError, + modulestore().has_item, BlockUsageLocator(version_guid=self.GUID_D1)) + self.assertRaises(InsufficientSpecificationError, + modulestore().has_item, BlockUsageLocator(course_id='GreekHero')) + + def test_get_item(self): + ''' + get_item(blocklocator) + ''' + # positive tests of various forms + locator = BlockUsageLocator(version_guid=self.GUID_D1, usage_id='head12345') + block = modulestore().get_item(locator) + self.assertIsInstance(block, CourseDescriptor) + + locator = BlockUsageLocator(course_id='GreekHero', usage_id='head12345', revision='draft') + block = modulestore().get_item(locator) + self.assertEqual(block.location.course_id, "GreekHero") + # look at this one in detail + self.assertEqual(len(block.tabs), 6, "wrong number of tabs") + self.assertEqual(block.display_name, "The Ancient Greek Hero") + self.assertEqual(block.advertised_start, "Fall 2013") + self.assertEqual(len(block.children), 3) + self.assertEqual(block.definition_locator.definition_id, "head12345_12") + # check dates and graders--forces loading of descriptor + self.assertEqual(block.edited_by, "testassist@edx.org") + self.assertDictEqual( + block.grade_cutoffs, {"Pass": 0.45}, + ) + + # try to look up other revisions + self.assertRaises(ItemNotFoundError, + modulestore().get_item, + BlockUsageLocator(course_id=locator.as_course_locator(), + usage_id=locator.usage_id, + revision='published')) + locator.revision = 'draft' + self.assertIsInstance( + modulestore().get_item(locator), + CourseDescriptor + ) + + def test_get_non_root(self): + # not a course obj + locator = BlockUsageLocator(course_id='GreekHero', usage_id='chapter1', revision='draft') + block = modulestore().get_item(locator) + self.assertEqual(block.location.course_id, "GreekHero") + self.assertEqual(block.category, 'chapter') + self.assertEqual(block.definition_locator.definition_id, "chapter12345_1") + self.assertEqual(block.display_name, "Hercules") + self.assertEqual(block.edited_by, "testassist@edx.org") + + # in published course + locator = BlockUsageLocator(course_id="wonderful", usage_id="head23456", revision='published') + self.assertIsInstance( + modulestore().get_item(locator), + CourseDescriptor + ) + + # negative tests--not found + # no such course or block + locator = BlockUsageLocator(course_id="doesnotexist", usage_id="head23456", revision='draft') + with self.assertRaises(ItemNotFoundError): + modulestore().get_item(locator) + locator = BlockUsageLocator(course_id="wonderful", usage_id="doesnotexist", revision='draft') + with self.assertRaises(ItemNotFoundError): + modulestore().get_item(locator) + + # negative tests--insufficient specification + with self.assertRaises(InsufficientSpecificationError): + modulestore().get_item(BlockUsageLocator(version_guid=self.GUID_D1)) + with self.assertRaises(InsufficientSpecificationError): + modulestore().get_item(BlockUsageLocator(course_id='GreekHero', revision='draft')) + + # pylint: disable=W0212 + def test_matching(self): + ''' + test the block and value matches help functions + ''' + self.assertTrue(modulestore()._value_matches('help', 'help')) + self.assertFalse(modulestore()._value_matches('help', 'Help')) + self.assertTrue(modulestore()._value_matches(['distract', 'help', 'notme'], 'help')) + self.assertFalse(modulestore()._value_matches(['distract', 'Help', 'notme'], 'help')) + self.assertFalse(modulestore()._value_matches({'field': ['distract', 'Help', 'notme']}, {'field': 'help'})) + self.assertFalse(modulestore()._value_matches(['distract', 'Help', 'notme'], {'field': 'help'})) + self.assertTrue(modulestore()._value_matches( + {'field': ['distract', 'help', 'notme'], + 'irrelevant': 2}, + {'field': 'help'})) + self.assertTrue(modulestore()._value_matches('I need some help', {'$regex': 'help'})) + self.assertTrue(modulestore()._value_matches(['I need some help', 'today'], {'$regex': 'help'})) + self.assertFalse(modulestore()._value_matches('I need some help', {'$regex': 'Help'})) + self.assertFalse(modulestore()._value_matches(['I need some help', 'today'], {'$regex': 'Help'})) + + self.assertTrue(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 1})) + self.assertTrue(modulestore()._block_matches({'a': 1, 'b': 2}, {'c': None})) + self.assertTrue(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 1, 'c': None})) + self.assertFalse(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 2})) + self.assertFalse(modulestore()._block_matches({'a': 1, 'b': 2}, {'c': 1})) + self.assertFalse(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 1, 'c': 1})) + + def test_get_items(self): + ''' + get_items(locator, qualifiers, [revision]) + ''' + locator = CourseLocator(version_guid=self.GUID_D0) + # get all modules + matches = modulestore().get_items(locator, {}) + self.assertEqual(len(matches), 6) + matches = modulestore().get_items(locator, {'category': 'chapter'}) + self.assertEqual(len(matches), 3) + matches = modulestore().get_items(locator, {'category': 'garbage'}) + self.assertEqual(len(matches), 0) + matches = modulestore().get_items( + locator, + { + 'category': 'chapter', + 'metadata': {'display_name': {'$regex': 'Hera'}} + } + ) + self.assertEqual(len(matches), 2) + + matches = modulestore().get_items(locator, {'children': 'chapter2'}) + self.assertEqual(len(matches), 1) + self.assertEqual(matches[0].location.usage_id, 'head12345') + + def test_get_parents(self): + ''' + get_parent_locations(locator, [usage_id], [revision]): [BlockUsageLocator] + ''' + locator = CourseLocator(course_id="GreekHero", revision='draft') + parents = modulestore().get_parent_locations(locator, usage_id='chapter1') + self.assertEqual(len(parents), 1) + self.assertEqual(parents[0].usage_id, 'head12345') + self.assertEqual(parents[0].course_id, "GreekHero") + locator.usage_id = 'chapter2' + parents = modulestore().get_parent_locations(locator) + self.assertEqual(len(parents), 1) + self.assertEqual(parents[0].usage_id, 'head12345') + parents = modulestore().get_parent_locations(locator, usage_id='nosuchblock') + self.assertEqual(len(parents), 0) + + def test_get_children(self): + """ + Test the existing get_children method on xdescriptors + """ + locator = BlockUsageLocator(course_id="GreekHero", usage_id="head12345", revision='draft') + block = modulestore().get_item(locator) + children = block.get_children() + expected_ids = [ + "chapter1", "chapter2", "chapter3" + ] + for child in children: + self.assertEqual(child.category, "chapter") + self.assertIn(child.location.usage_id, expected_ids) + expected_ids.remove(child.location.usage_id) + self.assertEqual(len(expected_ids), 0) + + +class TestItemCrud(SplitModuleTest): + """ + Test create update and delete of items + """ + # TODO do I need to test this case which I believe won't work: + # 1) fetch a course and some of its blocks + # 2) do a series of CRUD operations on those previously fetched elements + # The problem here will be that the version_guid of the items will be the version at time of fetch. + # Each separate save will change the head version; so, the 2nd piecemeal change will flag the version + # conflict. That is, if versions are v0..vn and start as v0 in initial fetch, the first CRUD op will + # say it's changing an object from v0, splitMongo will process it and make the current head v1, the next + # crud op will pass in its v0 element and splitMongo will flag the version conflict. + # What I don't know is how realistic this test is and whether to wrap the modulestore with a higher level + # transactional operation which manages the version change or make the threading cache reason out whether or + # not the changes are independent and additive and thus non-conflicting. + # A use case I expect is + # (client) change this metadata + # (server) done, here's the new info which, btw, updates the course version to v1 + # (client) add these children to this other node (which says it came from v0 or + # will the client have refreshed the version before doing the op?) + # In this case, having a server side transactional model won't help b/c the bug is a long-transaction on the + # on the client where it would be a mistake for the server to assume anything about client consistency. The best + # the server could do would be to see if the parent's children changed at all since v0. + + def test_create_minimal_item(self): + """ + create_item(course_or_parent_locator, category, user, definition_locator=None, new_def_data=None, + metadata=None): new_desciptor + """ + # grab link to course to ensure new versioning works + locator = CourseLocator(course_id="GreekHero", revision='draft') + premod_course = modulestore().get_course(locator) + premod_time = datetime.datetime.now(UTC) - datetime.timedelta(seconds=1) + # add minimal one w/o a parent + category = 'sequential' + new_module = modulestore().create_item( + locator, category, 'user123', + metadata={'display_name': 'new sequential'} + ) + # check that course version changed and course's previous is the other one + self.assertEqual(new_module.location.course_id, "GreekHero") + self.assertNotEqual(new_module.location.version_guid, premod_course.location.version_guid) + self.assertIsNone(locator.version_guid, "Version inadvertently filled in") + current_course = modulestore().get_course(locator) + self.assertEqual(new_module.location.version_guid, current_course.location.version_guid) + + history_info = modulestore().get_course_history_info(current_course.location) + self.assertEqual(history_info['previous_version'], premod_course.location.version_guid) + self.assertEqual(str(history_info['original_version']), self.GUID_D3) + self.assertEqual(history_info['edited_by'], "user123") + self.assertGreaterEqual(history_info['edited_on'], premod_time) + self.assertLessEqual(history_info['edited_on'], datetime.datetime.now(UTC)) + # check block's info: category, definition_locator, and display_name + self.assertEqual(new_module.category, 'sequential') + self.assertIsNotNone(new_module.definition_locator) + self.assertEqual(new_module.display_name, 'new sequential') + # check that block does not exist in previous version + locator = BlockUsageLocator( + version_guid=premod_course.location.version_guid, + usage_id=new_module.location.usage_id + ) + self.assertRaises(ItemNotFoundError, modulestore().get_item, locator) + + def test_create_parented_item(self): + """ + Test create_item w/ specifying the parent of the new item + """ + locator = BlockUsageLocator(course_id="wonderful", usage_id="head23456", revision='draft') + premod_course = modulestore().get_course(locator) + category = 'chapter' + new_module = modulestore().create_item( + locator, category, 'user123', + metadata={'display_name': 'new chapter'}, + definition_locator=DescriptionLocator("chapter12345_2") + ) + # check that course version changed and course's previous is the other one + self.assertNotEqual(new_module.location.version_guid, premod_course.location.version_guid) + parent = modulestore().get_item(locator) + self.assertIn(new_module.location.usage_id, parent.children) + self.assertEqual(new_module.definition_locator.definition_id, "chapter12345_2") + + def test_unique_naming(self): + """ + Check that 2 modules of same type get unique usage_ids. Also check that if creation provides + a definition id and new def data that it branches the definition in the db. + Actually, this tries to test all create_item features not tested above. + """ + locator = BlockUsageLocator(course_id="contender", usage_id="head345679", revision='draft') + category = 'problem' + premod_time = datetime.datetime.now(UTC) - datetime.timedelta(seconds=1) + new_payload = "empty" + new_module = modulestore().create_item( + locator, category, 'anotheruser', + metadata={'display_name': 'problem 1'}, + new_def_data=new_payload + ) + another_payload = "not empty" + another_module = modulestore().create_item( + locator, category, 'anotheruser', + metadata={'display_name': 'problem 2'}, + definition_locator=DescriptionLocator("problem12345_3_1"), + new_def_data=another_payload + ) + # check that course version changed and course's previous is the other one + parent = modulestore().get_item(locator) + self.assertNotEqual(new_module.location.usage_id, another_module.location.usage_id) + self.assertIn(new_module.location.usage_id, parent.children) + self.assertIn(another_module.location.usage_id, parent.children) + self.assertEqual(new_module.data, new_payload) + self.assertEqual(another_module.data, another_payload) + # check definition histories + new_history = modulestore().get_definition_history_info(new_module.definition_locator) + self.assertIsNone(new_history['previous_version']) + self.assertEqual(new_history['original_version'], new_module.definition_locator.definition_id) + self.assertEqual(new_history['edited_by'], "anotheruser") + self.assertLessEqual(new_history['edited_on'], datetime.datetime.now(UTC)) + self.assertGreaterEqual(new_history['edited_on'], premod_time) + another_history = modulestore().get_definition_history_info(another_module.definition_locator) + self.assertEqual(another_history['previous_version'], 'problem12345_3_1') + # TODO check that default fields are set + + def test_update_metadata(self): + """ + test updating an items metadata ensuring the definition doesn't version but the course does if it should + """ + locator = BlockUsageLocator(course_id="GreekHero", usage_id="problem3_2", revision='draft') + problem = modulestore().get_item(locator) + pre_def_id = problem.definition_locator.definition_id + pre_version_guid = problem.location.version_guid + self.assertIsNotNone(pre_def_id) + self.assertIsNotNone(pre_version_guid) + premod_time = datetime.datetime.now(UTC) - datetime.timedelta(seconds=1) + self.assertNotEqual(problem.max_attempts, 4, "Invalidates rest of test") + + problem.max_attempts = 4 + updated_problem = modulestore().update_item(problem, 'changeMaven') + # check that course version changed and course's previous is the other one + self.assertEqual(updated_problem.definition_locator.definition_id, pre_def_id) + self.assertNotEqual(updated_problem.location.version_guid, pre_version_guid) + self.assertEqual(updated_problem.max_attempts, 4) + # refetch to ensure original didn't change + original_location = BlockUsageLocator( + version_guid=pre_version_guid, + usage_id=problem.location.usage_id + ) + problem = modulestore().get_item(original_location) + self.assertNotEqual(problem.max_attempts, 4, "original changed") + + current_course = modulestore().get_course(locator) + self.assertEqual(updated_problem.location.version_guid, current_course.location.version_guid) + + history_info = modulestore().get_course_history_info(current_course.location) + self.assertEqual(history_info['previous_version'], pre_version_guid) + self.assertEqual(str(history_info['original_version']), self.GUID_D3) + self.assertEqual(history_info['edited_by'], "changeMaven") + self.assertGreaterEqual(history_info['edited_on'], premod_time) + self.assertLessEqual(history_info['edited_on'], datetime.datetime.now(UTC)) + + def test_update_children(self): + """ + test updating an item's children ensuring the definition doesn't version but the course does if it should + """ + locator = BlockUsageLocator(course_id="GreekHero", usage_id="chapter3", revision='draft') + block = modulestore().get_item(locator) + pre_def_id = block.definition_locator.definition_id + pre_version_guid = block.location.version_guid + + # reorder children + self.assertGreater(len(block.children), 0, "meaningless test") + moved_child = block.children.pop() + updated_problem = modulestore().update_item(block, 'childchanger') + # check that course version changed and course's previous is the other one + self.assertEqual(updated_problem.definition_locator.definition_id, pre_def_id) + self.assertNotEqual(updated_problem.location.version_guid, pre_version_guid) + self.assertEqual(updated_problem.children, block.children) + self.assertNotIn(moved_child, updated_problem.children) + locator.usage_id = "chapter1" + other_block = modulestore().get_item(locator) + other_block.children.append(moved_child) + other_updated = modulestore().update_item(other_block, 'childchanger') + self.assertIn(moved_child, other_updated.children) + + def test_update_definition(self): + """ + test updating an item's definition: ensure it gets versioned as well as the course getting versioned + """ + locator = BlockUsageLocator(course_id="GreekHero", usage_id="head12345", revision='draft') + block = modulestore().get_item(locator) + pre_def_id = block.definition_locator.definition_id + pre_version_guid = block.location.version_guid + + block.grading_policy['GRADER'][0]['min_count'] = 13 + updated_block = modulestore().update_item(block, 'definition_changer') + + self.assertNotEqual(updated_block.definition_locator.definition_id, pre_def_id) + self.assertNotEqual(updated_block.location.version_guid, pre_version_guid) + self.assertEqual(updated_block.grading_policy['GRADER'][0]['min_count'], 13) + + def test_update_manifold(self): + """ + Test updating metadata, children, and definition in a single call ensuring all the versioning occurs + """ + # first add 2 children to the course for the update to manipulate + locator = BlockUsageLocator(course_id="contender", usage_id="head345679", revision='draft') + category = 'problem' + new_payload = "empty" + modulestore().create_item( + locator, category, 'test_update_manifold', + metadata={'display_name': 'problem 1'}, + new_def_data=new_payload + ) + another_payload = "not empty" + modulestore().create_item( + locator, category, 'test_update_manifold', + metadata={'display_name': 'problem 2'}, + definition_locator=DescriptionLocator("problem12345_3_1"), + new_def_data=another_payload + ) + # pylint: disable=W0212 + modulestore()._clear_cache() + + # now begin the test + block = modulestore().get_item(locator) + pre_def_id = block.definition_locator.definition_id + pre_version_guid = block.location.version_guid + + self.assertNotEqual(block.grading_policy['GRADER'][0]['min_count'], 13) + block.grading_policy['GRADER'][0]['min_count'] = 13 + block.children = block.children[1:] + [block.children[0]] + block.advertised_start = "Soon" + + updated_block = modulestore().update_item(block, "test_update_manifold") + self.assertNotEqual(updated_block.definition_locator.definition_id, pre_def_id) + self.assertNotEqual(updated_block.location.version_guid, pre_version_guid) + self.assertEqual(updated_block.grading_policy['GRADER'][0]['min_count'], 13) + self.assertEqual(updated_block.children[0], block.children[0]) + self.assertEqual(updated_block.advertised_start, "Soon") + + def test_delete_item(self): + course = self.create_course_for_deletion() + self.assertRaises(ValueError, + modulestore().delete_item, + course.location, + 'deleting_user') + reusable_location = BlockUsageLocator( + course_id=course.location.course_id, + usage_id=course.location.usage_id, + revision='draft') + + # delete a leaf + problems = modulestore().get_items(reusable_location, {'category': 'problem'}) + locn_to_del = problems[0].location + new_course_loc = modulestore().delete_item(locn_to_del, 'deleting_user') + deleted = BlockUsageLocator(course_id=reusable_location.course_id, + revision=reusable_location.revision, + usage_id=locn_to_del.usage_id) + self.assertFalse(modulestore().has_item(deleted)) + self.assertRaises(VersionConflictError, modulestore().has_item, locn_to_del) + locator = BlockUsageLocator( + version_guid=locn_to_del.version_guid, + usage_id=locn_to_del.usage_id + ) + self.assertTrue(modulestore().has_item(locator)) + self.assertNotEqual(new_course_loc.version_guid, course.location.version_guid) + + # delete a subtree + nodes = modulestore().get_items(reusable_location, {'category': 'chapter'}) + new_course_loc = modulestore().delete_item(nodes[0].location, 'deleting_user') + # check subtree + + def check_subtree(node): + if node: + node_loc = node.location + self.assertFalse(modulestore().has_item( + BlockUsageLocator( + course_id=node_loc.course_id, + revision=node_loc.revision, + usage_id=node.location.usage_id))) + locator = BlockUsageLocator( + version_guid=node.location.version_guid, + usage_id=node.location.usage_id) + self.assertTrue(modulestore().has_item(locator)) + if node.has_children: + for sub in node.get_children(): + check_subtree(sub) + check_subtree(nodes[0]) + + def create_course_for_deletion(self): + course = modulestore().create_course('nihilx', 'deletion', 'deleting_user') + root = BlockUsageLocator( + course_id=course.location.course_id, + usage_id=course.location.usage_id, + revision='draft') + for _ in range(4): + self.create_subtree_for_deletion(root, ['chapter', 'vertical', 'problem']) + return modulestore().get_item(root) + + def create_subtree_for_deletion(self, parent, category_queue): + if not category_queue: + return + node = modulestore().create_item(parent, category_queue[0], 'deleting_user') + node_loc = BlockUsageLocator(parent.as_course_locator(), usage_id=node.location.usage_id) + for _ in range(4): + self.create_subtree_for_deletion(node_loc, category_queue[1:]) + + +class TestCourseCreation(SplitModuleTest): + """ + Test create_course, duh :-) + """ + def test_simple_creation(self): + """ + The simplest case but probing all expected results from it. + """ + # Oddly getting differences of 200nsec + pre_time = datetime.datetime.now(UTC) - datetime.timedelta(milliseconds=1) + new_course = modulestore().create_course('test_org', 'test_course', 'create_user') + new_locator = new_course.location + # check index entry + index_info = modulestore().get_course_index_info(new_locator) + self.assertEqual(index_info['org'], 'test_org') + self.assertEqual(index_info['prettyid'], 'test_course') + self.assertGreaterEqual(index_info["edited_on"], pre_time) + self.assertLessEqual(index_info["edited_on"], datetime.datetime.now(UTC)) + self.assertEqual(index_info['edited_by'], 'create_user') + # check structure info + structure_info = modulestore().get_course_history_info(new_locator) + self.assertEqual(structure_info['original_version'], index_info['versions']['draft']) + self.assertIsNone(structure_info['previous_version']) + self.assertGreaterEqual(structure_info["edited_on"], pre_time) + self.assertLessEqual(structure_info["edited_on"], datetime.datetime.now(UTC)) + self.assertEqual(structure_info['edited_by'], 'create_user') + # check the returned course object + self.assertIsInstance(new_course, CourseDescriptor) + self.assertEqual(new_course.category, 'course') + self.assertFalse(new_course.show_calculator) + self.assertTrue(new_course.allow_anonymous) + self.assertEqual(len(new_course.children), 0) + self.assertEqual(new_course.edited_by, "create_user") + self.assertEqual(len(new_course.grading_policy['GRADER']), 4) + self.assertDictEqual(new_course.grade_cutoffs, {"Pass": 0.5}) + + def test_cloned_course(self): + """ + Test making a course which points to an existing draft and published but not making any changes to either. + """ + pre_time = datetime.datetime.now(UTC) + original_locator = CourseLocator(course_id="wonderful", revision='draft') + original_index = modulestore().get_course_index_info(original_locator) + new_draft = modulestore().create_course( + 'leech', 'best_course', 'leech_master', id_root='best', + versions_dict=original_index['versions']) + new_draft_locator = new_draft.location + self.assertRegexpMatches(new_draft_locator.course_id, r'best.*') + # the edited_by and other meta fields on the new course will be the original author not this one + self.assertEqual(new_draft.edited_by, 'test@edx.org') + self.assertLess(new_draft.edited_on, pre_time) + self.assertEqual(new_draft.location.version_guid, original_index['versions']['draft']) + # however the edited_by and other meta fields on course_index will be this one + new_index = modulestore().get_course_index_info(new_draft_locator) + self.assertGreaterEqual(new_index["edited_on"], pre_time) + self.assertLessEqual(new_index["edited_on"], datetime.datetime.now(UTC)) + self.assertEqual(new_index['edited_by'], 'leech_master') + + new_published_locator = CourseLocator(course_id=new_draft_locator.course_id, revision='published') + new_published = modulestore().get_course(new_published_locator) + self.assertEqual(new_published.edited_by, 'test@edx.org') + self.assertLess(new_published.edited_on, pre_time) + self.assertEqual(new_published.location.version_guid, original_index['versions']['published']) + + # changing this course will not change the original course + # using new_draft.location will insert the chapter under the course root + new_item = modulestore().create_item( + new_draft.location, 'chapter', 'leech_master', + metadata={'display_name': 'new chapter'} + ) + new_draft_locator.version_guid = None + new_index = modulestore().get_course_index_info(new_draft_locator) + self.assertNotEqual(new_index['versions']['draft'], original_index['versions']['draft']) + new_draft = modulestore().get_course(new_draft_locator) + self.assertEqual(new_item.edited_by, 'leech_master') + self.assertGreaterEqual(new_item.edited_on, pre_time) + self.assertNotEqual(new_item.location.version_guid, original_index['versions']['draft']) + self.assertNotEqual(new_draft.location.version_guid, original_index['versions']['draft']) + structure_info = modulestore().get_course_history_info(new_draft_locator) + self.assertGreaterEqual(structure_info["edited_on"], pre_time) + self.assertLessEqual(structure_info["edited_on"], datetime.datetime.now(UTC)) + self.assertEqual(structure_info['edited_by'], 'leech_master') + + original_course = modulestore().get_course(original_locator) + self.assertEqual(original_course.location.version_guid, original_index['versions']['draft']) + self.assertFalse( + modulestore().has_item(BlockUsageLocator( + original_locator, + usage_id=new_item.location.usage_id + )) + ) + + def test_derived_course(self): + """ + Create a new course which overrides metadata and course_data + """ + pre_time = datetime.datetime.now(UTC) + original_locator = CourseLocator(course_id="contender", revision='draft') + original = modulestore().get_course(original_locator) + original_index = modulestore().get_course_index_info(original_locator) + data_payload = {} + metadata_payload = {} + for field in original.fields: + if field.scope == Scope.content and field.name != 'location': + data_payload[field.name] = getattr(original, field.name) + elif field.scope == Scope.settings: + metadata_payload[field.name] = getattr(original, field.name) + data_payload['grading_policy']['GRADE_CUTOFFS'] = {'A': .9, 'B': .8, 'C': .65} + metadata_payload['display_name'] = 'Derivative' + new_draft = modulestore().create_course( + 'leech', 'derivative', 'leech_master', id_root='counter', + versions_dict={'draft': original_index['versions']['draft']}, + course_data=data_payload, + metadata=metadata_payload + ) + new_draft_locator = new_draft.location + self.assertRegexpMatches(new_draft_locator.course_id, r'counter.*') + # the edited_by and other meta fields on the new course will be the original author not this one + self.assertEqual(new_draft.edited_by, 'leech_master') + self.assertGreaterEqual(new_draft.edited_on, pre_time) + self.assertNotEqual(new_draft.location.version_guid, original_index['versions']['draft']) + # however the edited_by and other meta fields on course_index will be this one + new_index = modulestore().get_course_index_info(new_draft_locator) + self.assertGreaterEqual(new_index["edited_on"], pre_time) + self.assertLessEqual(new_index["edited_on"], datetime.datetime.now(UTC)) + self.assertEqual(new_index['edited_by'], 'leech_master') + self.assertEqual(new_draft.display_name, metadata_payload['display_name']) + self.assertDictEqual( + new_draft.grading_policy['GRADE_CUTOFFS'], + data_payload['grading_policy']['GRADE_CUTOFFS'] + ) + + def test_update_course_index(self): + """ + Test changing the org, pretty id, etc of a course. Test that it doesn't allow changing the id, etc. + """ + locator = CourseLocator(course_id="GreekHero", revision='draft') + modulestore().update_course_index(locator, {'org': 'funkyU'}) + course_info = modulestore().get_course_index_info(locator) + self.assertEqual(course_info['org'], 'funkyU') + + modulestore().update_course_index(locator, {'org': 'moreFunky', 'prettyid': 'Ancient Greek Demagods'}) + course_info = modulestore().get_course_index_info(locator) + self.assertEqual(course_info['org'], 'moreFunky') + self.assertEqual(course_info['prettyid'], 'Ancient Greek Demagods') + + self.assertRaises(ValueError, modulestore().update_course_index, locator, {'_id': 'funkygreeks'}) + + with self.assertRaises(ValueError): + modulestore().update_course_index( + locator, + {'edited_on': datetime.datetime.now(UTC)} + ) + with self.assertRaises(ValueError): + modulestore().update_course_index( + locator, + {'edited_by': 'sneak'} + ) + + self.assertRaises(ValueError, modulestore().update_course_index, locator, + {'versions': {'draft': self.GUID_D1}}) + + # an allowed but not necessarily recommended way to revert the draft version + versions = course_info['versions'] + versions['draft'] = self.GUID_D1 + modulestore().update_course_index(locator, {'versions': versions}, update_versions=True) + course = modulestore().get_course(locator) + self.assertEqual(str(course.location.version_guid), self.GUID_D1) + + # an allowed but not recommended way to publish a course + versions['published'] = self.GUID_D1 + modulestore().update_course_index(locator, {'versions': versions}, update_versions=True) + course = modulestore().get_course(CourseLocator(course_id=locator.course_id, revision="published")) + self.assertEqual(str(course.location.version_guid), self.GUID_D1) + + +class TestInheritance(SplitModuleTest): + """ + Test the metadata inheritance mechanism. + """ + def test_inheritance(self): + """ + The actual test + """ + # Note, not testing value where defined (course) b/c there's no + # defined accessor for it on CourseDescriptor. + locator = BlockUsageLocator(course_id="GreekHero", usage_id="problem3_2", revision='draft') + node = modulestore().get_item(locator) + # inherited + self.assertEqual(node.graceperiod, datetime.timedelta(hours=2)) + locator = BlockUsageLocator(course_id="GreekHero", usage_id="problem1", revision='draft') + node = modulestore().get_item(locator) + # overridden + self.assertEqual(node.graceperiod, datetime.timedelta(hours=4)) + + # TODO test inheritance after set and delete of attrs + + +#=========================================== +# This mocks the django.modulestore() function and is intended purely to disentangle +# the tests from django +def modulestore(): + def load_function(path): + module_path, _, name = path.rpartition('.') + return getattr(import_module(module_path), name) + + if SplitModuleTest.modulestore is None: + SplitModuleTest.bootstrapDB() + class_ = load_function(SplitModuleTest.MODULESTORE['ENGINE']) + + options = {} + + options.update(SplitModuleTest.MODULESTORE['OPTIONS']) + options['render_template'] = render_to_template_mock + + # pylint: disable=W0142 + SplitModuleTest.modulestore = class_(**options) + + return SplitModuleTest.modulestore + + +# pylint: disable=W0613 +def render_to_template_mock(*args): + pass diff --git a/common/lib/xmodule/xmodule/modulestore/xml_importer.py b/common/lib/xmodule/xmodule/modulestore/xml_importer.py index 71c6983644..8df768f442 100644 --- a/common/lib/xmodule/xmodule/modulestore/xml_importer.py +++ b/common/lib/xmodule/xmodule/modulestore/xml_importer.py @@ -138,7 +138,7 @@ def import_module_from_xml(modulestore, static_content_store, course_data_path, # For example, what I'm seeing is -> # Note the dropped element closing tag. This causes the LMS to fail when rendering modules - that's # no good, so we have to do this kludge - if isinstance(module.data, str) or isinstance(module.data, unicode): # some module 'data' fields are non strings which blows up the link traversal code + if isinstance(module.data, str) or isinstance(module.data, unicode): # some module 'data' fields are non strings which blows up the link traversal code lxml_rewrite_links(module.data, lambda link: verify_content_links(module, course_data_path, static_content_store, link, remap_dict)) for key in remap_dict.keys(): @@ -315,7 +315,7 @@ def import_module(module, store, course_data_path, static_content_store, allow_n # For example, what I'm seeing is -> # Note the dropped element closing tag. This causes the LMS to fail when rendering modules - that's # no good, so we have to do this kludge - if isinstance(module_data, str) or isinstance(module_data, unicode): # some module 'data' fields are non strings which blows up the link traversal code + if isinstance(module_data, str) or isinstance(module_data, unicode): # some module 'data' fields are non strings which blows up the link traversal code lxml_rewrite_links(module_data, lambda link: verify_content_links(module, course_data_path, static_content_store, link, remap_dict)) for key in remap_dict.keys(): @@ -523,6 +523,26 @@ def validate_data_source_paths(data_dir, course_dir): return err_cnt, warn_cnt +def validate_course_policy(module_store, course_id): + """ + Validate that the course explicitly sets values for any fields whose defaults may have changed between + the export and the import. + + Does not add to error count as these are just warnings. + """ + # is there a reliable way to get the module location just given the course_id? + warn_cnt = 0 + for module in module_store.modules[course_id].itervalues(): + if module.location.category == 'course': + if not 'rerandomize' in module._model_data: + warn_cnt += 1 + print 'WARN: course policy does not specify value for "rerandomize" whose default is now "never". The behavior of your course may change.' + if not 'showanswer' in module._model_data: + warn_cnt += 1 + print 'WARN: course policy does not specify value for "showanswer" whose default is now "finished". The behavior of your course may change.' + return warn_cnt + + def perform_xlint(data_dir, course_dirs, default_class='xmodule.raw_module.RawDescriptor', load_error_modules=True): @@ -568,6 +588,8 @@ def perform_xlint(data_dir, course_dirs, err_cnt += validate_category_hierarchy(module_store, course_id, "chapter", "sequential") # constrain that sequentials only have 'verticals' err_cnt += validate_category_hierarchy(module_store, course_id, "sequential", "vertical") + # validate the course policy overrides any defaults which have changed over time + warn_cnt += validate_course_policy(module_store, course_id) # don't allow metadata on verticals, since we can't edit them in studio err_cnt += validate_no_non_editable_metadata(module_store, course_id, "vertical") # don't allow metadata on chapters, since we can't edit them in studio diff --git a/common/lib/xmodule/xmodule/open_ended_grading_classes/open_ended_module.py b/common/lib/xmodule/xmodule/open_ended_grading_classes/open_ended_module.py index 8d8a85f788..2e7a3eaf89 100644 --- a/common/lib/xmodule/xmodule/open_ended_grading_classes/open_ended_module.py +++ b/common/lib/xmodule/xmodule/open_ended_grading_classes/open_ended_module.py @@ -19,6 +19,7 @@ import openendedchild from numpy import median from datetime import datetime +from pytz import UTC from .combined_open_ended_rubric import CombinedOpenEndedRubric @@ -170,7 +171,7 @@ class OpenEndedModule(openendedchild.OpenEndedChild): if xqueue is None: return {'success': False, 'msg': "Couldn't submit feedback."} qinterface = xqueue['interface'] - qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat) + qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat) anonymous_student_id = system.anonymous_student_id queuekey = xqueue_interface.make_hashkey(str(system.seed) + qtime + anonymous_student_id + @@ -224,7 +225,7 @@ class OpenEndedModule(openendedchild.OpenEndedChild): if xqueue is None: return False qinterface = xqueue['interface'] - qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat) + qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat) anonymous_student_id = system.anonymous_student_id diff --git a/common/lib/xmodule/xmodule/open_ended_grading_classes/openendedchild.py b/common/lib/xmodule/xmodule/open_ended_grading_classes/openendedchild.py index 047ab0244c..10f939b270 100644 --- a/common/lib/xmodule/xmodule/open_ended_grading_classes/openendedchild.py +++ b/common/lib/xmodule/xmodule/open_ended_grading_classes/openendedchild.py @@ -5,6 +5,7 @@ import re import open_ended_image_submission from xmodule.progress import Progress +import capa.xqueue_interface as xqueue_interface from capa.util import * from .peer_grading_service import PeerGradingService, MockPeerGradingService import controller_query_service @@ -334,12 +335,15 @@ class OpenEndedChild(object): log.exception("Could not create image and check it.") if image_ok: - image_key = image_data.name + datetime.now().strftime("%Y%m%d%H%M%S") + image_key = image_data.name + datetime.now(UTC).strftime( + xqueue_interface.dateformat + ) try: image_data.seek(0) - success, s3_public_url = open_ended_image_submission.upload_to_s3(image_data, image_key, - self.s3_interface) + success, s3_public_url = open_ended_image_submission.upload_to_s3( + image_data, image_key, self.s3_interface + ) except: log.exception("Could not upload image to S3.") diff --git a/common/lib/xmodule/xmodule/progress.py b/common/lib/xmodule/xmodule/progress.py index bad5105fd0..4e3a8bc93e 100644 --- a/common/lib/xmodule/xmodule/progress.py +++ b/common/lib/xmodule/xmodule/progress.py @@ -146,7 +146,7 @@ class Progress(object): sending Progress objects to js to limit dependencies. ''' if progress is None: - return "NA" + return "0" return progress.ternary_str() @staticmethod @@ -157,5 +157,5 @@ class Progress(object): passing Progress objects to js to limit dependencies. ''' if progress is None: - return "NA" + return "0" return str(progress) diff --git a/common/lib/xmodule/xmodule/schematic_module.py b/common/lib/xmodule/xmodule/schematic_module.py deleted file mode 100644 index 83bcc5351d..0000000000 --- a/common/lib/xmodule/xmodule/schematic_module.py +++ /dev/null @@ -1,11 +0,0 @@ - -from .x_module import XModule, XModuleDescriptor - - -class ModuleDescriptor(XModuleDescriptor): - pass - - -class Module(XModule): - def get_html(self): - return ''.format(item_id=self.item_id) diff --git a/common/lib/xmodule/xmodule/tests/test_capa_module.py b/common/lib/xmodule/xmodule/tests/test_capa_module.py index 0f3dfa5b85..80c4e41e8f 100644 --- a/common/lib/xmodule/xmodule/tests/test_capa_module.py +++ b/common/lib/xmodule/xmodule/tests/test_capa_module.py @@ -1233,6 +1233,37 @@ class CapaModuleTest(unittest.TestCase): mock_log.exception.assert_called_once_with('Got bad progress') mock_log.reset_mock() + @patch('xmodule.capa_module.Progress') + def test_get_progress_calculate_progress_fraction(self, mock_progress): + """ + Check that score and total are calculated correctly for the progress fraction. + """ + module = CapaFactory.create() + module.weight = 1 + module.get_progress() + mock_progress.assert_called_with(0, 1) + + other_module = CapaFactory.create(correct=True) + other_module.weight = 1 + other_module.get_progress() + mock_progress.assert_called_with(1, 1) + + def test_get_html(self): + """ + Check that get_html() calls get_progress() with no arguments. + """ + module = CapaFactory.create() + module.get_progress = Mock(wraps=module.get_progress) + module.get_html() + module.get_progress.assert_called_once_with() + + def test_get_problem(self): + """ + Check that get_problem() returns the expected dictionary. + """ + module = CapaFactory.create() + self.assertEquals(module.get_problem("data"), {'html': module.get_problem_html(encapsulate=False)}) + class ComplexEncoderTest(unittest.TestCase): def test_default(self): diff --git a/common/lib/xmodule/xmodule/tests/test_combined_open_ended.py b/common/lib/xmodule/xmodule/tests/test_combined_open_ended.py index 8162d588bb..4fd0ddccf7 100644 --- a/common/lib/xmodule/xmodule/tests/test_combined_open_ended.py +++ b/common/lib/xmodule/xmodule/tests/test_combined_open_ended.py @@ -14,6 +14,7 @@ from xmodule.modulestore import Location from lxml import etree import capa.xqueue_interface as xqueue_interface from datetime import datetime +from pytz import UTC import logging log = logging.getLogger(__name__) @@ -212,7 +213,7 @@ class OpenEndedModuleTest(unittest.TestCase): 'submission_id': '1', 'grader_id': '1', 'score': 3} - qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat) + qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat) student_info = {'anonymous_student_id': self.test_system.anonymous_student_id, 'submission_time': qtime} contents = { @@ -233,7 +234,7 @@ class OpenEndedModuleTest(unittest.TestCase): def test_send_to_grader(self): submission = "This is a student submission" - qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat) + qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat) student_info = {'anonymous_student_id': self.test_system.anonymous_student_id, 'submission_time': qtime} contents = self.openendedmodule.payload.copy() @@ -632,6 +633,7 @@ class OpenEndedModuleXmlTest(unittest.TestCase, DummyModulestore): module.handle_ajax("reset", {}) self.assertEqual(module.state, "initial") + class OpenEndedModuleXmlAttemptTest(unittest.TestCase, DummyModulestore): """ Test if student is able to reset the problem diff --git a/common/lib/xmodule/xmodule/tests/test_progress.py b/common/lib/xmodule/xmodule/tests/test_progress.py index d1a32d5de5..e7af7a0c09 100644 --- a/common/lib/xmodule/xmodule/tests/test_progress.py +++ b/common/lib/xmodule/xmodule/tests/test_progress.py @@ -90,15 +90,15 @@ class ProgressTest(unittest.TestCase): self.assertEqual(Progress.to_js_status_str(self.not_started), "none") self.assertEqual(Progress.to_js_status_str(self.half_done), "in_progress") self.assertEqual(Progress.to_js_status_str(self.done), "done") - self.assertEqual(Progress.to_js_status_str(None), "NA") + self.assertEqual(Progress.to_js_status_str(None), "0") def test_to_js_detail_str(self): '''Test the Progress.to_js_detail_str() method''' f = Progress.to_js_detail_str for p in (self.not_started, self.half_done, self.done): self.assertEqual(f(p), str(p)) - # But None should be encoded as NA - self.assertEqual(f(None), "NA") + # But None should be encoded as 0 + self.assertEqual(f(None), "0") def test_add(self): '''Test the Progress.add_counts() method''' diff --git a/common/lib/xmodule/xmodule/tests/test_xblock_wrappers.py b/common/lib/xmodule/xmodule/tests/test_xblock_wrappers.py new file mode 100644 index 0000000000..d4df2e1054 --- /dev/null +++ b/common/lib/xmodule/xmodule/tests/test_xblock_wrappers.py @@ -0,0 +1,160 @@ +""" +Tests for the wrapping layer that provides the XBlock API using XModule/Descriptor +functionality +""" + +from nose.tools import assert_equal +from unittest.case import SkipTest +from mock import Mock + +from xmodule.annotatable_module import AnnotatableDescriptor +from xmodule.capa_module import CapaDescriptor +from xmodule.course_module import CourseDescriptor +from xmodule.combined_open_ended_module import CombinedOpenEndedDescriptor +from xmodule.discussion_module import DiscussionDescriptor +from xmodule.gst_module import GraphicalSliderToolDescriptor +from xmodule.html_module import HtmlDescriptor +from xmodule.peer_grading_module import PeerGradingDescriptor +from xmodule.poll_module import PollDescriptor +from xmodule.video_module import VideoDescriptor +from xmodule.word_cloud_module import WordCloudDescriptor +from xmodule.crowdsource_hinter import CrowdsourceHinterDescriptor +from xmodule.videoalpha_module import VideoAlphaDescriptor +from xmodule.seq_module import SequenceDescriptor +from xmodule.conditional_module import ConditionalDescriptor +from xmodule.randomize_module import RandomizeDescriptor +from xmodule.vertical_module import VerticalDescriptor +from xmodule.wrapper_module import WrapperDescriptor + +LEAF_XMODULES = ( + AnnotatableDescriptor, + CapaDescriptor, + CombinedOpenEndedDescriptor, + DiscussionDescriptor, + GraphicalSliderToolDescriptor, + HtmlDescriptor, + PeerGradingDescriptor, + PollDescriptor, + VideoDescriptor, + # This is being excluded because it has dependencies on django + #VideoAlphaDescriptor, + WordCloudDescriptor, +) + + +CONTAINER_XMODULES = ( + CrowdsourceHinterDescriptor, + CourseDescriptor, + SequenceDescriptor, + ConditionalDescriptor, + RandomizeDescriptor, + VerticalDescriptor, + WrapperDescriptor, + CourseDescriptor, +) + + +class TestXBlockWrapper(object): + + @property + def leaf_module_runtime(self): + runtime = Mock() + runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs)) + runtime.anonymous_student_id = 'anonymous_student_id' + runtime.open_ended_grading_interface = {} + runtime.seed = 5 + runtime.get = lambda x: getattr(runtime, x) + runtime.position = 2 + runtime.ajax_url = 'ajax_url' + runtime.xblock_model_data = lambda d: d._model_data + return runtime + + @property + def leaf_descriptor_runtime(self): + runtime = Mock() + runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs)) + return runtime + + def leaf_descriptor(self, descriptor_cls): + return descriptor_cls( + self.leaf_descriptor_runtime, + {'location': 'i4x://org/course/catagory/name'} + ) + + def leaf_module(self, descriptor_cls): + return self.leaf_descriptor(descriptor_cls).xmodule(self.leaf_module_runtime) + + def container_module_runtime(self, depth): + runtime = self.leaf_module_runtime + if depth == 0: + runtime.get_module.side_effect = lambda x: self.leaf_module(HtmlDescriptor) + else: + runtime.get_module.side_effect = lambda x: self.container_module(VerticalDescriptor, depth-1) + return runtime + + @property + def container_descriptor_runtime(self): + runtime = Mock() + runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs)) + return runtime + + def container_descriptor(self, descriptor_cls): + return descriptor_cls( + self.container_descriptor_runtime, + { + 'location': 'i4x://org/course/catagory/name', + 'children': range(3) + } + ) + + def container_module(self, descriptor_cls, depth): + return self.container_descriptor(descriptor_cls).xmodule(self.container_module_runtime(depth)) + +class TestStudentView(TestXBlockWrapper): + + # Test that for all of the leaf XModule Descriptors, + # the student_view wrapper returns the same thing in its content + # as get_html returns + def test_student_view_leaf_node(self): + for descriptor_cls in LEAF_XMODULES: + yield self.check_student_view_leaf_node, descriptor_cls + + # Check that when an xmodule is instantiated from descriptor_cls + # it generates the same thing from student_view that it does from get_html + def check_student_view_leaf_node(self, descriptor_cls): + xmodule = self.leaf_module(descriptor_cls) + assert_equal(xmodule.get_html(), xmodule.student_view(None).content) + + + # Test that for all container XModule Descriptors, + # their corresponding XModule renders the same thing using student_view + # as it does using get_html, under the following conditions: + # a) All of its descendents are xmodules + # b) Some of its descendents are xmodules and some are xblocks + # c) All of its descendents are xblocks + def test_student_view_container_node(self): + for descriptor_cls in CONTAINER_XMODULES: + yield self.check_student_view_container_node_xmodules_only, descriptor_cls + yield self.check_student_view_container_node_mixed, descriptor_cls + yield self.check_student_view_container_node_xblocks_only, descriptor_cls + + + # Check that when an xmodule is generated from descriptor_cls + # with only xmodule children, it generates the same html from student_view + # as it does using get_html + def check_student_view_container_node_xmodules_only(self, descriptor_cls): + xmodule = self.container_module(descriptor_cls, 2) + assert_equal(xmodule.get_html(), xmodule.student_view(None).content) + + # Check that when an xmodule is generated from descriptor_cls + # with mixed xmodule and xblock children, it generates the same html from student_view + # as it does using get_html + def check_student_view_container_node_mixed(self, descriptor_cls): + raise SkipTest("XBlock support in XDescriptor not yet fully implemented") + + # Check that when an xmodule is generated from descriptor_cls + # with only xblock children, it generates the same html from student_view + # as it does using get_html + def check_student_view_container_node_xblocks_only(self, descriptor_cls): + raise SkipTest("XBlock support in XModules not yet fully implemented") + diff --git a/common/lib/xmodule/xmodule/video_module.py b/common/lib/xmodule/xmodule/video_module.py index 5354297c2b..6c5640346e 100644 --- a/common/lib/xmodule/xmodule/video_module.py +++ b/common/lib/xmodule/xmodule/video_module.py @@ -27,11 +27,13 @@ class VideoFields(object): scope=Scope.settings, # it'd be nice to have a useful default but it screws up other things; so, # use display_name_with_default for those - default="Video Title" + default="Video" ) - data = String(help="XML data for the problem", + data = String( + help="XML data for the problem", default='', - scope=Scope.content) + scope=Scope.content + ) position = Integer(help="Current position in the video", scope=Scope.user_state, default=0) show_captions = Boolean(help="This controls whether or not captions are shown by default.", display_name="Show Captions", scope=Scope.settings, default=True) youtube_id_1_0 = String(help="This is the Youtube ID reference for the normal speed video.", display_name="Default Speed", scope=Scope.settings, default="OEoXaMPEzfM") @@ -125,7 +127,7 @@ class VideoDescriptor(VideoFields, url identifiers """ video = super(VideoDescriptor, cls).from_xml(xml_data, system, org, course) - _parse_video_xml(video, xml_data) + _parse_video_xml(video, video.data) return video def definition_to_xml(self, resource_fs): @@ -146,10 +148,6 @@ def _parse_video_xml(video, xml_data): display_name = xml.get('display_name') if display_name: video.display_name = display_name - elif video.url_name is not None: - # copies the logic of display_name_with_default in order that studio created videos will have an - # initial non guid name - video.display_name = video.url_name.replace('_', ' ') youtube = xml.get('youtube') if youtube: diff --git a/common/lib/xmodule/xmodule/x_module.py b/common/lib/xmodule/xmodule/x_module.py index aee8e26171..47642e9f90 100644 --- a/common/lib/xmodule/xmodule/x_module.py +++ b/common/lib/xmodule/xmodule/x_module.py @@ -8,9 +8,11 @@ from collections import namedtuple from pkg_resources import resource_listdir, resource_string, resource_isdir from xmodule.modulestore import inheritance, Location -from xmodule.modulestore.exceptions import ItemNotFoundError, InsufficientSpecificationError +from xmodule.modulestore.exceptions import ItemNotFoundError, InsufficientSpecificationError, InvalidLocationError from xblock.core import XBlock, Scope, String, Integer, Float, ModelType +from xblock.fragment import Fragment +from xmodule.modulestore.locator import BlockUsageLocator log = logging.getLogger(__name__) @@ -27,7 +29,13 @@ class LocationField(ModelType): """ Parse the json value as a Location """ - return Location(value) + try: + return Location(value) + except InvalidLocationError: + if isinstance(value, BlockUsageLocator): + return value + else: + return BlockUsageLocator(value) def to_json(self, value): """ @@ -166,6 +174,10 @@ class XModule(XModuleFields, HTMLSnippet, XBlock): self.url_name = self.location.name if not hasattr(self, 'category'): self.category = self.location.category + elif isinstance(self.location, BlockUsageLocator): + self.url_name = self.location.usage_id + if not hasattr(self, 'category'): + raise InsufficientSpecificationError() else: raise InsufficientSpecificationError() self._loaded_children = None @@ -191,6 +203,13 @@ class XModule(XModuleFields, HTMLSnippet, XBlock): ''' if self._loaded_children is None: child_descriptors = self.get_child_descriptors() + + # This deliberately uses system.get_module, rather than runtime.get_block, + # because we're looking at XModule children, rather than XModuleDescriptor children. + # That means it can use the deprecated XModule apis, rather than future XBlock apis + + # TODO: Once we're in a system where this returns a mix of XModuleDescriptors + # and XBlocks, we're likely to have to change this more children = [self.system.get_module(descriptor) for descriptor in child_descriptors] # get_module returns None if the current user doesn't have access # to the location. @@ -296,6 +315,19 @@ class XModule(XModuleFields, HTMLSnippet, XBlock): return "" + # ~~~~~~~~~~~~~~~ XBlock API Wrappers ~~~~~~~~~~~~~~~~ + def student_view(self, context): + """ + Return a fragment with the html from this XModule + + Doesn't yet add any of the javascript to the fragment, nor the css. + Also doesn't expect any javascript binding, yet. + + Makes no use of the context parameter + """ + return Fragment(self.get_html()) + + def policy_key(location): """ Get the key for a location in a policy file. (Since the policy file is @@ -436,8 +468,17 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock): self.url_name = self.location.name if not hasattr(self, 'category'): self.category = self.location.category + elif isinstance(self.location, BlockUsageLocator): + self.url_name = self.location.usage_id + if not hasattr(self, 'category'): + raise InsufficientSpecificationError() else: raise InsufficientSpecificationError() + # update_version is the version which last updated this xblock v prev being the penultimate updater + # leaving off original_version since it complicates creation w/o any obv value yet and is computable + # by following previous until None + # definition_locator is only used by mongostores which separate definitions from blocks + self.edited_by = self.edited_on = self.previous_version = self.update_version = self.definition_locator = None self._child_instances = None @property @@ -473,7 +514,7 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock): child = child_loc else: try: - child = self.system.load_item(child_loc) + child = self.runtime.get_block(child_loc) except ItemNotFoundError: log.exception('Unable to load item {loc}, skipping'.format(loc=child_loc)) continue @@ -514,22 +555,30 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock): # ================================= JSON PARSING =========================== @staticmethod - def load_from_json(json_data, system, default_class=None): + def load_from_json(json_data, system, default_class=None, parent_xblock=None): """ This method instantiates the correct subclass of XModuleDescriptor based - on the contents of json_data. + on the contents of json_data. It does not persist it and can create one which + has no usage id. - json_data must contain a 'location' element, and must be suitable to be - passed into the subclasses `from_json` method as model_data + parent_xblock is used to compute inherited metadata as well as to append the new xblock. + + json_data: + - 'location' : must have this field + - 'category': the xmodule category (required or location must be a Location) + - 'metadata': a dict of locally set metadata (not inherited) + - 'children': a list of children's usage_ids w/in this course + - 'definition': + - '_id' (optional): the usage_id of this. Will generate one if not given one. """ class_ = XModuleDescriptor.load_class( - json_data['location']['category'], + json_data.get('category', json_data.get('location', {}).get('category')), default_class ) - return class_.from_json(json_data, system) + return class_.from_json(json_data, system, parent_xblock) @classmethod - def from_json(cls, json_data, system): + def from_json(cls, json_data, system, parent_xblock=None): """ Creates an instance of this descriptor from the supplied json_data. This may be overridden by subclasses @@ -547,28 +596,25 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock): Otherwise, it contains the single field 'data' 4) Any value later in this list overrides a value earlier in this list - system: A DescriptorSystem for interacting with external resources + json_data: + - 'category': the xmodule category (required) + - 'metadata': a dict of locally set metadata (not inherited) + - 'children': a list of children's usage_ids w/in this course + - 'definition': + - '_id' (optional): the usage_id of this. Will generate one if not given one. """ - model_data = {} + usage_id = json_data.get('_id', None) + if not '_inherited_metadata' in json_data and parent_xblock is not None: + json_data['_inherited_metadata'] = parent_xblock.xblock_kvs.get_inherited_metadata().copy() + json_metadata = json_data.get('metadata', {}) + for field in inheritance.INHERITABLE_METADATA: + if field in json_metadata: + json_data['_inherited_metadata'][field] = json_metadata[field] - for key, value in json_data.get('metadata', {}).items(): - model_data[cls._translate(key)] = value - - model_data.update(json_data.get('metadata', {})) - - definition = json_data.get('definition', {}) - if 'children' in definition: - model_data['children'] = definition['children'] - - if 'data' in definition: - if isinstance(definition['data'], dict): - model_data.update(definition['data']) - else: - model_data['data'] = definition['data'] - - model_data['location'] = json_data['location'] - - return cls(system, model_data) + new_block = system.xblock_from_json(cls, usage_id, json_data) + if parent_xblock is not None: + parent_xblock.children.append(new_block) + return new_block @classmethod def _translate(cls, key): @@ -649,6 +695,8 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock): """ Use w/ caution. Really intended for use by the persistence layer. """ + # if caller wants kvs, caller's assuming it's up to date; so, decache it + self.save() return self._model_data._kvs # =============================== BUILTIN METHODS ========================== @@ -780,6 +828,10 @@ class DescriptorSystem(object): self.resources_fs = resources_fs self.error_tracker = error_tracker + def get_block(self, block_id): + """See documentation for `xblock.runtime:Runtime.get_block`""" + return self.load_item(block_id) + class XMLParsingSystem(DescriptorSystem): def __init__(self, load_item, resources_fs, error_tracker, process_xml, policy, **kwargs): @@ -868,8 +920,8 @@ class ModuleSystem(object): publish(event) - A function that allows XModules to publish events (such as grade changes) - xblock_model_data - A dict-like object containing the all data available to this - xblock + xblock_model_data - A function that constructs a model_data for an xblock from its + corresponding descriptor cache - A cache object with two methods: .get(key) returns an object from the cache or None. diff --git a/common/lib/xmodule/xmodule/xml_module.py b/common/lib/xmodule/xmodule/xml_module.py index 882e308c77..5b8d2c8aee 100644 --- a/common/lib/xmodule/xmodule/xml_module.py +++ b/common/lib/xmodule/xmodule/xml_module.py @@ -306,6 +306,7 @@ class XmlDescriptor(XModuleDescriptor): org and course are optional strings that will be used in the generated modules url identifiers """ + xml_object = etree.fromstring(xml_data) # VS[compat] -- just have the url_name lookup, once translation is done url_name = xml_object.get('url_name', xml_object.get('slug')) @@ -318,7 +319,8 @@ class XmlDescriptor(XModuleDescriptor): filepath = cls._format_filepath(xml_object.tag, name_to_pathname(url_name)) definition_xml = cls.load_file(filepath, system.resources_fs, location) else: - definition_xml = xml_object # this is just a pointer, not the real definition content + definition_xml = xml_object + filepath = None definition, children = cls.load_definition(definition_xml, system, location) # note this removes metadata diff --git a/common/test/data/splitmongo_json/active_versions.json b/common/test/data/splitmongo_json/active_versions.json new file mode 100644 index 0000000000..b41440e0e7 --- /dev/null +++ b/common/test/data/splitmongo_json/active_versions.json @@ -0,0 +1,27 @@ +[{"_id" : "GreekHero", + "org" : "testx", + "prettyid" : "test_course", + "versions" : { + "draft" : { "$oid" : "1d00000000000000dddd0000" } + }, + "edited_on" : {"$date" : 1364481713238}, + "edited_by" : "test@edx.org"}, + + {"_id" : "wonderful", + "org" : "testx", + "prettyid" : "another_course", + "versions" : { + "draft" : { "$oid" : "1d00000000000000dddd2222" }, + "published" : { "$oid" : "1d00000000000000eeee0000" } + }, + "edited_on" : {"$date" : 1364481313238}, + "edited_by" : "test@edx.org"}, + + {"_id" : "contender", + "org" : "guestx", + "prettyid" : "test_course", + "versions" : { + "draft" : { "$oid" : "1d00000000000000dddd5555" }}, + "edited_on" : {"$date" : 1364491313238}, + "edited_by" : "test@guestx.edu"} +] diff --git a/common/test/data/splitmongo_json/definitions.json b/common/test/data/splitmongo_json/definitions.json new file mode 100644 index 0000000000..0ed42112aa --- /dev/null +++ b/common/test/data/splitmongo_json/definitions.json @@ -0,0 +1,334 @@ +[ + { + "_id":"head12345_12", + "category":"course", + "data":{ + "textbooks":[ + + ], + "grading_policy":{ + "GRADER":[ + { + "min_count":4, + "weight":0.15, + "type":"Homework", + "drop_count":2, + "short_label":"HWa" + }, + { + "short_label":"", + "min_count":12, + "type":"Lab", + "drop_count":2, + "weight":0.15 + }, + { + "short_label":"Midterm", + "min_count":1, + "type":"Midterm Exam", + "drop_count":0, + "weight":0.3 + }, + { + "short_label":"Final", + "min_count":1, + "type":"Final Exam", + "drop_count":0, + "weight":0.4 + } + ], + "GRADE_CUTOFFS":{ + "Pass":0.45 + } + }, + "wiki_slug":null + }, + "edited_by":"testassist@edx.org", + "edited_on":{"$date" : 1364481713238}, + "previous_version":"head12345_11", + "original_version":"head12345_10" + }, + { + "_id":"head12345_11", + "category":"course", + "data":{ + "textbooks":[ + + ], + "grading_policy":{ + "GRADER":[ + { + "min_count":5, + "weight":0.15, + "type":"Homework", + "drop_count":1, + "short_label":"HWa" + }, + { + "short_label":"", + "min_count":12, + "type":"Lab", + "drop_count":2, + "weight":0.15 + }, + { + "short_label":"Midterm", + "min_count":1, + "type":"Midterm Exam", + "drop_count":0, + "weight":0.3 + }, + { + "short_label":"Final", + "min_count":1, + "type":"Final Exam", + "drop_count":0, + "weight":0.4 + } + ], + "GRADE_CUTOFFS":{ + "Pass":0.55 + } + }, + "wiki_slug":null + }, + "edited_by":"testassist@edx.org", + "edited_on":{"$date" : 1364481713238}, + "previous_version":"head12345_10", + "original_version":"head12345_10" + }, + { + "_id":"head12345_10", + "category":"course", + "data":{ + "textbooks":[ + + ], + "grading_policy":{ + "GRADER":[ + { + "min_count":5, + "weight":0.15, + "type":"Homework", + "drop_count":1, + "short_label":"HWa" + }, + { + "short_label":"", + "min_count":2, + "type":"Lab", + "drop_count":0, + "weight":0.15 + }, + { + "short_label":"Midterm", + "min_count":1, + "type":"Midterm Exam", + "drop_count":0, + "weight":0.3 + }, + { + "short_label":"Final", + "min_count":1, + "type":"Final Exam", + "drop_count":0, + "weight":0.4 + } + ], + "GRADE_CUTOFFS":{ + "Pass":0.75 + } + }, + "wiki_slug":null + }, + "edited_by":"test@edx.org", + "edited_on":{"$date": 1364473713238}, + "previous_version":null, + "original_version":"head12345_10" + }, + { + "_id":"head23456_1", + "category":"course", + "data":{ + "textbooks":[ + + ], + "grading_policy":{ + "GRADER":[ + { + "min_count":14, + "weight":0.25, + "type":"Homework", + "drop_count":1, + "short_label":"HWa" + }, + { + "short_label":"", + "min_count":12, + "type":"Lab", + "drop_count":2, + "weight":0.25 + }, + { + "short_label":"Midterm", + "min_count":1, + "type":"Midterm Exam", + "drop_count":0, + "weight":0.2 + }, + { + "short_label":"Final", + "min_count":1, + "type":"Final Exam", + "drop_count":0, + "weight":0.3 + } + ], + "GRADE_CUTOFFS":{ + "Pass":0.45 + } + }, + "wiki_slug":null + }, + "edited_by":"test@edx.org", + "edited_on":{"$date": 1364481313238}, + "previous_version":"head23456_0", + "original_version":"head23456_0" + }, + { + "_id":"head23456_0", + "category":"course", + "data":{ + "textbooks":[ + + ], + "grading_policy":{ + "GRADER":[ + { + "min_count":14, + "weight":0.25, + "type":"Homework", + "drop_count":1, + "short_label":"HWa" + }, + { + "short_label":"", + "min_count":12, + "type":"Lab", + "drop_count":2, + "weight":0.25 + }, + { + "short_label":"Midterm", + "min_count":1, + "type":"Midterm Exam", + "drop_count":0, + "weight":0.2 + }, + { + "short_label":"Final", + "min_count":1, + "type":"Final Exam", + "drop_count":0, + "weight":0.3 + } + ], + "GRADE_CUTOFFS":{ + "Pass":0.95 + } + }, + "wiki_slug":null + }, + "edited_by":"test@edx.org", + "edited_on":{"$date" : 1364481313238}, + "previous_version":null, + "original_version":"head23456_0" + }, + { + "_id":"head345679_1", + "category":"course", + "data":{ + "textbooks":[ + + ], + "grading_policy":{ + "GRADER":[ + { + "min_count":4, + "weight":0.25, + "type":"Homework", + "drop_count":0, + "short_label":"HW" + }, + { + "short_label":"Midterm", + "min_count":1, + "type":"Midterm Exam", + "drop_count":0, + "weight":0.4 + }, + { + "short_label":"Final", + "min_count":1, + "type":"Final Exam", + "drop_count":0, + "weight":0.35 + } + ], + "GRADE_CUTOFFS":{ + "Pass":0.25 + } + }, + "wiki_slug":null + }, + "edited_by":"test@edx.org", + "edited_on":{"$date" : 1364481313238}, + "previous_version":null, + "original_version":"head23456_0" + }, + { + "_id":"chapter12345_1", + "category":"chapter", + "data":null, + "edited_by":"testassist@edx.org", + "edited_on":{"$date" : 1364483713238}, + "previous_version":null, + "original_version":"chapter12345_1" + }, + { + "_id":"chapter12345_2", + "category":"chapter", + "data":null, + "edited_by":"testassist@edx.org", + "edited_on":{"$date" : 1364483713238}, + "previous_version":null, + "original_version":"chapter12345_2" + }, + { + "_id":"chapter12345_3", + "category":"chapter", + "data":null, + "edited_by":"testassist@edx.org", + "edited_on":{"$date" : 1364483713238}, + "previous_version":null, + "original_version":"chapter12345_3" + }, + { + "_id":"problem12345_3_1", + "category":"problem", + "data":"", + "edited_by":"testassist@edx.org", + "edited_on":{"$date" : 1364483713238}, + "previous_version":null, + "original_version":"problem12345_3_1" + }, + { + "_id":"problem12345_3_2", + "category":"problem", + "data":"", + "edited_by":"testassist@edx.org", + "edited_on":{"$date" : 1364483713238}, + "previous_version":null, + "original_version":"problem12345_3_2" + } +] \ No newline at end of file diff --git a/common/test/data/splitmongo_json/structures.json b/common/test/data/splitmongo_json/structures.json new file mode 100644 index 0000000000..0021225213 --- /dev/null +++ b/common/test/data/splitmongo_json/structures.json @@ -0,0 +1,471 @@ +[ + { + "_id": { "$oid" : "1d00000000000000dddd0000"}, + "root":"head12345", + "original_version":{ "$oid" : "1d00000000000000dddd3333" }, + "previous_version":{ "$oid" : "1d00000000000000dddd1111" }, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364483713238 + }, + "blocks":{ + "head12345":{ + "children":[ + "chapter1", + "chapter2", + "chapter3" + ], + "category":"course", + "definition":"head12345_12", + "metadata":{ + "end":"2013-06-13T04:30", + "tabs":[ + { + "type":"courseware" + }, + { + "type":"course_info", + "name":"Course Info" + }, + { + "type":"discussion", + "name":"Discussion" + }, + { + "type":"wiki", + "name":"Wiki" + }, + { + "type":"static_tab", + "name":"Syllabus", + "url_slug":"01356a17b5924b17a04b7fc2426a3798" + }, + { + "type":"static_tab", + "name":"Advice for Students", + "url_slug":"57e9991c0d794ff58f7defae3e042e39" + } + ], + "enrollment_start":"2013-01-01T05:00", + "graceperiod":"2 hours 0 minutes 0 seconds", + "start":"2013-02-14T05:00", + "enrollment_end":"2013-03-02T05:00", + "data_dir":"MITx-2-Base", + "advertised_start":"Fall 2013", + "display_name":"The Ancient Greek Hero" + }, + "update_version":{ "$oid" : "1d00000000000000dddd0000" }, + "previous_version":{ "$oid" : "1d00000000000000dddd1111" }, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364483713238 + } + }, + "chapter1":{ + "children":[ + + ], + "category":"chapter", + "definition":"chapter12345_1", + "metadata":{ + "display_name":"Hercules" + }, + "update_version":{ "$oid" : "1d00000000000000dddd0000" }, + "previous_version":null, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364483713238 + } + }, + "chapter2":{ + "children":[ + + ], + "category":"chapter", + "definition":"chapter12345_2", + "metadata":{ + "display_name":"Hera heckles Hercules" + }, + "update_version":{ "$oid" : "1d00000000000000dddd0000" }, + "previous_version":null, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364483713238 + } + }, + "chapter3":{ + "children":[ + "problem1", + "problem3_2" + ], + "category":"chapter", + "definition":"chapter12345_3", + "metadata":{ + "display_name":"Hera cuckolds Zeus" + }, + "update_version":{ "$oid" : "1d00000000000000dddd0000" }, + "previous_version":null, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364483713238 + } + }, + "problem1":{ + "children":[ + + ], + "category":"problem", + "definition":"problem12345_3_1", + "metadata":{ + "display_name":"Problem 3.1", + "graceperiod":"4 hours 0 minutes 0 seconds" + }, + "update_version":{ "$oid" : "1d00000000000000dddd0000" }, + "previous_version":null, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364483713238 + } + }, + "problem3_2":{ + "children":[ + + ], + "category":"problem", + "definition":"problem12345_3_2", + "metadata":{ + "display_name":"Problem 3.2" + }, + "update_version":{ "$oid" : "1d00000000000000dddd0000" }, + "previous_version":null, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364483713238 + } + } + } + }, + { + "_id": { "$oid" : "1d00000000000000dddd1111"}, + "root":"head12345", + "original_version":{ "$oid" : "1d00000000000000dddd3333" }, + "previous_version":{ "$oid" : "1d00000000000000dddd3333" }, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364481713238 + }, + "blocks":{ + "head12345":{ + "children":[ + + ], + "category":"course", + "definition":"head12345_11", + "metadata":{ + "end":"2013-04-13T04:30", + "tabs":[ + { + "type":"courseware" + }, + { + "type":"course_info", + "name":"Course Info" + }, + { + "type":"discussion", + "name":"Discussion" + }, + { + "type":"wiki", + "name":"Wiki" + }, + { + "type":"static_tab", + "name":"Syllabus", + "url_slug":"01356a17b5924b17a04b7fc2426a3798" + }, + { + "type":"static_tab", + "name":"Advice for Students", + "url_slug":"57e9991c0d794ff58f7defae3e042e39" + } + ], + "enrollment_start":null, + "graceperiod":"2 hours 0 minutes 0 seconds", + "start":"2013-02-14T05:00", + "enrollment_end":null, + "data_dir":"MITx-2-Base", + "advertised_start":null, + "display_name":"The Ancient Greek Hero" + }, + "update_version":{ "$oid" : "1d00000000000000dddd1111" }, + "previous_version":{ "$oid" : "1d00000000000000dddd3333" }, + "edited_by":"testassist@edx.org", + "edited_on":{ + "$date":1364481713238 + } + } + } + }, + { + "_id": { "$oid" : "1d00000000000000dddd3333"}, + "root":"head12345", + "original_version":{ "$oid" : "1d00000000000000dddd3333" }, + "previous_version":null, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364473713238 + }, + "blocks":{ + "head12345":{ + "children":[ + + ], + "category":"course", + "definition":"head12345_10", + "metadata":{ + "end":null, + "tabs":[ + { + "type":"courseware" + }, + { + "type":"course_info", + "name":"Course Info" + }, + { + "type":"discussion", + "name":"Discussion" + }, + { + "type":"wiki", + "name":"Wiki" + } + ], + "enrollment_start":null, + "graceperiod":null, + "start":"2013-02-14T05:00", + "enrollment_end":null, + "data_dir":"MITx-2-Base", + "advertised_start":null, + "display_name":"The Ancient Greek Hero" + }, + "update_version":{ "$oid" : "1d00000000000000dddd3333" }, + "previous_version":null, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364473713238 + } + } + } + }, + { + "_id": { "$oid" : "1d00000000000000dddd2222"}, + "root":"head23456", + "original_version":{ "$oid" : "1d00000000000000dddd4444" }, + "previous_version":{ "$oid" : "1d00000000000000dddd4444" }, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364481313238 + }, + "blocks":{ + "head23456":{ + "children":[ + + ], + "category":"course", + "definition":"head23456_1", + "metadata":{ + "end":null, + "tabs":[ + { + "type":"courseware" + }, + { + "type":"course_info", + "name":"Course Info" + }, + { + "type":"discussion", + "name":"Discussion" + }, + { + "type":"wiki", + "name":"Wiki" + } + ], + "enrollment_start":null, + "graceperiod":null, + "start":"2013-02-14T05:00", + "enrollment_end":null, + "data_dir":"MITx-2-Base", + "advertised_start":null, + "display_name":"The most wonderful course" + }, + "update_version":{ "$oid" : "1d00000000000000dddd2222" }, + "previous_version":{ "$oid" : "1d00000000000000dddd4444" }, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364481313238 + } + + } + } + }, + { + "_id": { "$oid" : "1d00000000000000dddd4444"}, + "root":"head23456", + "original_version":{ "$oid" : "1d00000000000000dddd4444" }, + "previous_version":null, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364480313238 + }, + "blocks":{ + "head23456":{ + "children":[ + + ], + "category":"course", + "definition":"head23456_0", + "metadata":{ + "end":null, + "tabs":[ + { + "type":"courseware" + }, + { + "type":"course_info", + "name":"Course Info" + }, + { + "type":"discussion", + "name":"Discussion" + }, + { + "type":"wiki", + "name":"Wiki" + } + ], + "enrollment_start":null, + "graceperiod":null, + "start":"2013-02-14T05:00", + "enrollment_end":null, + "data_dir":"MITx-2-Base", + "advertised_start":null, + "display_name":"A wonderful course" + }, + "update_version":{ "$oid" : "1d00000000000000dddd4444" }, + "previous_version":null, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364480313238 + } + } + } + }, + { + "_id": { "$oid" : "1d00000000000000eeee0000"}, + "root":"head23456", + "original_version":{ "$oid" : "1d00000000000000eeee0000" }, + "previous_version":null, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364481333238 + }, + "blocks":{ + "head23456":{ + "children":[ + + ], + "category":"course", + "definition":"head23456_1", + "metadata":{ + "end":null, + "tabs":[ + { + "type":"courseware" + }, + { + "type":"course_info", + "name":"Course Info" + }, + { + "type":"discussion", + "name":"Discussion" + }, + { + "type":"wiki", + "name":"Wiki" + } + ], + "enrollment_start":null, + "graceperiod":null, + "start":"2013-02-14T05:00", + "enrollment_end":null, + "data_dir":"MITx-2-Base", + "advertised_start":null, + "display_name":"The most wonderful course" + }, + "update_version":{ "$oid" : "1d00000000000000eeee0000" }, + "previous_version":null, + "edited_by":"test@edx.org", + "edited_on":{ + "$date":1364481333238 + } + } + } + }, + { + "_id": { "$oid" : "1d00000000000000dddd5555"}, + "root":"head345679", + "original_version":{ "$oid" : "1d00000000000000dddd5555" }, + "previous_version":null, + "edited_by":"test@guestx.edu", + "edited_on":{ + "$date":1364491313238 + }, + "blocks":{ + "head345679":{ + "children":[ + + ], + "category":"course", + "definition":"head345679_1", + "metadata":{ + "end":null, + "tabs":[ + { + "type":"courseware" + }, + { + "type":"course_info", + "name":"Course Info" + }, + { + "type":"discussion", + "name":"Discussion" + }, + { + "type":"wiki", + "name":"Wiki" + } + ], + "enrollment_start":null, + "graceperiod":null, + "start":"2013-03-14T05:00", + "enrollment_end":null, + "data_dir":"MITx-3-Base", + "advertised_start":null, + "display_name":"Yet another contender" + }, + "update_version":{ "$oid" : "1d00000000000000dddd5555" }, + "previous_version":null, + "edited_by":"test@guestx.edu", + "edited_on":{ + "$date":1364491313238 + } + } + } + } +] diff --git a/common/test/data/toy/chapter/secret/magic.xml b/common/test/data/toy/chapter/secret/magic.xml index 62756b8896..f85d2e75da 100644 --- a/common/test/data/toy/chapter/secret/magic.xml +++ b/common/test/data/toy/chapter/secret/magic.xml @@ -1,3 +1,3 @@ - diff --git a/common/test/data/toy/course/2012_Fall.xml b/common/test/data/toy/course/2012_Fall.xml index c3500040c2..679f7bbfdb 100644 --- a/common/test/data/toy/course/2012_Fall.xml +++ b/common/test/data/toy/course/2012_Fall.xml @@ -3,10 +3,11 @@ - - diff --git a/common/test/data/toy/html/toyjumpto.html b/common/test/data/toy/html/toyjumpto.html new file mode 100644 index 0000000000..df2a5a42aa --- /dev/null +++ b/common/test/data/toy/html/toyjumpto.html @@ -0,0 +1 @@ +      This is a link to another page and some Chinese 四節比分和七年前

      Some more Chinese 四節比分和七年前

      diff --git a/common/test/data/toy/html/toyjumpto.xml b/common/test/data/toy/html/toyjumpto.xml new file mode 100644 index 0000000000..af9cdf4f74 --- /dev/null +++ b/common/test/data/toy/html/toyjumpto.xml @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/common/test/data/toy/vertical/vertical_test.xml b/common/test/data/toy/vertical/vertical_test.xml index e801a4ac86..68c5745f37 100644 --- a/common/test/data/toy/vertical/vertical_test.xml +++ b/common/test/data/toy/vertical/vertical_test.xml @@ -1,4 +1,6 @@ +
+% if show_chat: +
+
+ Open Chat + Close Chat +
+
+ ## The Candy.js plugin wants to render in an element with #candy +
+
+
+% endif % if course.show_calculator:
diff --git a/lms/templates/courseware/submission_history.html b/lms/templates/courseware/submission_history.html index 683c61c5a0..b4ce5ed2a6 100644 --- a/lms/templates/courseware/submission_history.html +++ b/lms/templates/courseware/submission_history.html @@ -1,5 +1,5 @@ <% import json %> -

${username} > ${course_id} > ${location}

+

${username | h} > ${course_id | h} > ${location | h}

% for i, entry in enumerate(history_entries):
diff --git a/lms/templates/problem.html b/lms/templates/problem.html index fc49ab7ce7..f4f8e78b66 100644 --- a/lms/templates/problem.html +++ b/lms/templates/problem.html @@ -1,11 +1,11 @@ <%namespace name='static' file='static_content.html'/>

${ problem['name'] } - % if problem['weight'] != 1 and problem['weight'] is not None: - : ${ problem['weight'] } points - % endif

+
+
+
${ problem['html'] } diff --git a/lms/templates/problem_ajax.html b/lms/templates/problem_ajax.html index 42cd18c4e3..1babda1ae2 100644 --- a/lms/templates/problem_ajax.html +++ b/lms/templates/problem_ajax.html @@ -1 +1 @@ -
+
diff --git a/lms/templates/seq_module.html b/lms/templates/seq_module.html index f372d495ca..fff1279cd6 100644 --- a/lms/templates/seq_module.html +++ b/lms/templates/seq_module.html @@ -3,7 +3,7 @@ - +
    % for idx, item in enumerate(items): @@ -16,7 +16,7 @@ data-id="${item['id']}" data-element="${idx+1}" href="javascript:void(0);"> -

    ${item['title']}, ${item['type']}

    +

    ${item['title']}, ${item['type']}

    % endfor diff --git a/lms/templates/unsubscribe.html b/lms/templates/unsubscribe.html new file mode 100644 index 0000000000..6f8c042d76 --- /dev/null +++ b/lms/templates/unsubscribe.html @@ -0,0 +1,17 @@ +<%! from django.core.urlresolvers import reverse %> +<%inherit file="main.html" /> + +<%namespace name='static' file='static_content.html'/> + +
    + +
    +

    Unsubscribe Successful!

    +
    + +

    + You will no longer receive notification emails from edX. + Click here to return to your dashboard. +

    +
    +
    diff --git a/lms/urls.py b/lms/urls.py index 085a35b9f4..9670a67a4a 100644 --- a/lms/urls.py +++ b/lms/urls.py @@ -59,6 +59,8 @@ urlpatterns = ('', # nopep8 name='auth_password_reset_done'), url(r'^heartbeat$', include('heartbeat.urls')), + + url(r'^user_api/', include('user_api.urls')), ) # University profiles only make sense in the default edX context @@ -175,6 +177,8 @@ if settings.COURSEWARE_ENABLED: urlpatterns += ( url(r'^courses/(?P[^/]+/[^/]+/[^/]+)/jump_to/(?P.*)$', 'courseware.views.jump_to', name="jump_to"), + url(r'^courses/(?P[^/]+/[^/]+/[^/]+)/jump_to_id/(?P.*)$', + 'courseware.views.jump_to_id', name="jump_to_id"), url(r'^courses/(?P[^/]+/[^/]+/[^/]+)/modx/(?P.*?)/(?P[^/]*)$', 'courseware.module_render.modx_dispatch', name='modx_dispatch'), @@ -329,7 +333,10 @@ if settings.COURSEWARE_ENABLED: url(r'^courses/(?P[^/]+/[^/]+/[^/]+)/news$', 'courseware.views.news', name="news"), url(r'^courses/(?P[^/]+/[^/]+/[^/]+)/discussion/', - include('django_comment_client.urls')) + include('django_comment_client.urls')), + url(r'^notification_prefs/enable/', 'notification_prefs.views.ajax_enable'), + url(r'^notification_prefs/disable/', 'notification_prefs.views.ajax_disable'), + url(r'^notification_prefs/unsubscribe/(?P[a-zA-Z0-9-_=]+)/', 'notification_prefs.views.unsubscribe'), ) urlpatterns += ( # This MUST be the last view in the courseware--it's a catch-all for custom tabs. diff --git a/manage.py b/manage.py new file mode 100755 index 0000000000..948bf147d4 --- /dev/null +++ b/manage.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python +""" +Usage: manage.py {lms|cms} [--settings env] ... + +Run django management commands. Because edx-platform contains multiple django projects, +the first argument specifies which project to run (cms [Studio] or lms [Learning Management System]). + +By default, those systems run in with a settings file appropriate for development. However, +by passing the --settings flag, you can specify what environment specific settings file to use. + +Any arguments not understood by this manage.py will be passed to django-admin.py +""" + +import os +import sys +import glob2 +import imp +from argparse import ArgumentParser + +def parse_args(): + """Parse edx specific arguments to manage.py""" + parser = ArgumentParser() + subparsers = parser.add_subparsers(title='system', description='edx service to run') + + lms = subparsers.add_parser( + 'lms', + help='Learning Management System', + add_help=False, + usage='%(prog)s [options] ...' + ) + lms.add_argument('-h', '--help', action='store_true', help='show this help message and exit') + lms.add_argument( + '--settings', + help="Which django settings module to use from inside of lms.envs. If not provided, the DJANGO_SETTINGS_MODULE " + "environment variable will be used if it is set, otherwise will default to lms.envs.dev") + lms.add_argument( + '-s', '--service-variant', + choices=['lms', 'lms-xml', 'lms-preview'], + default='lms', + help='Which service variant to run, when using the aws environment') + lms.set_defaults( + help_string=lms.format_help(), + settings_base='lms/envs', + default_settings='lms.envs.dev' + ) + + cms = subparsers.add_parser( + 'cms', + help='Studio', + add_help=False, + usage='%(prog)s [options] ...' + ) + cms.add_argument( + '--settings', + help="Which django settings module to use from inside cms.envs. If not provided, the DJANGO_SETTINGS_MODULE " + "environment variable will be used if it is set, otherwise will default to cms.envs.dev") + cms.add_argument('-h', '--help', action='store_true', help='show this help message and exit') + cms.set_defaults( + help_string=cms.format_help(), + settings_base='cms/envs', + default_settings='cms.envs.dev', + service_variant='cms' + ) + + + edx_args, django_args = parser.parse_known_args() + + if edx_args.help: + print "edX:" + print edx_args.help_string + + return edx_args, django_args + + +if __name__ == "__main__": + edx_args, django_args = parse_args() + + if edx_args.settings: + os.environ["DJANGO_SETTINGS_MODULE"] = edx_args.settings_base.replace('/', '.') + "." + edx_args.settings + else: + os.environ.setdefault("DJANGO_SETTINGS_MODULE", edx_args.default_settings) + os.environ.setdefault("SERVICE_VARIANT", edx_args.service_variant) + if edx_args.help: + print "Django:" + # This will trigger django-admin.py to print out its help + django_args.insert(0, '--help') + + from django.core.management import execute_from_command_line + + execute_from_command_line([sys.argv[0]] + django_args) diff --git a/rakelib/django.rake b/rakelib/django.rake index 53c299ac9f..40005e19e3 100644 --- a/rakelib/django.rake +++ b/rakelib/django.rake @@ -11,8 +11,7 @@ end task :fastlms do # this is >2 times faster that rake [lms], and does not need web, good for local dev - django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin') - sh("#{django_admin} runserver --traceback --settings=lms.envs.dev --pythonpath=.") + sh("./manage.py lms runserver --traceback") end # Start :system locally with the specified :env and :options. @@ -36,9 +35,8 @@ end desc "Start #{system} Celery worker" task "#{system}_worker", [:options] => [:predjango] do |t, args| args.with_defaults(:options => default_options[system]) - django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin') command = 'celery worker' - sh("#{django_admin} #{command} --loglevel=INFO --settings=#{system}.envs.dev_with_worker --pythonpath=. #{args.join(' ')}") + sh("./manage.py #{system} --settings dev_with_worker #{command} --loglevel=INFO #{args.join(' ')}") end # Per environment tasks diff --git a/rakelib/helpers.rb b/rakelib/helpers.rb index 925b7e1b28..6826570a18 100644 --- a/rakelib/helpers.rb +++ b/rakelib/helpers.rb @@ -12,8 +12,7 @@ def select_executable(*cmds) end def django_admin(system, env, command, *args) - django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin') - return "#{django_admin} #{command} --traceback --settings=#{system}.envs.#{env} --pythonpath=. #{args.join(' ')}" + return "./manage.py #{system} --settings #{env} #{command} --traceback #{args.join(' ')}" end def report_dir_path(dir) diff --git a/rakelib/quality.rake b/rakelib/quality.rake index b9254528cf..7cbe10ce1f 100644 --- a/rakelib/quality.rake +++ b/rakelib/quality.rake @@ -43,3 +43,17 @@ end end task :pep8 => :"pep8:#{system}" end + +dquality_dir = File.join(REPORT_DIR, "diff_quality") +directory dquality_dir + +desc "Build the html diff quality reports, and print the reports to the console." +task :quality => dquality_dir do + # Generage diff-quality html report for pep8, and print to console + sh("diff-quality --violations=pep8 --html-report #{dquality_dir}/diff_quality_pep8.html") + sh("diff-quality --violations=pep8") + + # Generage diff-quality html report for pylint, and print to console + sh("diff-quality --violations=pylint --html-report #{dquality_dir}/diff_quality_pylint.html") + sh("diff-quality --violations=pylint") +end \ No newline at end of file diff --git a/rakelib/tests.rake b/rakelib/tests.rake index 7e3e672f39..57861902bc 100644 --- a/rakelib/tests.rake +++ b/rakelib/tests.rake @@ -161,4 +161,4 @@ task :coverage => :report_dirs do if not found_coverage_info puts "No coverage info found. Run `rake test` before running `rake coverage`." end -end +end \ No newline at end of file diff --git a/requirements/edx/base.txt b/requirements/edx/base.txt index 910b6f3def..dd9d8b5a3e 100644 --- a/requirements/edx/base.txt +++ b/requirements/edx/base.txt @@ -7,6 +7,7 @@ celery==3.0.19 distribute>=0.6.28 django-celery==3.0.17 django-countries==1.5 +django-filter==0.6.0 django-followit==0.0.3 django-keyedcache==1.4-6 django-kombu==0.9.4 @@ -20,6 +21,7 @@ django-ses==0.4.1 django-storages==1.1.5 django-threaded-multihost==1.4-1 django-method-override==0.1.0 +djangorestframework==2.3.5 django==1.4.5 feedparser==5.1.3 fs==0.4.0 @@ -35,6 +37,7 @@ path.py==3.0.1 Pillow==1.7.8 pip>=1.3 polib==1.0.3 +pycrypto>=2.6 pygments==1.5 pygraphviz==1.1 pymongo==2.4.1 diff --git a/requirements/edx/github.txt b/requirements/edx/github.txt index 64231bc6b7..79e3af89a1 100644 --- a/requirements/edx/github.txt +++ b/requirements/edx/github.txt @@ -3,11 +3,11 @@ # Third-party: -e git://github.com/edx/django-staticfiles.git@6d2504e5c8#egg=django-staticfiles -e git://github.com/edx/django-pipeline.git#egg=django-pipeline --e git://github.com/edx/django-wiki.git@ac906abe#egg=django-wiki +-e git://github.com/edx/django-wiki.git@41815e2ef1b0323f92900f8e60711b0f0c37766b#egg=django-wiki -e git://github.com/dementrock/pystache_custom.git@776973740bdaad83a3b029f96e415a7d1e8bec2f#egg=pystache_custom-dev -e git://github.com/eventbrite/zendesk.git@d53fe0e81b623f084e91776bcf6369f8b7b63879#egg=zendesk # Our libraries: --e git+https://github.com/edx/XBlock.git@3974e999fe853a37dfa6fadf0611289434349409#egg=XBlock +-e git+https://github.com/edx/XBlock.git@b697bebd45deebd0f868613fab6722a0460ca0c1#egg=XBlock -e git+https://github.com/edx/codejail.git@c08967fb44d1bcdb259d3ec58812e3ac592539c2#egg=codejail --e git+https://github.com/edx/diff-cover.git@v0.1.3#egg=diff_cover +-e git+https://github.com/edx/diff-cover.git@v0.2.0#egg=diff_cover diff --git a/scripts/create-dev-env.sh b/scripts/create-dev-env.sh index 8699cc7656..25e1b7520b 100755 --- a/scripts/create-dev-env.sh +++ b/scripts/create-dev-env.sh @@ -495,8 +495,8 @@ mkdir -p "$BASE/log" mkdir -p "$BASE/db" mkdir -p "$BASE/data" -rake django-admin[syncdb,lms,dev,--noinput] -rake django-admin[migrate] +./manage.py lms syncdb --noinput --migrate +./manage.py cms syncdb --noinput --migrate # Configure Git @@ -520,18 +520,13 @@ if [[ ! $quiet ]]; then $ workon mitx - To initialize Django - - $ rake django-admin[syncdb] - $ rake django-admin[migrate] - To start the Django on port 8000 $ rake lms Or to start Django on a different - $ rake django-admin[runserver,lms,dev,] + $ ./manage.py lms runserver If the Django development server starts properly you should see: diff --git a/scripts/release-email-list.sh b/scripts/release-email-list.sh index 64fa7c00d1..f54018f4f5 100755 --- a/scripts/release-email-list.sh +++ b/scripts/release-email-list.sh @@ -4,27 +4,36 @@ LOG_CMD="git --no-pager log $1..$2" RESPONSIBLE=$(sort -u <($LOG_CMD --format='tformat:%ae' && $LOG_CMD --format='tformat:%ce')) +echo "~~~~ Email ~~~~~" + echo -n 'To: ' echo ${RESPONSIBLE} | sed "s/ /, /g" echo echo "You've made changes that are about to be released. All of the commits that you either authored or committed are listed below. Please verify them on -\$ENVIRONMENT" +\$ENVIRONMENT. + +Please record your notes on https://edx-wiki.atlassian.net/wiki/display/ENG/Release+Page%3A+\$DATE +and add any bugs found to the Release Candidate Bugs section" echo +echo "~~~~~ Wiki Table ~~~~~" +echo "Type Ctrl+Shift+D on Confluence to embed the following table in your release wiki page" +echo + +echo '||Author||Changes||Commit Link||Testing Notes||' + for EMAIL in $RESPONSIBLE; do AUTHORED_BY="$LOG_CMD --author=<${EMAIL}>" COMMITTED_BY="$LOG_CMD --committer=<${EMAIL}>" COMMITTED_NOT_AUTHORED="$COMMITTED_BY $($AUTHORED_BY --format='tformat:^%h')" - echo $EMAIL "authored the following commits:" - $AUTHORED_BY --format='tformat: %s - https://github.com/edx/edx-platform/commit/%h' - echo + $AUTHORED_BY --format="tformat:|$EMAIL|%s|[commit|https://github.com/edx/edx-platform/commit/%h]| |" | head -n 1 + $AUTHORED_BY --format="tformat:| |%s|[commit|https://github.com/edx/edx-platform/commit/%h]| |" | tail -n +2 if [[ $($COMMITTED_NOT_AUTHORED) != "" ]]; then - echo $EMAIL "committed but didn't author the following commits:" - $COMMITTED_NOT_AUTHORED --format='tformat: %s - https://github.com/edx/edx-platform/commit/%h' - echo + $COMMITTED_NOT_AUTHORED --format="tformat:|$EMAIL|%s|[commit|https://github.com/edx/edx-platform/commit/%h]|Committed, didn't author|" | head -n 1 + $COMMITTED_NOT_AUTHORED --format="tformat:| |%s|[commit|https://github.com/edx/edx-platform/commit/%h]| |" | tail -n +2 fi done \ No newline at end of file diff --git a/scripts/runone.py b/scripts/runone.py index a644aa077b..b403b09ff9 100755 --- a/scripts/runone.py +++ b/scripts/runone.py @@ -41,18 +41,17 @@ def main(argv): test_py_path = find_full_path(test_py_path) test_spec = "%s:%s.%s" % (test_py_path, test_class, test_method) - settings = None + system = None if test_py_path.startswith('cms'): - settings = 'cms.envs.test' + system = 'cms' elif test_py_path.startswith('lms'): - settings = 'lms.envs.test' + system = 'lms' - if settings: + if system: # Run as a django test suite from django.core import management - django_args = ["django-admin.py", "test", "--pythonpath=."] - django_args.append("--settings=%s" % settings) + django_args = ["./manage.py", system, "--settings", "test", "test"] if args.nocapture: django_args.append("-s") django_args.append(test_spec) diff --git a/scripts/vagrant-provisioning.sh b/scripts/vagrant-provisioning.sh index d4f627b2c4..0243cd36ae 100755 --- a/scripts/vagrant-provisioning.sh +++ b/scripts/vagrant-provisioning.sh @@ -4,6 +4,7 @@ # # Authors: Xavier Antoviaque # David Baumgold +# Yarko Tymciurak # # This software's license gives you freedom; you can copy, convey, # propagate, redistribute and/or modify this program under the terms of @@ -30,102 +31,112 @@ # # This script is ran by `$ vagrant up`, see the README for more explanations +on_create() +{ + # APT - Packages ############################################################## -# APT - Packages ############################################################## - -apt-get update -apt-get install -y python-software-properties vim + apt-get update + apt-get install -y python-software-properties vim -# Curl - No progress bar ###################################################### + # Curl - No progress bar ###################################################### -[[ -f ~vagrant/.curlrc ]] || echo "silent show-error" > ~vagrant/.curlrc -chown vagrant.vagrant ~vagrant/.curlrc + [[ -f ~vagrant/.curlrc ]] || echo "silent show-error" > ~vagrant/.curlrc + chown vagrant.vagrant ~vagrant/.curlrc -# SSH - Known hosts ########################################################### + # SSH - Known hosts ########################################################### -# Github -([[ -f ~vagrant/.ssh/known_hosts ]] && grep "zBX7bKA= ssh" ~vagrant/.ssh/known_hosts) || { - echo "|1|4DtBcMsTM4zgl/jTS7h3ZkmS/Vc=|XkRnn2xEhr8ixOxeskJAzBX7bKA= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~vagrant/.ssh/known_hosts -} -([[ -f ~vagrant/.ssh/known_hosts ]] && grep "jO3J5bvw= ssh" ~vagrant/.ssh/known_hosts) || { - echo "|1|9rANf/qOAPgKH/TXpGuZCAgGxMs=|x9VYWEDI8kiotbhhNXqjO3J5bvw= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~vagrant/.ssh/known_hosts -} -chown vagrant.vagrant ~vagrant/.ssh/known_hosts + # Github + ([[ -f ~vagrant/.ssh/known_hosts ]] && grep "zBX7bKA= ssh" ~vagrant/.ssh/known_hosts) || { + echo "|1|4DtBcMsTM4zgl/jTS7h3ZkmS/Vc=|XkRnn2xEhr8ixOxeskJAzBX7bKA= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~vagrant/.ssh/known_hosts + } + ([[ -f ~vagrant/.ssh/known_hosts ]] && grep "jO3J5bvw= ssh" ~vagrant/.ssh/known_hosts) || { + echo "|1|9rANf/qOAPgKH/TXpGuZCAgGxMs=|x9VYWEDI8kiotbhhNXqjO3J5bvw= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~vagrant/.ssh/known_hosts + } + chown vagrant.vagrant ~vagrant/.ssh/known_hosts -# edX - Development environment ############################################### + # edX - Development environment ############################################### -# Node modules require a filesystem with symlinks (Windows support) -mkdir -p /opt/edx/node_modules /opt/edx/edx-platform/node_modules -([[ -f /etc/fstab ]] && grep '/opt/edx/node_modules' /etc/fstab) || { - echo '/opt/edx/node_modules /opt/edx/edx-platform/node_modules none bind,noauto 0 0' >> /etc/fstab - mount /opt/edx/node_modules -} -# Must be mounted *after* the NFS mount, made manually by Vagrant -([[ -f /etc/cron.d/nodemodules ]] && grep '/opt/edx/node_modules' /etc/cron.d/nodemodules) || { - echo '@reboot root until [ -n "`mount |grep "/opt/edx/edx-platform type"`" ]; do sleep 1; done; mount /opt/edx/node_modules' > /etc/cron.d/nodemodules -} + # Node modules require a filesystem with symlinks (Windows support) + mkdir -p /opt/edx/node_modules /opt/edx/edx-platform/node_modules + ([[ -f /etc/fstab ]] && grep '/opt/edx/node_modules' /etc/fstab) || { + echo '/opt/edx/node_modules /opt/edx/edx-platform/node_modules none bind,noauto 0 0' >> /etc/fstab + mount /opt/edx/node_modules + } + # Must be mounted *after* the NFS mount, made manually by Vagrant + ([[ -f /etc/cron.d/nodemodules ]] && grep '/opt/edx/node_modules' /etc/cron.d/nodemodules) || { + echo '@reboot root until [ -n "`mount |grep "/opt/edx/edx-platform type"`" ]; do sleep 1; done; mount /opt/edx/node_modules' > /etc/cron.d/nodemodules + } -# Force rechecking all prerequisites (could have been fetched outside of the VM) -rm -rf /opt/edx/edx-platform/.prereqs_cache + # Force rechecking all prerequisites (could have been fetched outside of the VM) + rm -rf /opt/edx/edx-platform/.prereqs_cache -# Permissions -chown vagrant.vagrant /opt/edx /opt/edx/node_modules /opt/edx/edx-platform/node_modules + # Permissions + chown vagrant.vagrant /opt/edx /opt/edx/node_modules /opt/edx/edx-platform/node_modules -# For convenience with `vagrant ssh`, the `edx-platform` virtualenv is always -# loaded after the first run, so we need to deactivate that behavior to run -# `create-dev-env.sh`. -[[ -f ~vagrant/.bash_profile ]] && { - mv ~vagrant/.bash_profile ~vagrant/.bash_profile.bak -} -sudo -u vagrant -i bash -c "cd /opt/edx/edx-platform && PROJECT_HOME=/opt/edx ./scripts/create-dev-env.sh -ynq" + # For convenience with `vagrant ssh`, the `edx-platform` virtualenv is always + # loaded after the first run, so we need to deactivate that behavior to run + # `create-dev-env.sh`. + [[ -f ~vagrant/.bash_profile ]] && { + mv ~vagrant/.bash_profile ~vagrant/.bash_profile.bak + } + sudo -u vagrant -i bash -c "cd /opt/edx/edx-platform && PROJECT_HOME=/opt/edx ./scripts/create-dev-env.sh -ynq" -# Load .bashrc ################################################################ -([[ -f ~vagrant/.bash_profile ]] && grep ".bashrc" ~vagrant/.bash_profile) || { - echo -e "\n. /home/vagrant/.bashrc\n" >> ~vagrant/.bash_profile -} + # Load .bashrc ################################################################ + ([[ -f ~vagrant/.bash_profile ]] && grep ".bashrc" ~vagrant/.bash_profile) || { + echo ". /home/vagrant/.bashrc" >> ~vagrant/.bash_profile + } -# Virtualenv - Always load #################################################### + # Virtualenv - Always load #################################################### -([[ -f ~vagrant/.bash_profile ]] && grep "edx-platform/bin/activate" ~vagrant/.bash_profile) || { - echo -e "\n. /home/vagrant/.virtualenvs/edx-platform/bin/activate\n" >> ~vagrant/.bash_profile -} + ([[ -f ~vagrant/.bash_profile ]] && grep "edx-platform/bin/activate" ~vagrant/.bash_profile) || { + echo ". /home/vagrant/.virtualenvs/edx-platform/bin/activate" >> ~vagrant/.bash_profile + } -# Directory ################################################################### + # Directory ################################################################### -grep "cd /opt/edx/edx-platform" ~vagrant/.bash_profile || { - echo -e "\ncd /opt/edx/edx-platform\n" >> ~vagrant/.bash_profile -} + grep "cd /opt/edx/edx-platform" ~vagrant/.bash_profile || { + echo "cd /opt/edx/edx-platform" >> ~vagrant/.bash_profile + } + # Permissions + chown vagrant.vagrant ~vagrant/.bash_profile -# End ######################################################################### - -cat << EOF + cat << EOF ============================================================================== -Success! +Success - Created your development environment! ============================================================================== -Now, from the virtual machine (connect with "vagrant ssh" if vagrant didn't -log you in already), you can start Studio & LMS with the following commands: - -- Learning management system (LMS): - $ rake lms[cms.dev,0.0.0.0:8000] - - => http://192.168.20.40:8000/ - -- Studio: - $ rake cms[dev,0.0.0.0:8001] - - => http://192.168.20.40:8001/ - - -See the README for details. - - EOF +} # End on_create() ######################################################## +## only initialize / setup the development environment once: +# we create node_modules, so that's a good test: +[[ -d /opt/edx/node_modules ]] || on_create +# grab what the Vagrantfile spec'd our IP to be: +# expecting: +# - relevant ip on eth1; +# - line of interest to look like: +# inet 192.168.20.40/24 brd 192.168.20.255 scope global eth1 +MY_IP=$(ip addr show dev eth1 | sed -n '/inet /{s/.*[ ]\(.*\)\/.*/\1/;p}') + +cat << EOF +Connect to your virtual machine with "vagrant ssh". +Some examples you can use from your virtual machine: + +- Start Learning management system (LMS): + $ rake lms[cms.dev,0.0.0.0:8000] + => http://${MY_IP}:8000/ + +- Start Studio: + $ rake cms[dev,0.0.0.0:8001] + => http://${MY_IP}:8001/ + +See the README for more. + +EOF