Andal.LND/edx-platform
6
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into feature/vik/oe-ui

Browse Source
This commit is contained in:
Vik Paruchuri
2013-07-25 11:09:53 -04:00
parent 2dd5338378 c80a05f4d8
commit 5365d54202
258 changed files with 13857 additions and 694 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
AUTHORS
View File

@@ -82,3 +82,5 @@ Adam Palay <adam@edx.org>
Ian Hoover <ihoover@edx.org>
Mukul Goyal <miki@edx.org>
Robert Marks <rmarks@edx.org>
Yarko Tymciurak <yarkot1@gmail.com>

10
CHANGELOG.rst
View File

@@ -5,6 +5,16 @@ These are notable changes in edx-platform. This is a rolling list of changes,
in roughly chronological order, most recent first. Add your entries at or near
the top. Include a label indicating the component affected.
LMS: Added user preferences (arbitrary user/key/value tuples, for which
which user/key is unique) and a REST API for reading users and
preferences. Access to the REST API is restricted by use of the
X-Edx-Api-Key HTTP header (which must match settings.EDX_API_KEY; if
the setting is not present, the API is disabled).
LMS: Added endpoints for AJAX requests to enable/disable notifications
(which are not yet implemented) and a one-click unsubscribe page.
Common: Add a manage.py that knows about edx-platform specific settings and projects
Common: Added *experimental* support for jsinput type.

169
README.md
View File

@@ -12,30 +12,35 @@ installation process.
1. Make sure you have plenty of available disk space, >5GB
2. Install Git: http://git-scm.com/downloads
3. Install VirtualBox: https://www.virtualbox.org/wiki/Download_Old_Builds_4_2
(you need version 4.2.12, as later/earlier versions might not work well with
Vagrant)
3. Install VirtualBox: https://www.virtualbox.org/wiki/Downloads
See http://docs.vagrantup.com/v2/providers/index.html for a list of supported
Providers. You should use VirtualBox >= 4.2.12.
(Windows: later/earlier VirtualBox versions than 4.2.12 have been reported to not work well with
Vagrant. If this is still a problem, you can
install 4.2.12 from https://www.virtualbox.org/wiki/Download_Old_Builds_4_2).
4. Install Vagrant: http://www.vagrantup.com/ (Vagrant 1.2.2 or later)
5. Open a terminal
6. Download the project: `git clone git://github.com/edx/edx-platform.git`
7. Enter the project directory: `cd edx-platform/`
8. (Windows only) Run the commands to
8. (Windows only) Run the commands to
[deal with line endings and symlinks under Windows](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#dealing-with-line-endings-and-symlinks-under-windows)
9. Start: `vagrant up`
9. Create the development environment and start it: `vagrant up`
The last step might require your host machine's administrator password to setup NFS.
The initial `vagrant up` will download a Linux image, then boot and ask for your
host machine's administrator password to setup file sharing between your computer and the VM.
Once file sharing is established, `edx-platform/scripts/create-dev-env.sh` will
install dependencies and configure the VM.
This will take a while; go grab a coffee.
Afterwards, it will download an image, install all the dependencies and configure
the VM. It will take a while, go grab a coffee.
When complete, you should see a _"Success!"_ message.
If not, refer to the
[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).
Once completed, hopefully you should see a "Success!" message indicating that the
installation went fine. (If not, refer to the
[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).)
Your development environment is initialized only on the first bring-up.
Subsequently `vagrant up` commands will boot your virtual machine normally.
Note: by default, the VM will get the IP `192.168.20.40`. If you need to use a
different IP, you can edit the file `Vagrantfile`. If you have already started the
VM with `vagrant up`, see "Stopping and restarting the VM" below to take the change
into account.
Note: by default, the VM will get the IP `192.168.20.40`.
You can change this in your `Vagrantfile` (the startup message will reflect your VM's actual IP).
Accessing the VM
----------------
@@ -46,15 +51,24 @@ Once the installation is finished, to log into the virtual machine:
$ vagrant ssh
```
Note: This won't work from Windows, install install PuTTY from
http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html instead. Then
connect to 127.0.0.1, port 2222, using vagrant/vagrant as a user/password.
Note: This won't work from Windows. Instead, install PuTTY from
http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html. Then
connect to 192.168.20.40, port 2222, using vagrant/vagrant as a user/password.
Using edX
---------
Once inside the VM, you can start Studio and LMS with the following commands
(from the `/opt/edx/edx-platform` folder):
When you login to your VM, you are in
`/opt/edx/edx-platform` by default, which is shared from your host workspace.
Your host computer contains the edx-project development code and repository.
Your VM runs edx-platform code mounted from your host, so
you can develop by editing on your host.
After logging into your VM with `vagrant ssh`,
start the _Studio_ and
_Learning management system (LMS)_
servers (run these from `/opt/edx/edx-platform`):
Learning management system (LMS):
@@ -62,46 +76,85 @@ Learning management system (LMS):
$ rake lms[cms.dev,0.0.0.0:8000]
```
Studio:
Studio (CMS):
```
$ rake cms[dev,0.0.0.0:8001]
```
Once started, open the following URLs in your browser:
The servers will come up to these URLs:
* Learning management system (LMS): http://192.168.20.40:8000/
* Studio (CMS): http://192.168.20.40:8001/
- LMS: http://192.168.20.40:8000/
- CMS: http://192.168.20.40:8001/
You can develop by editing the files directly in the `edx-platform/` directory you
downloaded before, you don't need to connect to the VM to edit them (the VM uses
those files to run edX, mirroring the folder in `/opt/edx/edx-platform`).
Your VM's port 8000 is forwarded to host port 9000
so you can also access the LMS with [http://localhost:9000/]().
Similarly, VM port 8001 is forwarded to host port 9001.
These are set in your `Vagrantfile`.
You may also want to create a super-user with:
```
$ rake django-admin["createsuperuser"]
```
Also note that if you register a new user through the web interface,
the activiation email will be posted to your VM's terminal window (search for
lines similar to):
Note that when you register a new user through the web interface,
by default the activiation email will be appear on your VM's terminal.
Search for lines similar to:
```
Subject: Your account for edX Studio
From: registration@edx.org
```
and find the activation URL for the account you've created.
and find the activation URL.
See the [Frequently Asked Questions](https://github.com/edx/edx-platform/wiki/Frequently-Asked-Questions)
for more usage tips.
Django admin & debug toolbar
-----------------------------
You can enable admin logins and the debug_toolbar by editing
`lms/envs/common.py`:
- enable ADMIN login page by setting:
- ```
'ENABLE_DJANGO_ADMIN_SITE': True
```
- enable debug toolbar by uncommenting:
- ```
# 'debug_toolbar.middleware.DebugToolbarMiddleware',
```
These are also defined in `lms/envs/dev.py`,
and usually active on localhost.
To get at your VM's 127.0.0.1, explicitly forward one of VM's available localhost ports to your computer.
Instead of `vagrant ssh`, login with:
```
$ ssh -L 6080:127.0.0.1:8080 vagrant@192.168.20.40
```
The password is _vagrant_.
From your VM, start the LMS as a localhost instance:
```
$ rake lms[cms.dev,127.0.0.1:8080]
```
You should see the debug toolbar now on [http:/localhost:6080/]().
You should now also see a login on [http://localhost:6080/admin/]()
You will need a privileged user for the admin login.
You can create a CMS/LMS super-user with:
```
$ ./manage.py lms createsuperuser
```
Stopping & starting
-------------------
To stop the VM (from your `edx-platform/` directory):
To stop the VM (from your `edx-platform/` directory):
```
$ vagrant halt
```
@@ -112,16 +165,27 @@ To restart:
$ vagrant up
```
or, to start without attempting to update the dependencies:
To suspend and resume tasks in progress on your VM:
```
$ vagrant suspend
$ # and later...
$ vagrant resume
```
Your development environment is normally created once, on first `vagrant up`.
You can continue to fetch changes in edx-platform
as you work with your VM.
To re-create your VM and create a fresh development environment:
```
$ vagrant up --no-provision
$ vagrant destroy
$ vagrant up # will make a new VM
```
Troubleshooting
---------------
If anything doesn't work as expected, see the
If anything doesn't work as expected, see the
[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).
Installation - Advanced
@@ -229,23 +293,12 @@ or any other process management tool.
Configuring Your Project
------------------------
We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our
project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T`
to view a summary.
Before you run your project, you need to create a sqlite database, create
tables in that database, run database migrations, and populate templates for
CMS templates. Fortunately, `rake` will do all of this for you! Just run:
tables in that database, and run database migrations. Fortunately, `django`
will do all of this for you
$ rake django-admin[syncdb]
$ rake django-admin[migrate]
If you are running these commands using the [`zsh`](http://www.zsh.org/) shell,
zsh will assume that you are doing
[shell globbing](https://en.wikipedia.org/wiki/Glob_%28programming%29), search for
a file in your directory named `django-adminsyncdb` or `django-adminmigrate`,
and fail. To fix this, just surround the argument with quotation marks, so that
you're running `rake "django-admin[syncdb]"`.
$ ./manage.py lms syncdb --migrate
$ ./manage.py cms syncdb --migrate
Run Your Project
----------------
@@ -253,6 +306,10 @@ edX has two components: Studio, the course authoring system; and the LMS
(learning management system) used by students. These two systems communicate
through the MongoDB database, which stores course information.
We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our
project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T`
to view a summary.
To run Studio, run:
$ rake cms

2
cms/djangoapps/contentstore/features/checklists.feature
View File

@@ -8,7 +8,7 @@ Feature: Course checklists
Scenario: A course author can mark tasks as complete
Given I have opened Checklists
Then I can check and uncheck tasks in a checklist
And They are correctly selected after I reload the page
And They are correctly selected after reloading the page
Scenario: A task can link to a location within Studio
Given I have opened Checklists

2
cms/djangoapps/contentstore/features/checklists.py
View File

@@ -45,7 +45,7 @@ def i_can_check_and_uncheck_tasks(step):
verifyChecklist2Status(2, 7, 29)
@step('They are correctly selected after I reload the page$')
@step('They are correctly selected after reloading the page$')
def tasks_correctly_selected_after_reload(step):
reload_the_page(step)
verifyChecklist2Status(2, 7, 29)

14
cms/djangoapps/contentstore/features/common.py
View File

@@ -209,7 +209,8 @@ def i_created_a_video_component(step):
world.create_component_instance(
step, '.large-video-icon',
'video',
'.xmodule_VideoModule'
'.xmodule_VideoModule',
has_multiple_templates=False
)
@@ -238,6 +239,17 @@ def save_button_disabled(step):
assert world.css_has_class(button_css, disabled)
@step('I confirm the prompt')
def confirm_the_prompt(step):
prompt_css = 'a.button.action-primary'
world.css_click(prompt_css)
@step(u'I am shown a (.*)$')
def i_am_shown_a_notification(step, notification_type):
assert world.is_css_present('.wrapper-%s' % notification_type)
def type_in_codemirror(index, text):
world.css_click(".CodeMirror", index=index)
g = world.css_find("div.CodeMirror.CodeMirror-focused > div > textarea")

18
cms/djangoapps/contentstore/features/component.feature
View File

@@ -67,3 +67,21 @@ Feature: Component Adding
When I will confirm all alerts
And I delete all components
Then I see no components
Scenario: I see a prompt on delete
Given I have opened a new course in studio
And I am editing a new unit
And I add the following components:
| Component |
| Discussion |
And I delete a component
Then I am shown a prompt
Scenario: I see a notification on save
Given I have opened a new course in studio
And I am editing a new unit
And I add the following components:
| Component |
| Discussion |
And I edit and save a component
Then I am shown a notification

11
cms/djangoapps/contentstore/features/component.py
View File

@@ -41,6 +41,17 @@ def see_no_components(steps):
assert world.is_css_not_present('li.component')
@step(u'I delete a component')
def delete_one_component(step):
world.css_click('a.delete-button')
@step(u'I edit and save a component')
def edit_and_save_component(step):
world.css_click('.edit-button')
world.css_click('.save-button')
def step_selector_list(data_type, path, index=1):
selector_list = ['a[data-type="{}"]'.format(data_type)]
if index != 1:

13
cms/djangoapps/contentstore/features/component_settings_editor_helpers.py
View File

@@ -7,10 +7,16 @@ from terrain.steps import reload_the_page
@world.absorb
def create_component_instance(step, component_button_css, category, expected_css, boilerplate=None):
click_new_component_button(step, component_button_css)
click_component_from_menu(category, boilerplate, expected_css)
def create_component_instance(step, component_button_css, category,
expected_css, boilerplate=None,
has_multiple_templates=True):
click_new_component_button(step, component_button_css)
if has_multiple_templates:
click_component_from_menu(category, boilerplate, expected_css)
assert_equal(1, len(world.css_find(expected_css)))
@world.absorb
def click_new_component_button(step, component_button_css):
@@ -34,7 +40,6 @@ def click_component_from_menu(category, boilerplate, expected_css):
elements = world.css_find(elem_css)
assert_equal(len(elements), 1)
world.css_click(elem_css)
assert_equal(1, len(world.css_find(expected_css)))
@world.absorb

12
cms/djangoapps/contentstore/features/studio-overview-togglesection.feature → cms/djangoapps/contentstore/features/course-overview.feature
View File

@@ -1,7 +1,7 @@
Feature: Overview Toggle Section
In order to quickly view the details of a course's section or to scan the inventory of sections
Feature: Course Overview
In order to quickly view the details of a course's section and set release dates and grading
As a course author
I want to toggle the visibility of each section's subsection details in the overview listing
I want to use the course overview page
Scenario: The default layout for the overview page is to show sections in expanded view
Given I have a course with multiple sections
@@ -57,3 +57,9 @@ Feature: Overview Toggle Section
And I click the "Expand All Sections" link
Then I see the "Collapse All Sections" link
And all sections are expanded
Scenario: Notification is shown on grading status changes
Given I have a course with 1 section
When I navigate to the course overview page
And I change an assignment's grading status
Then I am shown a notification

6
cms/djangoapps/contentstore/features/studio-overview-togglesection.py → cms/djangoapps/contentstore/features/course-overview.py
View File

@@ -118,3 +118,9 @@ def all_sections_are_collapsed(step):
subsections = world.css_find(subsection_locator)
for index in range(len(subsections)):
assert_false(world.css_visible(subsection_locator, index=index))
@step(u"I change an assignment's grading status")
def change_grading_status(step):
world.css_find('a.menu-toggle').click()
world.css_find('.menu li').first.click()

3
cms/djangoapps/contentstore/features/discussion-editor.py
View File

@@ -9,7 +9,8 @@ def i_created_discussion_tag(step):
world.create_component_instance(
step, '.large-discussion-icon',
'discussion',
'.xmodule_DiscussionModule'
'.xmodule_DiscussionModule',
has_multiple_templates=False
)

2
cms/djangoapps/contentstore/features/html-editor.py
View File

@@ -14,4 +14,4 @@ def i_created_blank_html_page(step):
@step('I see only the HTML display name setting$')
def i_see_only_the_html_display_name(step):
world.verify_all_setting_entries([['Display Name', "Blank HTML Page", False]])
world.verify_all_setting_entries([['Display Name', "Text", False]])

3
cms/djangoapps/contentstore/features/problem-editor.py
View File

@@ -170,7 +170,8 @@ def edit_latex_source(step):
@step('my change to the High Level Source is persisted')
def high_level_source_persisted(step):
def verify_text(driver):
return world.css_text('.problem') == 'hi'
css_sel = '.problem div>span'
return world.css_text(css_sel) == 'hi'
world.wait_for(verify_text)

1
cms/djangoapps/contentstore/features/section.feature
View File

@@ -33,4 +33,5 @@ Feature: Create Section
And I have added a new section
When I will confirm all alerts
And I press the "section" delete icon
And I confirm the prompt
Then the section does not exist

1
cms/djangoapps/contentstore/features/subsection.feature
View File

@@ -38,4 +38,5 @@ Feature: Create Subsection
And I see my subsection on the Courseware page
When I will confirm all alerts
And I press the "subsection" delete icon
And I confirm the prompt
Then the subsection does not exist

2
cms/djangoapps/contentstore/features/video-editor.py
View File

@@ -7,7 +7,7 @@ from lettuce import world, step
@step('I see the correct settings and default values$')
def i_see_the_correct_settings_and_values(step):
world.verify_all_setting_entries([['Default Speed', 'OEoXaMPEzfM', False],
['Display Name', 'Video Title', False],
['Display Name', 'Video', False],
['Download Track', '', False],
['Download Video', '', False],
['Show Captions', 'True', False],

17
cms/djangoapps/contentstore/tests/test_contentstore.py
View File

@@ -312,6 +312,23 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase):
self.assertGreater(len(course.textbooks), 0)
def test_default_tabs_on_create_course(self):
module_store = modulestore('direct')
CourseFactory.create(org='edX', course='999', display_name='Robot Super Course')
course_location = Location(['i4x', 'edX', '999', 'course', 'Robot_Super_Course', None])
course = module_store.get_item(course_location)
expected_tabs = []
expected_tabs.append({u'type': u'courseware'})
expected_tabs.append({u'type': u'course_info', u'name': u'Course Info'})
expected_tabs.append({u'type': u'textbooks'})
expected_tabs.append({u'type': u'discussion', u'name': u'Discussion'})
expected_tabs.append({u'type': u'wiki', u'name': u'Wiki'})
expected_tabs.append({u'type': u'progress', u'name': u'Progress'})
self.assertEqual(course.tabs, expected_tabs)
def test_static_tab_reordering(self):
module_store = modulestore('direct')
CourseFactory.create(org='edX', course='999', display_name='Robot Super Course')

5
cms/djangoapps/contentstore/tests/test_course_updates.py
View File

@@ -36,8 +36,11 @@ class CourseUpdateTest(CourseTestCase):
'provided_id': payload['id']})
content += '<div>div <p>p<br/></p></div>'
payload['content'] = content
# POST requests were coming in w/ these header values causing an error; so, repro error here
resp = self.client.post(first_update_url, json.dumps(payload),
"application/json")
"application/json",
HTTP_X_HTTP_METHOD_OVERRIDE="PUT",
REQUEST_METHOD="POST")
self.assertHTMLEqual(content, json.loads(resp.content)['content'],
"iframe w/ div")

186
cms/djangoapps/contentstore/tests/test_crud.py Normal file
View File

@@ -0,0 +1,186 @@
'''
Created on May 7, 2013
@author: dmitchell
'''
import unittest
from xmodule import templates
from xmodule.modulestore.tests import persistent_factories
from xmodule.course_module import CourseDescriptor
from xmodule.modulestore.django import modulestore
from xmodule.seq_module import SequenceDescriptor
from xmodule.x_module import XModuleDescriptor
from xmodule.capa_module import CapaDescriptor
from xmodule.modulestore.locator import CourseLocator, BlockUsageLocator
from xmodule.modulestore.exceptions import ItemNotFoundError
from xmodule.html_module import HtmlDescriptor
class TemplateTests(unittest.TestCase):
"""
Test finding and using the templates (boilerplates) for xblocks.
"""
def test_get_templates(self):
found = templates.all_templates()
self.assertIsNotNone(found.get('course'))
self.assertIsNotNone(found.get('about'))
self.assertIsNotNone(found.get('html'))
self.assertIsNotNone(found.get('problem'))
self.assertEqual(len(found.get('course')), 0)
self.assertEqual(len(found.get('about')), 1)
self.assertGreaterEqual(len(found.get('html')), 2)
self.assertGreaterEqual(len(found.get('problem')), 10)
dropdown = None
for template in found['problem']:
self.assertIn('metadata', template)
self.assertIn('display_name', template['metadata'])
if template['metadata']['display_name'] == 'Dropdown':
dropdown = template
break
self.assertIsNotNone(dropdown)
self.assertIn('markdown', dropdown['metadata'])
self.assertIn('data', dropdown)
self.assertRegexpMatches(dropdown['metadata']['markdown'], r'^Dropdown.*')
self.assertRegexpMatches(dropdown['data'], r'<problem>\s*<p>Dropdown.*')
def test_get_some_templates(self):
self.assertEqual(len(SequenceDescriptor.templates()), 0)
self.assertGreater(len(HtmlDescriptor.templates()), 0)
self.assertIsNone(SequenceDescriptor.get_template('doesntexist.yaml'))
self.assertIsNone(HtmlDescriptor.get_template('doesntexist.yaml'))
self.assertIsNotNone(HtmlDescriptor.get_template('announcement.yaml'))
def test_factories(self):
test_course = persistent_factories.PersistentCourseFactory.create(org='testx', prettyid='tempcourse',
display_name='fun test course', user_id='testbot')
self.assertIsInstance(test_course, CourseDescriptor)
self.assertEqual(test_course.display_name, 'fun test course')
index_info = modulestore('split').get_course_index_info(test_course.location)
self.assertEqual(index_info['org'], 'testx')
self.assertEqual(index_info['prettyid'], 'tempcourse')
test_chapter = persistent_factories.ItemFactory.create(display_name='chapter 1',
parent_location=test_course.location)
self.assertIsInstance(test_chapter, SequenceDescriptor)
# refetch parent which should now point to child
test_course = modulestore('split').get_course(test_chapter.location)
self.assertIn(test_chapter.location.usage_id, test_course.children)
def test_temporary_xblocks(self):
"""
Test using load_from_json to create non persisted xblocks
"""
test_course = persistent_factories.PersistentCourseFactory.create(org='testx', prettyid='tempcourse',
display_name='fun test course', user_id='testbot')
test_chapter = XModuleDescriptor.load_from_json({'category': 'chapter',
'metadata': {'display_name': 'chapter n'}},
test_course.system, parent_xblock=test_course)
self.assertIsInstance(test_chapter, SequenceDescriptor)
self.assertEqual(test_chapter.display_name, 'chapter n')
self.assertIn(test_chapter, test_course.get_children())
# test w/ a definition (e.g., a problem)
test_def_content = '<problem>boo</problem>'
test_problem = XModuleDescriptor.load_from_json({'category': 'problem',
'definition': {'data': test_def_content}},
test_course.system, parent_xblock=test_chapter)
self.assertIsInstance(test_problem, CapaDescriptor)
self.assertEqual(test_problem.data, test_def_content)
self.assertIn(test_problem, test_chapter.get_children())
test_problem.display_name = 'test problem'
self.assertEqual(test_problem.display_name, 'test problem')
def test_persist_dag(self):
"""
try saving temporary xblocks
"""
test_course = persistent_factories.PersistentCourseFactory.create(org='testx', prettyid='tempcourse',
display_name='fun test course', user_id='testbot')
test_chapter = XModuleDescriptor.load_from_json({'category': 'chapter',
'metadata': {'display_name': 'chapter n'}},
test_course.system, parent_xblock=test_course)
test_def_content = '<problem>boo</problem>'
test_problem = XModuleDescriptor.load_from_json({'category': 'problem',
'definition': {'data': test_def_content}},
test_course.system, parent_xblock=test_chapter)
# better to pass in persisted parent over the subdag so
# subdag gets the parent pointer (otherwise 2 ops, persist dag, update parent children,
# persist parent
persisted_course = modulestore('split').persist_xblock_dag(test_course, 'testbot')
self.assertEqual(len(persisted_course.children), 1)
persisted_chapter = persisted_course.get_children()[0]
self.assertEqual(persisted_chapter.category, 'chapter')
self.assertEqual(persisted_chapter.display_name, 'chapter n')
self.assertEqual(len(persisted_chapter.children), 1)
persisted_problem = persisted_chapter.get_children()[0]
self.assertEqual(persisted_problem.category, 'problem')
self.assertEqual(persisted_problem.data, test_def_content)
def test_delete_course(self):
test_course = persistent_factories.PersistentCourseFactory.create(
org='testx',
prettyid='edu.harvard.history.doomed',
display_name='doomed test course',
user_id='testbot')
persistent_factories.ItemFactory.create(display_name='chapter 1',
parent_location=test_course.location)
id_locator = CourseLocator(course_id=test_course.location.course_id, revision='draft')
guid_locator = CourseLocator(version_guid=test_course.location.version_guid)
# verify it can be retireved by id
self.assertIsInstance(modulestore('split').get_course(id_locator), CourseDescriptor)
# and by guid
self.assertIsInstance(modulestore('split').get_course(guid_locator), CourseDescriptor)
modulestore('split').delete_course(id_locator.course_id)
# test can no longer retrieve by id
self.assertRaises(ItemNotFoundError, modulestore('split').get_course, id_locator)
# but can by guid
self.assertIsInstance(modulestore('split').get_course(guid_locator), CourseDescriptor)
def test_block_generations(self):
"""
Test get_block_generations
"""
test_course = persistent_factories.PersistentCourseFactory.create(
org='testx',
prettyid='edu.harvard.history.hist101',
display_name='history test course',
user_id='testbot')
chapter = persistent_factories.ItemFactory.create(display_name='chapter 1',
parent_location=test_course.location, user_id='testbot')
sub = persistent_factories.ItemFactory.create(display_name='subsection 1',
parent_location=chapter.location, user_id='testbot', category='vertical')
first_problem = persistent_factories.ItemFactory.create(display_name='problem 1',
parent_location=sub.location, user_id='testbot', category='problem', data="<problem></problem>")
first_problem.max_attempts = 3
updated_problem = modulestore('split').update_item(first_problem, 'testbot')
updated_loc = modulestore('split').delete_item(updated_problem.location, 'testbot')
second_problem = persistent_factories.ItemFactory.create(display_name='problem 2',
parent_location=BlockUsageLocator(updated_loc, usage_id=sub.location.usage_id),
user_id='testbot', category='problem', data="<problem></problem>")
# course root only updated 2x
version_history = modulestore('split').get_block_generations(test_course.location)
self.assertEqual(version_history.locator.version_guid, test_course.location.version_guid)
self.assertEqual(len(version_history.children), 1)
self.assertEqual(version_history.children[0].children, [])
self.assertEqual(version_history.children[0].locator.version_guid, chapter.location.version_guid)
# sub changed on add, add problem, delete problem, add problem in strict linear seq
version_history = modulestore('split').get_block_generations(sub.location)
self.assertEqual(len(version_history.children), 1)
self.assertEqual(len(version_history.children[0].children), 1)
self.assertEqual(len(version_history.children[0].children[0].children), 1)
self.assertEqual(len(version_history.children[0].children[0].children[0].children), 0)
# first and second problem may show as same usage_id; so, need to ensure their histories are right
version_history = modulestore('split').get_block_generations(updated_problem.location)
self.assertEqual(version_history.locator.version_guid, first_problem.location.version_guid)
self.assertEqual(len(version_history.children), 1) # updated max_attempts
self.assertEqual(len(version_history.children[0].children), 0)
version_history = modulestore('split').get_block_generations(second_problem.location)
self.assertNotEqual(version_history.locator.version_guid, first_problem.location.version_guid)

5
cms/djangoapps/contentstore/views/assets.py
View File

@@ -13,7 +13,7 @@ from django_future.csrf import ensure_csrf_cookie
from django.core.urlresolvers import reverse
from django.core.servers.basehttp import FileWrapper
from django.core.files.temp import NamedTemporaryFile
from django.views.decorators.http import require_POST
from django.views.decorators.http import require_POST, require_http_methods
from mitxmako.shortcuts import render_to_response
from cache_toolbox.core import del_cached_content
@@ -249,6 +249,7 @@ def remove_asset(request, org, course, name):
@ensure_csrf_cookie
@require_http_methods(("GET", "POST", "PUT"))
@login_required
def import_course(request, org, course, name):
"""
@@ -256,7 +257,7 @@ def import_course(request, org, course, name):
"""
location = get_location_and_verify_access(request, org, course, name)
if request.method == 'POST':
if request.method in ('POST', 'PUT'):
filename = request.FILES['course-data'].name
if not filename.endswith('.tar.gz'):

3
cms/djangoapps/contentstore/views/checklist.py
View File

@@ -37,6 +37,7 @@ def get_checklists(request, org, course, name):
checklists, modified = expand_checklist_action_urls(course_module)
if copied or modified:
course_module.save()
modulestore.update_metadata(location, own_metadata(course_module))
return render_to_response('checklists.html',
{
@@ -69,6 +70,7 @@ def update_checklist(request, org, course, name, checklist_index=None):
# seeming noop which triggers kvs to record that the metadata is not default
course_module.checklists = course_module.checklists
checklists, _ = expand_checklist_action_urls(course_module)
course_module.save()
modulestore.update_metadata(location, own_metadata(course_module))
return JsonResponse(checklists[index])
else:
@@ -79,6 +81,7 @@ def update_checklist(request, org, course, name, checklist_index=None):
# In the JavaScript view initialize method, we do a fetch to get all the checklists.
checklists, modified = expand_checklist_action_urls(course_module)
if modified:
course_module.save()
modulestore.update_metadata(location, own_metadata(course_module))
return JsonResponse(checklists)

3
cms/djangoapps/contentstore/views/component.py
View File

@@ -245,6 +245,7 @@ def edit_unit(request, location):
@expect_json
@login_required
@require_http_methods(("GET", "POST", "PUT"))
@ensure_csrf_cookie
def assignment_type_update(request, org, course, category, name):
'''
@@ -256,7 +257,7 @@ def assignment_type_update(request, org, course, category, name):
if request.method == 'GET':
return JsonResponse(CourseGradingModel.get_section_grader_type(location))
elif request.method == 'POST': # post or put, doesn't matter.
elif request.method in ('POST', 'PUT'): # post or put, doesn't matter.
return JsonResponse(CourseGradingModel.update_section_grader_type(location, request.POST))

12
cms/djangoapps/contentstore/views/course.py
View File

@@ -42,8 +42,7 @@ from .component import (
ADVANCED_COMPONENT_POLICY_KEY)
from django_comment_common.utils import seed_permissions_roles
import datetime
from django.utils.timezone import UTC
from xmodule.html_module import AboutDescriptor
__all__ = ['course_index', 'create_new_course', 'course_info',
'course_info_updates', 'get_course_settings',
@@ -176,6 +175,7 @@ def course_info(request, org, course, name, provided_id=None):
@expect_json
@require_http_methods(("GET", "POST", "PUT", "DELETE"))
@login_required
@ensure_csrf_cookie
def course_info_updates(request, org, course, provided_id=None):
@@ -206,7 +206,7 @@ def course_info_updates(request, org, course, provided_id=None):
except:
return HttpResponseBadRequest("Failed to delete",
content_type="text/plain")
elif request.method == 'POST':
elif request.method in ('POST', 'PUT'): # can be either and sometimes django is rewriting one to the other
try:
return JsonResponse(update_course_updates(location, request.POST, provided_id))
except:
@@ -300,7 +300,7 @@ def course_settings_updates(request, org, course, name, section):
if request.method == 'GET':
# Cannot just do a get w/o knowing the course name :-(
return JsonResponse(manager.fetch(Location(['i4x', org, course, 'course', name])), encoder=CourseSettingsEncoder)
elif request.method == 'POST': # post or put, doesn't matter.
elif request.method in ('POST', 'PUT'): # post or put, doesn't matter.
return JsonResponse(manager.update_from_json(request.POST), encoder=CourseSettingsEncoder)
@@ -479,7 +479,7 @@ def textbook_index(request, org, course, name):
if request.is_ajax():
if request.method == 'GET':
return JsonResponse(course_module.pdf_textbooks)
elif request.method == 'POST':
elif request.method in ('POST', 'PUT'): # can be either and sometimes django is rewriting one to the other
try:
textbooks = validate_textbooks_json(request.body)
except TextbookValidationError as err:
@@ -580,7 +580,7 @@ def textbook_by_id(request, org, course, name, tid):
if not textbook:
return JsonResponse(status=404)
return JsonResponse(textbook)
elif request.method in ('POST', 'PUT'):
elif request.method in ('POST', 'PUT'): # can be either and sometimes django is rewriting one to the other
try:
new_textbook = validate_textbook_json(request.body)
except TextbookValidationError as err:

3
cms/djangoapps/contentstore/views/user.py
View File

@@ -83,6 +83,9 @@ def add_user(request, location):
}
return JsonResponse(msg, 400)
# remove leading/trailing whitespace if necessary
email = email.strip()
# check that logged in user has admin permissions to this course
if not has_access(request.user, location, role=INSTRUCTOR_ROLE_NAME):
raise PermissionDenied()

5
cms/envs/aws.py
View File

@@ -92,6 +92,7 @@ LOG_DIR = ENV_TOKENS['LOG_DIR']
CACHES = ENV_TOKENS['CACHES']
SESSION_COOKIE_DOMAIN = ENV_TOKENS.get('SESSION_COOKIE_DOMAIN')
SESSION_ENGINE = ENV_TOKENS.get('SESSION_ENGINE', SESSION_ENGINE)
# allow for environments to specify what cookie name our login subsystem should use
# this is to fix a bug regarding simultaneous logins between edx.org and edge.edx.org which can
@@ -122,6 +123,10 @@ LOGGING = get_logger_config(LOG_DIR,
debug=False,
service_variant=SERVICE_VARIANT)
#theming start:
PLATFORM_NAME = ENV_TOKENS.get('PLATFORM_NAME', 'edX')
################ SECURE AUTH ITEMS ###############################
# Secret things: passwords, access keys, etc.
with open(ENV_ROOT / CONFIG_PREFIX + "auth.json") as auth_file:

4
cms/envs/dev.py
View File

@@ -33,6 +33,10 @@ MODULESTORE = {
'direct': {
'ENGINE': 'xmodule.modulestore.mongo.MongoModuleStore',
'OPTIONS': modulestore_options
},
'split': {
'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore',
'OPTIONS': modulestore_options
}
}

4
cms/envs/test.py
View File

@@ -63,6 +63,10 @@ MODULESTORE = {
'draft': {
'ENGINE': 'xmodule.modulestore.draft.DraftModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
},
'split': {
'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
}
}

16
cms/manage.py
View File

@@ -1,16 +0,0 @@
#!/usr/bin/env python
from django.core.management import execute_manager
import imp
try:
imp.find_module('settings') # Assumed to be in the same directory.
except ImportError:
import sys
sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. "
"It appears you've customized things.\nYou'll have to run django-admin.py, "
"passing it your settings module.\n" % __file__)
sys.exit(1)
import settings
if __name__ == "__main__":
execute_manager(settings)

33
cms/static/coffee/spec/views/overview_spec.coffee
View File

@@ -40,17 +40,30 @@ describe "Course Overview", ->
</div>
"""#"
appendSetFixtures """
<section class="courseware-section branch" data-id="a-location-goes-here">
<li class="branch collapsed id-holder" data-id="an-id-goes-here">
<a href="#" class="delete-section-button"></a>
</li>
</section>
"""#"
spyOn(window, 'saveSetSectionScheduleDate').andCallThrough()
# Have to do this here, as it normally gets bound in document.ready()
$('a.save-button').click(saveSetSectionScheduleDate)
$('a.delete-section-button').click(deleteSection)
@notificationSpy = spyOn(CMS.Views.Notification.Mini.prototype, 'show').andCallThrough()
window.analytics = jasmine.createSpyObj('analytics', ['track'])
window.course_location_analytics = jasmine.createSpy()
sinon.useFakeXMLHttpRequest()
@xhr = sinon.useFakeXMLHttpRequest()
requests = @requests = []
@xhr.onCreate = (req) -> requests.push(req)
afterEach ->
delete window.analytics
delete window.course_location_analytics
@notificationSpy.reset()
it "should save model when save is clicked", ->
$('a.edit-button').click()
@@ -61,3 +74,21 @@ describe "Course Overview", ->
$('a.edit-button').click()
$('a.save-button').click()
expect(@notificationSpy).toHaveBeenCalled()
it "should delete model when delete is clicked", ->
deleteSpy = spyOn(window, '_deleteItem').andCallThrough()
$('a.delete-section-button').click()
$('a.action-primary').click()
expect(deleteSpy).toHaveBeenCalled()
expect(@requests[0].url).toEqual('/delete_item')
it "should not delete model when cancel is clicked", ->
deleteSpy = spyOn(window, '_deleteItem').andCallThrough()
$('a.delete-section-button').click()
$('a.action-secondary').click()
expect(@requests.length).toEqual(0)
it "should show a confirmation on delete", ->
$('a.delete-section-button').click()
$('a.action-primary').click()
expect(@notificationSpy).toHaveBeenCalled()

4
cms/static/coffee/src/views/module_edit.coffee
View File

@@ -84,11 +84,15 @@ class CMS.Views.ModuleEdit extends Backbone.View
data.metadata = _.extend(data.metadata || {}, @changedMetadata())
@hideModal()
saving = new CMS.Views.Notification.Mini
title: gettext('Saving') + '&hellip;'
saving.show()
@model.save(data).done( =>
# # showToastMessage("Your changes have been saved.", null, 3)
@module = null
@render()
@$el.removeClass('editing')
saving.hide()
).fail( ->
showToastMessage(gettext("There was an error saving your changes. Please try again."), null, 3)
)

60
cms/static/coffee/src/views/unit.coffee
View File

@@ -67,8 +67,8 @@ class CMS.Views.UnitEdit extends Backbone.View
type = $(event.currentTarget).data('type')
@$newComponentTypePicker.slideUp(250)
@$(".new-component-#{type}").slideDown(250)
$('html, body').animate({
scrollTop: @$(".new-component-#{type}").offset().top
$('html, body').animate({
scrollTop: @$(".new-component-#{type}").offset().top
}, 500)
closeNewComponent: (event) =>
@@ -115,27 +115,43 @@ class CMS.Views.UnitEdit extends Backbone.View
@model.save()
deleteComponent: (event) =>
if not confirm 'Are you sure you want to delete this component? This action cannot be undone.'
return
$component = $(event.currentTarget).parents('.component')
$.post('/delete_item', {
id: $component.data('id')
}, =>
analytics.track "Deleted a Component",
course: course_location_analytics
unit_id: unit_location_analytics
id: $component.data('id')
msg = new CMS.Views.Prompt.Warning(
title: gettext('Delete this component?'),
message: gettext('Deleting this component is permanent and cannot be undone.'),
actions:
primary:
text: gettext('Yes, delete this component'),
click: (view) =>
view.hide()
deleting = new CMS.Views.Notification.Mini
title: gettext('Deleting') + '&hellip;',
deleting.show()
$component = $(event.currentTarget).parents('.component')
$.post('/delete_item', {
id: $component.data('id')
}, =>
deleting.hide()
analytics.track "Deleted a Component",
course: course_location_analytics
unit_id: unit_location_analytics
id: $component.data('id')
$component.remove()
# b/c we don't vigilantly keep children up to date
# get rid of it before it hurts someone
# sorry for the js, i couldn't figure out the coffee equivalent
`_this.model.save({children: _this.components()},
{success: function(model) {
model.unset('children');
}}
);`
$component.remove()
# b/c we don't vigilantly keep children up to date
# get rid of it before it hurts someone
# sorry for the js, i couldn't figure out the coffee equivalent
`_this.model.save({children: _this.components()},
{success: function(model) {
model.unset('children');
}}
);`
)
secondary:
text: gettext('Cancel'),
click: (view) ->
view.hide()
)
msg.show()
deleteDraft: (event) ->
@wait(true)
@@ -236,7 +252,7 @@ class CMS.Views.UnitEdit.NameEdit extends Backbone.View
class CMS.Views.UnitEdit.LocationState extends Backbone.View
initialize: =>
@model.on('change:state', @render)
render: =>
@$el.toggleClass("#{@model.previous('state')}-item #{@model.get('state')}-item")

64
cms/static/js/base.js
View File

@@ -356,39 +356,61 @@ function createNewUnit(e) {
function deleteUnit(e) {
e.preventDefault();
_deleteItem($(this).parents('li.leaf'));
_deleteItem($(this).parents('li.leaf'), 'Unit');
}
function deleteSubsection(e) {
e.preventDefault();
_deleteItem($(this).parents('li.branch'));
_deleteItem($(this).parents('li.branch'), 'Subsection');
}
function deleteSection(e) {
e.preventDefault();
_deleteItem($(this).parents('section.branch'));
_deleteItem($(this).parents('section.branch'), 'Section');
}
function _deleteItem($el) {
if (!confirm(gettext('Are you sure you wish to delete this item. It cannot be reversed!'))) return;
function _deleteItem($el, type) {
var confirm = new CMS.Views.Prompt.Warning({
title: gettext('Delete this ' + type + '?'),
message: gettext('Deleting this ' + type + ' is permanent and cannot be undone.'),
actions: {
primary: {
text: gettext('Yes, delete this ' + type),
click: function(view) {
view.hide();
var id = $el.data('id');
var id = $el.data('id');
analytics.track('Deleted an Item', {
'course': course_location_analytics,
'id': id
});
$.post('/delete_item', {
'id': id,
'delete_children': true,
'delete_all_versions': true
},
function(data) {
$el.remove();
analytics.track('Deleted an Item', {
'course': course_location_analytics,
'id': id
});
var deleting = new CMS.Views.Notification.Mini({
title: gettext('Deleting') + '&hellip;'
});
deleting.show();
$.post('/delete_item',
{'id': id,
'delete_children': true,
'delete_all_versions': true},
function(data) {
$el.remove();
deleting.hide();
}
);
}
},
secondary: {
text: gettext('Cancel'),
click: function(view) {
view.hide();
}
}
}
});
confirm.show();
}
function markAsLoaded() {
@@ -728,7 +750,7 @@ function saveSetSectionScheduleDate(e) {
var $thisSection = $('.courseware-section[data-id="' + id + '"]');
var html = _.template(
'<span class="published-status">' +
'<strong>' + gettext("Will Release: ") + '</strong>' +
'<strong>' + gettext("Will Release:") + '&nbsp;</strong>' +
gettext("<%= date %> at <%= time %> UTC") +
'</span>' +
'<a href="#" class="edit-button" data-date="<%= date %>" data-time="<%= time %>" data-id="<%= id %>">' +

2
cms/static/js/views/assets.js
View File

@@ -9,7 +9,7 @@ function removeAsset(e){
e.preventDefault();
var that = this;
var msg = new CMS.Views.Prompt.Confirmation({
var msg = new CMS.Views.Prompt.Warning({
title: gettext("Delete File Confirmation"),
message: gettext("Are you sure you wish to delete this item. It cannot be reversed!\n\nAlso any content that links/refers to this item will no longer work (e.g. broken images and/or links)"),
actions: {

11
cms/static/js/views/grader-select-view.js
View File

@@ -81,9 +81,18 @@ CMS.Views.OverviewAssignmentGrader = Backbone.View.extend({
this.removeMenu(e);
var saving = new CMS.Views.Notification.Mini({
title: gettext('Saving') + '&hellip;'
});
saving.show();
// TODO I'm not happy with this string fetch via the html for what should be an id. I'd rather use the id attr
// of the CourseGradingPolicy model or null for Not Graded (NOTE, change template's if check for is-selected accordingly)
this.assignmentGrade.save('graderType', $(e.target).text());
this.assignmentGrade.save(
'graderType',
$(e.target).text(),
{success: function () { saving.hide(); }}
);
this.render();
}

1
cms/static/js/views/settings/main_settings_view.js
View File

@@ -107,6 +107,7 @@ CMS.Views.Settings.Details = CMS.Views.ValidatingView.extend({
// to pick up when the date is typed directly in the field.
datefield.change(setfield);
timefield.on('changeTime', setfield);
timefield.on('input', setfield);
datefield.datepicker('setDate', this.model.get(fieldName));
// timepicker doesn't let us set null, so check that we have a time

2
cms/static/js/views/textbook.js
View File

@@ -241,7 +241,7 @@ CMS.Views.EditChapter = Backbone.View.extend({
asset_path: this.$("input.chapter-asset-path").val()
});
var msg = new CMS.Models.FileUpload({
title: _.template(gettext("Upload a new asset to “<%= name %>”"),
title: _.template(gettext("Upload a new PDF to “<%= name %>”"),
{name: section.escape('name')}),
message: "Files must be in PDF format."
});

5
cms/static/sass/views/_index.scss
View File

@@ -71,8 +71,13 @@ body.index {
color: $white;
}
.wrapper-text-welcome, .logo {
display: inline-block;
}
.logo {
font-weight: 600;
margin-left: ($baseline/2);
}
.tagline {

19
cms/static/sass/views/_unit.scss
View File

@@ -747,6 +747,7 @@ body.unit {
// Unit Page Sidebar
.unit-settings {
.window-contents {
padding: $baseline/2 $baseline;
}
@@ -854,6 +855,24 @@ body.unit {
}
.unit-location {
// unit id
.wrapper-unit-id {
.unit-id {
.label {
@extend .t-title7;
margin-bottom: ($baseline/4);
color: $gray-d1;
}
.value {
margin-bottom: 0;
}
}
}
.url {
box-shadow: none;
width: 100%;

41
cms/templates/edit-static-page.html
View File

@@ -1,41 +0,0 @@
<%inherit file="base.html" />
<%! from django.core.urlresolvers import reverse %>
<%block name="title">Editing Static Page</%block>
<%block name="bodyclass">is-signedin course pages edit-static-page</%block>
<%block name="content">
<div class="main-wrapper">
<div class="inner-wrapper">
<div class="main-column">
<article class="static-page-details">
<div class="row">
<label>Display Name:</label>
<input type="text" value="Syllabus" class="page-display-name-input" data-metadata-name="display_name"/>
</div>
<div class="row">
<label>Page Content:</label>
<textarea class="page-contents"></textarea>
</div>
</article>
</div>
<div class="sidebar">
<div class="unit-settings window">
<h4 class="header">Page Settings</h4>
<div class="window-contents">
<div class="row visibility">
<label class="inline-label">Visibility:</label>
<select>
<option>Public</option>
<option>Private</option>
</select>
</div>
<div class="row unit-actions">
<a href="#" class="save-button">Save</a>
<a href="#" target="_blank" class="preview-button">Preview</a>
</div>
</div>
</div>
</div>
</div>
</div>
</%block>

13
cms/templates/editable_preview.html
View File

@@ -1,13 +0,0 @@
<section class='editable-preview'>
${content}
<div class="component-actions">
<a href="#" class="edit-button"><span class="edit-icon white"></span>Edit</a>
<a href="#" class="delete-button"><span class="delete-icon white"></span>Delete</a>
</div>
<a data-tooltip="Drag to reorder" href="#" class="drag-handle"></a>
<div class="component-editor">
<h5>Edit Video Component</h5>
<textarea class="component-source"><video youtube="1.50:q1xkuPsOY6Q,1.25:9WOY2dHz5i4,1.0:4rpg8Bq6hb4,0.75:KLim9Xkp7IY"/></textarea>
<a href="#" class="save-button">Save</a><a href="#" class="cancel-button">Cancel</a>
</div>
<section>

2
cms/templates/howitworks.html
View File

@@ -11,7 +11,7 @@
<section class="content content-header">
<header>
## "edX Studio" should not be translated
<h1>${_('Welcome to')}<span class="logo">&nbsp;edX Studio</span></h1>
<h1><span class="wrapper-text-welcome">${_('Welcome to')}</span><span class="logo">edX Studio</span></h1>
<p class="tagline">${_("Studio helps manage your courses online, so you can focus on teaching them")}</p>
</header>
</section>

2
cms/templates/js/edit-chapter.underscore
View File

@@ -9,6 +9,6 @@
<label for="chapter<%= order %>-asset-path"><%= gettext("Chapter Asset") %></label>
<input id="chapter<%= order %>-asset-path" name="chapter<%= order %>-asset-path" class="chapter-asset-path" placeholder="<%= _.str.sprintf(gettext("path/to/introductionToCookieBaking-CH%d.pdf"), order) %>" value="<%= asset_path %>" type="text">
<span class="tip tip-stacked"><%= gettext("upload a PDF file or provide the path to a Studio asset file") %></span>
<button class="action action-upload"><%= gettext("Upload Asset") %></button>
<button class="action action-upload"><%= gettext("Upload PDF") %></button>
</div>
<a href="" class="action action-close"><i class="icon-remove-sign"></i> <span class="sr"><%= gettext("delete chapter") %></span></a>

2
cms/templates/overview.html
View File

@@ -1,4 +1,4 @@
/<%! from django.utils.translation import ugettext as _ %>
<%! from django.utils.translation import ugettext as _ %>
<%inherit file="base.html" />
<%!
import logging

9
cms/templates/unit.html
View File

@@ -171,10 +171,15 @@
<div class="window unit-location">
<h4 class="header">${_("Unit Location")}</h4>
<div class="window-contents">
<div><input type="text" class="url" value="/courseware/${section.url_name}/${subsection.url_name}" disabled /></div>
<div class="row wrapper-unit-id">
<p class="unit-id">
<span class="label">${_("Unit Identifier:")}</span>
<input type="text" class="url value" value="${unit.location.name}" disabled />
</p>
</div>
<ol>
<li>
<a href="#" class="section-item">${section.display_name_with_default}</a>
<a href="${reverse('course_index', kwargs=dict(org=context_course.location.org, course=context_course.location.course, name=context_course.location.name))}" class="section-item">${section.display_name_with_default}</a>
<ol>
<li>
<a href="${reverse('edit_subsection', args=[subsection.location])}" class="section-item">

6
common/djangoapps/external_auth/models.py
View File

@@ -4,9 +4,9 @@ WE'RE USING MIGRATIONS!
If you make changes to this model, be sure to create an appropriate migration
file and check it in at the same time as your model changes. To do that,
1. Go to the mitx dir
2. django-admin.py schemamigration student --auto --settings=lms.envs.dev --pythonpath=. description_of_your_change
3. Add the migration file created in mitx/common/djangoapps/external_auth/migrations/
1. Go to the edx-platform dir
2. ./manage.py lms schemamigration student --auto description_of_your_change
3. Add the migration file created in edx-platform/common/djangoapps/external_auth/migrations/
"""
from django.db import models

4
common/djangoapps/heartbeat/views.py
View File

@@ -1,16 +1,18 @@
import json
from datetime import datetime
from pytz import UTC
from django.http import HttpResponse
from xmodule.modulestore.django import modulestore
from dogapi import dog_stats_api
@dog_stats_api.timed('edxapp.heartbeat')
def heartbeat(request):
"""
Simple view that a loadbalancer can check to verify that the app is up
"""
output = {
'date': datetime.now().isoformat(),
'date': datetime.now(UTC).isoformat(),
'courses': [course.location.url() for course in modulestore().get_courses()],
}
return HttpResponse(json.dumps(output, indent=4))

30
common/djangoapps/static_replace/__init__.py
View File

@@ -43,6 +43,35 @@ def try_staticfiles_lookup(path):
return url
def replace_jump_to_id_urls(text, course_id, jump_to_id_base_url):
"""
This will replace a link to another piece of courseware to a 'jump_to'
URL that will redirect to the right place in the courseware
NOTE: This is similar to replace_course_urls in terms of functionality
but it is intended to be used when we only have a 'id' that the
course author provides. This is much more helpful when using
Studio authored courses since they don't need to know the path. This
is also durable with respect to item moves.
text: The content over which to perform the subtitutions
course_id: The course_id in which this rewrite happens
jump_to_id_base_url:
A app-tier (e.g. LMS) absolute path to the base of the handler that will perform the
redirect. e.g. /courses/<org>/<course>/<run>/jump_to_id. NOTE the <id> will be appended to
the end of this URL at re-write time
output: <text> after the link rewriting rules are applied
"""
def replace_jump_to_id_url(match):
quote = match.group('quote')
rest = match.group('rest')
return "".join([quote, jump_to_id_base_url + rest, quote])
return re.sub(_url_replace_regex('/jump_to_id/'), replace_jump_to_id_url, text)
def replace_course_urls(text, course_id):
"""
Replace /course/$stuff urls with /courses/$course_id/$stuff urls
@@ -53,7 +82,6 @@ def replace_course_urls(text, course_id):
returns: text with the links replaced
"""
def replace_course_url(match):
quote = match.group('quote')
rest = match.group('rest')

2
common/djangoapps/student/management/commands/pearson_transfer.py
View File

@@ -20,7 +20,7 @@ class Command(BaseCommand):
files and then uploads over SFTP to Pearson and stuffs the entry in an
S3 bucket for archive purposes.
Usage: django-admin.py pearson-transfer --mode [import|export|both]
Usage: ./manage.py pearson-transfer --mode [import|export|both]
"""
option_list = BaseCommand.option_list + (

64
common/djangoapps/student/models.py
View File

@@ -6,9 +6,9 @@ Migration Notes
If you make changes to this model, be sure to create an appropriate migration
file and check it in at the same time as your model changes. To do that,
1. Go to the mitx dir
2. django-admin.py schemamigration student --auto --settings=lms.envs.dev --pythonpath=. description_of_your_change
3. Add the migration file created in mitx/common/djangoapps/student/migrations/
1. Go to the edx-platform dir
2. ./manage.py lms schemamigration student --auto description_of_your_change
3. Add the migration file created in edx-platform/common/djangoapps/student/migrations/
"""
from datetime import datetime
import hashlib
@@ -69,30 +69,33 @@ class UserProfile(models.Model):
location = models.CharField(blank=True, max_length=255, db_index=True)
# Optional demographic data we started capturing from Fall 2012
this_year = datetime.now().year
this_year = datetime.now(UTC).year
VALID_YEARS = range(this_year, this_year - 120, -1)
year_of_birth = models.IntegerField(blank=True, null=True, db_index=True)
GENDER_CHOICES = (('m', 'Male'), ('f', 'Female'), ('o', 'Other'))
gender = models.CharField(blank=True, null=True, max_length=6, db_index=True,
choices=GENDER_CHOICES)
gender = models.CharField(
blank=True, null=True, max_length=6, db_index=True, choices=GENDER_CHOICES
)
# [03/21/2013] removed these, but leaving comment since there'll still be
# p_se and p_oth in the existing data in db.
# ('p_se', 'Doctorate in science or engineering'),
# ('p_oth', 'Doctorate in another field'),
LEVEL_OF_EDUCATION_CHOICES = (('p', 'Doctorate'),
('m', "Master's or professional degree"),
('b', "Bachelor's degree"),
('a', "Associate's degree"),
('hs', "Secondary/high school"),
('jhs', "Junior secondary/junior high/middle school"),
('el', "Elementary/primary school"),
('none', "None"),
('other', "Other"))
LEVEL_OF_EDUCATION_CHOICES = (
('p', 'Doctorate'),
('m', "Master's or professional degree"),
('b', "Bachelor's degree"),
('a', "Associate's degree"),
('hs', "Secondary/high school"),
('jhs', "Junior secondary/junior high/middle school"),
('el', "Elementary/primary school"),
('none', "None"),
('other', "Other")
)
level_of_education = models.CharField(
blank=True, null=True, max_length=6, db_index=True,
choices=LEVEL_OF_EDUCATION_CHOICES
)
blank=True, null=True, max_length=6, db_index=True,
choices=LEVEL_OF_EDUCATION_CHOICES
)
mailing_address = models.TextField(blank=True, null=True)
goals = models.TextField(blank=True, null=True)
allow_certificate = models.BooleanField(default=1)
@@ -307,18 +310,18 @@ class TestCenterUserForm(ModelForm):
ACCOMMODATION_REJECTED_CODE = 'NONE'
ACCOMMODATION_CODES = (
(ACCOMMODATION_REJECTED_CODE, 'No Accommodation Granted'),
('EQPMNT', 'Equipment'),
('ET12ET', 'Extra Time - 1/2 Exam Time'),
('ET30MN', 'Extra Time - 30 Minutes'),
('ETDBTM', 'Extra Time - Double Time'),
('SEPRMM', 'Separate Room'),
('SRREAD', 'Separate Room and Reader'),
('SRRERC', 'Separate Room and Reader/Recorder'),
('SRRECR', 'Separate Room and Recorder'),
('SRSEAN', 'Separate Room and Service Animal'),
('SRSGNR', 'Separate Room and Sign Language Interpreter'),
)
(ACCOMMODATION_REJECTED_CODE, 'No Accommodation Granted'),
('EQPMNT', 'Equipment'),
('ET12ET', 'Extra Time - 1/2 Exam Time'),
('ET30MN', 'Extra Time - 30 Minutes'),
('ETDBTM', 'Extra Time - Double Time'),
('SEPRMM', 'Separate Room'),
('SRREAD', 'Separate Room and Reader'),
('SRRERC', 'Separate Room and Reader/Recorder'),
('SRRECR', 'Separate Room and Recorder'),
('SRSEAN', 'Separate Room and Service Animal'),
('SRSGNR', 'Separate Room and Sign Language Interpreter'),
)
ACCOMMODATION_CODE_DICT = {code: name for (code, name) in ACCOMMODATION_CODES}
@@ -572,7 +575,6 @@ class TestCenterRegistrationForm(ModelForm):
return code
def get_testcenter_registration(user, course_id, exam_series_code):
try:
tcu = TestCenterUser.objects.get(user=user)

8
common/djangoapps/student/views.py
View File

@@ -111,9 +111,9 @@ def get_date_for_press(publish_date):
# strip off extra months, and just use the first:
date = re.sub(multimonth_pattern, ", ", publish_date)
if re.search(day_pattern, date):
date = datetime.datetime.strptime(date, "%B %d, %Y")
date = datetime.datetime.strptime(date, "%B %d, %Y").replace(tzinfo=UTC)
else:
date = datetime.datetime.strptime(date, "%B, %Y")
date = datetime.datetime.strptime(date, "%B, %Y").replace(tzinfo=UTC)
return date
@@ -1100,7 +1100,7 @@ def confirm_email_change(request, key):
meta = up.get_meta()
if 'old_emails' not in meta:
meta['old_emails'] = []
meta['old_emails'].append([user.email, datetime.datetime.now().isoformat()])
meta['old_emails'].append([user.email, datetime.datetime.now(UTC).isoformat()])
up.set_meta(meta)
up.save()
# Send it to the old email...
@@ -1198,7 +1198,7 @@ def accept_name_change_by_id(id):
meta = up.get_meta()
if 'old_names' not in meta:
meta['old_names'] = []
meta['old_names'].append([up.name, pnc.rationale, datetime.datetime.now().isoformat()])
meta['old_names'].append([up.name, pnc.rationale, datetime.datetime.now(UTC).isoformat()])
up.set_meta(meta)
up.name = pnc.new_name

7
common/djangoapps/terrain/steps.py
View File

@@ -129,6 +129,13 @@ def should_have_link_with_id_and_text(step, link_id, text):
assert_equals(link.text, text)
@step(r'should see a link to "([^"]*)" with the text "([^"]*)"$')
def should_have_link_with_path_and_text(step, path, text):
link = world.browser.find_link_by_text(text)
assert len(link) > 0
assert_equals(link.first["href"], django_url(path))
@step(r'should( not)? see "(.*)" (?:somewhere|anywhere) (?:in|on) (?:the|this) page')
def should_see_in_the_page(step, doesnt_appear, text):
if doesnt_appear:

22
common/djangoapps/xmodule_modifiers.py
View File

@@ -42,6 +42,28 @@ def wrap_xmodule(get_html, module, template, context=None):
return _get_html
def replace_jump_to_id_urls(get_html, course_id, jump_to_id_base_url):
"""
This will replace a link between courseware in the format
/jump_to/<id> with a URL for a page that will correctly redirect
This is similar to replace_course_urls, but much more flexible and
durable for Studio authored courses. See more comments in static_replace.replace_jump_to_urls
course_id: The course_id in which this rewrite happens
jump_to_id_base_url:
A app-tier (e.g. LMS) absolute path to the base of the handler that will perform the
redirect. e.g. /courses/<org>/<course>/<run>/jump_to_id. NOTE the <id> will be appended to
the end of this URL at re-write time
output: a wrapped get_html() function pointer, which, when called, will apply the
rewrite rules
"""
@wraps(get_html)
def _get_html():
return static_replace.replace_jump_to_id_urls(get_html(), course_id, jump_to_id_base_url)
return _get_html
def replace_course_urls(get_html, course_id):
"""
Updates the supplied module with a new get_html function that wraps

68
common/lib/capa/capa/capa_problem.py
View File

@@ -32,6 +32,8 @@ import capa.xqueue_interface as xqueue_interface
import capa.responsetypes as responsetypes
from capa.safe_exec import safe_exec
from pytz import UTC
# dict of tagname, Response Class -- this should come from auto-registering
response_tag_dict = dict([(x.response_tag, x) for x in responsetypes.__all__])
@@ -42,13 +44,22 @@ solution_tags = ['solution']
response_properties = ["codeparam", "responseparam", "answer", "openendedparam"]
# special problem tags which should be turned into innocuous HTML
html_transforms = {'problem': {'tag': 'div'},
'text': {'tag': 'span'},
'math': {'tag': 'span'},
}
html_transforms = {
'problem': {'tag': 'div'},
'text': {'tag': 'span'},
'math': {'tag': 'span'},
}
# These should be removed from HTML output, including all subelements
html_problem_semantics = ["codeparam", "responseparam", "answer", "script", "hintgroup", "openendedparam", "openendedrubric"]
html_problem_semantics = [
"codeparam",
"responseparam",
"answer",
"script",
"hintgroup",
"openendedparam",
"openendedrubric"
]
log = logging.getLogger(__name__)
@@ -242,11 +253,15 @@ class LoncapaProblem(object):
return None
# Get a list of timestamps of all queueing requests, then convert it to a DateTime object
queuetime_strs = [self.correct_map.get_queuetime_str(answer_id)
for answer_id in self.correct_map
if self.correct_map.is_queued(answer_id)]
queuetimes = [datetime.strptime(qt_str, xqueue_interface.dateformat)
for qt_str in queuetime_strs]
queuetime_strs = [
self.correct_map.get_queuetime_str(answer_id)
for answer_id in self.correct_map
if self.correct_map.is_queued(answer_id)
]
queuetimes = [
datetime.strptime(qt_str, xqueue_interface.dateformat).replace(tzinfo=UTC)
for qt_str in queuetime_strs
]
return max(queuetimes)
@@ -404,10 +419,16 @@ class LoncapaProblem(object):
# open using ModuleSystem OSFS filestore
ifp = self.system.filestore.open(filename)
except Exception as err:
log.warning('Error %s in problem xml include: %s' % (
err, etree.tostring(inc, pretty_print=True)))
log.warning('Cannot find file %s in %s' % (
filename, self.system.filestore))
log.warning(
'Error %s in problem xml include: %s' % (
err, etree.tostring(inc, pretty_print=True)
)
)
log.warning(
'Cannot find file %s in %s' % (
filename, self.system.filestore
)
)
# if debugging, don't fail - just log error
# TODO (vshnayder): need real error handling, display to users
if not self.system.get('DEBUG'):
@@ -418,8 +439,11 @@ class LoncapaProblem(object):
# read in and convert to XML
incxml = etree.XML(ifp.read())
except Exception as err:
log.warning('Error %s in problem xml include: %s' % (
err, etree.tostring(inc, pretty_print=True)))
log.warning(
'Error %s in problem xml include: %s' % (
err, etree.tostring(inc, pretty_print=True)
)
)
log.warning('Cannot parse XML in %s' % (filename))
# if debugging, don't fail - just log error
# TODO (vshnayder): same as above
@@ -579,8 +603,9 @@ class LoncapaProblem(object):
# let each Response render itself
if problemtree in self.responders:
overall_msg = self.correct_map.get_overall_message()
return self.responders[problemtree].render_html(self._extract_html,
response_msg=overall_msg)
return self.responders[problemtree].render_html(
self._extract_html, response_msg=overall_msg
)
# let each custom renderer render itself:
if problemtree.tag in customrender.registry.registered_tags():
@@ -628,9 +653,10 @@ class LoncapaProblem(object):
answer_id = 1
input_tags = inputtypes.registry.registered_tags()
inputfields = tree.xpath("|".join(['//' + response.tag + '[@id=$id]//' + x
for x in (input_tags + solution_tags)]),
id=response_id_str)
inputfields = tree.xpath(
"|".join(['//' + response.tag + '[@id=$id]//' + x for x in (input_tags + solution_tags)]),
id=response_id_str
)
# assign one answer_id for each input type or solution type
for entry in inputfields:

34
common/lib/capa/capa/correctmap.py
View File

@@ -37,23 +37,27 @@ class CorrectMap(object):
return self.cmap.__iter__()
# See the documentation for 'set_dict' for the use of kwargs
def set(self,
answer_id=None,
correctness=None,
npoints=None,
msg='',
hint='',
hintmode=None,
queuestate=None, **kwargs):
def set(
self,
answer_id=None,
correctness=None,
npoints=None,
msg='',
hint='',
hintmode=None,
queuestate=None,
**kwargs
):
if answer_id is not None:
self.cmap[str(answer_id)] = {'correctness': correctness,
'npoints': npoints,
'msg': msg,
'hint': hint,
'hintmode': hintmode,
'queuestate': queuestate,
}
self.cmap[str(answer_id)] = {
'correctness': correctness,
'npoints': npoints,
'msg': msg,
'hint': hint,
'hintmode': hintmode,
'queuestate': queuestate,
}
def __repr__(self):
return repr(self.cmap)

27
common/lib/capa/capa/responsetypes.py
View File

@@ -33,6 +33,7 @@ from shapely.geometry import Point, MultiPoint
from calc import evaluator, UndefinedVariable
from . import correctmap
from datetime import datetime
from pytz import UTC
from .util import *
from lxml import etree
from lxml.html.soupparser import fromstring as fromstring_bs # uses Beautiful Soup!!! FIXME?
@@ -1365,9 +1366,11 @@ class CodeResponse(LoncapaResponse):
# Note that submission can be a file
submission = student_answers[self.answer_id]
except Exception as err:
log.error('Error in CodeResponse %s: cannot get student answer for %s;'
' student_answers=%s' %
(err, self.answer_id, convert_files_to_filenames(student_answers)))
log.error(
'Error in CodeResponse %s: cannot get student answer for %s;'
' student_answers=%s' %
(err, self.answer_id, convert_files_to_filenames(student_answers))
)
raise Exception(err)
# We do not support xqueue within Studio.
@@ -1381,19 +1384,20 @@ class CodeResponse(LoncapaResponse):
#------------------------------------------------------------
qinterface = self.system.xqueue['interface']
qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
anonymous_student_id = self.system.anonymous_student_id
# Generate header
queuekey = xqueue_interface.make_hashkey(str(self.system.seed) + qtime +
anonymous_student_id +
self.answer_id)
queuekey = xqueue_interface.make_hashkey(
str(self.system.seed) + qtime + anonymous_student_id + self.answer_id
)
callback_url = self.system.xqueue['construct_callback']()
xheader = xqueue_interface.make_xheader(
lms_callback_url=callback_url,
lms_key=queuekey,
queue_name=self.queue_name)
queue_name=self.queue_name
)
# Generate body
if is_list_of_files(submission):
@@ -1406,9 +1410,10 @@ class CodeResponse(LoncapaResponse):
# Metadata related to the student submission revealed to the external
# grader
student_info = {'anonymous_student_id': anonymous_student_id,
'submission_time': qtime,
}
student_info = {
'anonymous_student_id': anonymous_student_id,
'submission_time': qtime,
}
contents.update({'student_info': json.dumps(student_info)})
# Submit request. When successful, 'msg' is the prior length of the

17
common/lib/capa/capa/tests/test_responsetypes.py
View File

@@ -18,6 +18,8 @@ from capa.correctmap import CorrectMap
from capa.util import convert_files_to_filenames
from capa.xqueue_interface import dateformat
from pytz import UTC
class ResponseTest(unittest.TestCase):
""" Base class for tests of capa responses."""
@@ -333,8 +335,9 @@ class SymbolicResponseTest(ResponseTest):
correct_map = problem.grade_answers(input_dict)
self.assertEqual(correct_map.get_correctness('1_2_1'),
expected_correctness)
self.assertEqual(
correct_map.get_correctness('1_2_1'), expected_correctness
)
class OptionResponseTest(ResponseTest):
@@ -702,7 +705,7 @@ class CodeResponseTest(ResponseTest):
# Now we queue the LCP
cmap = CorrectMap()
for i, answer_id in enumerate(answer_ids):
queuestate = CodeResponseTest.make_queuestate(i, datetime.now())
queuestate = CodeResponseTest.make_queuestate(i, datetime.now(UTC))
cmap.update(CorrectMap(answer_id=answer_ids[i], queuestate=queuestate))
self.problem.correct_map.update(cmap)
@@ -718,7 +721,7 @@ class CodeResponseTest(ResponseTest):
old_cmap = CorrectMap()
for i, answer_id in enumerate(answer_ids):
queuekey = 1000 + i
queuestate = CodeResponseTest.make_queuestate(queuekey, datetime.now())
queuestate = CodeResponseTest.make_queuestate(queuekey, datetime.now(UTC))
old_cmap.update(CorrectMap(answer_id=answer_ids[i], queuestate=queuestate))
# Message format common to external graders
@@ -778,13 +781,15 @@ class CodeResponseTest(ResponseTest):
cmap = CorrectMap()
for i, answer_id in enumerate(answer_ids):
queuekey = 1000 + i
latest_timestamp = datetime.now()
latest_timestamp = datetime.now(UTC)
queuestate = CodeResponseTest.make_queuestate(queuekey, latest_timestamp)
cmap.update(CorrectMap(answer_id=answer_id, queuestate=queuestate))
self.problem.correct_map.update(cmap)
# Queue state only tracks up to second
latest_timestamp = datetime.strptime(datetime.strftime(latest_timestamp, dateformat), dateformat)
latest_timestamp = datetime.strptime(
datetime.strftime(latest_timestamp, dateformat), dateformat
).replace(tzinfo=UTC)
self.assertEquals(self.problem.get_recentmost_queuetime(), latest_timestamp)

19
common/lib/capa/capa/xqueue_interface.py
View File

@@ -30,9 +30,11 @@ def make_xheader(lms_callback_url, lms_key, queue_name):
'queue_name': designate a specific queue within xqueue server, e.g. 'MITx-6.00x' (string)
}
"""
return json.dumps({'lms_callback_url': lms_callback_url,
'lms_key': lms_key,
'queue_name': queue_name})
return json.dumps({
'lms_callback_url': lms_callback_url,
'lms_key': lms_key,
'queue_name': queue_name
})
def parse_xreply(xreply):
@@ -60,7 +62,7 @@ class XQueueInterface(object):
'''
def __init__(self, url, django_auth, requests_auth=None):
self.url = url
self.url = url
self.auth = django_auth
self.session = requests.session(auth=requests_auth)
@@ -95,13 +97,13 @@ class XQueueInterface(object):
return (error, msg)
def _login(self):
payload = {'username': self.auth['username'],
'password': self.auth['password']}
payload = {
'username': self.auth['username'],
'password': self.auth['password']
}
return self._http_post(self.url + '/xqueue/login/', payload)
def _send_to_queue(self, header, body, files_to_upload):
payload = {'xqueue_header': header,
'xqueue_body': body}
@@ -112,7 +114,6 @@ class XQueueInterface(object):
return self._http_post(self.url + '/xqueue/submit/', payload, files=files)
def _http_post(self, url, data, files=None):
try:
r = self.session.post(url, data=data, files=files)

40
common/lib/xmodule/xmodule/capa_module.py
View File

@@ -309,7 +309,13 @@ class CapaModule(CapaFields, XModule):
d = self.get_score()
score = d['score']
total = d['total']
if total > 0:
if self.weight is not None:
# scale score and total by weight/total:
score = score * self.weight / total
total = self.weight
try:
return Progress(score, total)
except (TypeError, ValueError):
@@ -321,11 +327,13 @@ class CapaModule(CapaFields, XModule):
"""
Return some html with data about the module
"""
progress = self.get_progress()
return self.system.render_template('problem_ajax.html', {
'element_id': self.location.html_id(),
'id': self.id,
'ajax_url': self.system.ajax_url,
'progress': Progress.to_js_status_str(self.get_progress())
'progress_status': Progress.to_js_status_str(progress),
'progress_detail': Progress.to_js_detail_str(progress),
})
def check_button_name(self):
@@ -485,8 +493,7 @@ class CapaModule(CapaFields, XModule):
"""
Return html for the problem.
Adds check, reset, save buttons as necessary based on the problem config
and state.
Adds check, reset, save buttons as necessary based on the problem config and state.
"""
try:
@@ -516,13 +523,12 @@ class CapaModule(CapaFields, XModule):
'reset_button': self.should_show_reset_button(),
'save_button': self.should_show_save_button(),
'answer_available': self.answer_available(),
'ajax_url': self.system.ajax_url,
'attempts_used': self.attempts,
'attempts_allowed': self.max_attempts,
'progress': self.get_progress(),
}
html = self.system.render_template('problem.html', context)
if encapsulate:
html = u'<div id="problem_{id}" class="problem" data-url="{ajax_url}">'.format(
id=self.location.html_id(), ajax_url=self.system.ajax_url
@@ -584,6 +590,7 @@ class CapaModule(CapaFields, XModule):
result.update({
'progress_changed': after != before,
'progress_status': Progress.to_js_status_str(after),
'progress_detail': Progress.to_js_detail_str(after),
})
return json.dumps(result, cls=ComplexEncoder)
@@ -614,6 +621,7 @@ class CapaModule(CapaFields, XModule):
Problem can be completely wrong.
Pressing RESET button makes this function to return False.
"""
# used by conditional module
return self.lcp.done
def is_attempted(self):
@@ -757,6 +765,7 @@ class CapaModule(CapaFields, XModule):
"""
return {'html': self.get_problem_html(encapsulate=False)}
@staticmethod
def make_dict_of_responses(data):
"""
@@ -1117,8 +1126,12 @@ class CapaDescriptor(CapaFields, RawDescriptor):
mako_template = "widgets/problem-edit.html"
js = {'coffee': [resource_string(__name__, 'js/src/problem/edit.coffee')]}
js_module_name = "MarkdownEditingDescriptor"
css = {'scss': [resource_string(__name__, 'css/editor/edit.scss'),
resource_string(__name__, 'css/problem/edit.scss')]}
css = {
'scss': [
resource_string(__name__, 'css/editor/edit.scss'),
resource_string(__name__, 'css/problem/edit.scss')
]
}
# Capa modules have some additional metadata:
# TODO (vshnayder): do problems have any other metadata? Do they
@@ -1146,19 +1159,6 @@ class CapaDescriptor(CapaFields, RawDescriptor):
path[8:],
]
@classmethod
def from_xml(cls, xml_data, system, org=None, course=None):
"""
Augment regular translation w/ setting the pre-Studio defaults.
"""
problem = super(CapaDescriptor, cls).from_xml(xml_data, system, org, course)
# pylint: disable=W0212
if 'showanswer' not in problem._model_data:
problem.showanswer = "closed"
if 'rerandomize' not in problem._model_data:
problem.rerandomize = "always"
return problem
@property
def non_editable_metadata_fields(self):
non_editable_fields = super(CapaDescriptor, self).non_editable_metadata_fields

13
common/lib/xmodule/xmodule/course_module.py
View File

@@ -15,6 +15,7 @@ import json
from xblock.core import Scope, List, String, Dict, Boolean
from .fields import Date
from xmodule.modulestore.locator import CourseLocator
from django.utils.timezone import UTC
from xmodule.util import date_utils
@@ -192,9 +193,8 @@ class CourseFields(object):
}},
scope=Scope.content)
show_calculator = Boolean(help="Whether to show the calculator in this course", default=False, scope=Scope.settings)
display_name = String(
help="Display name for this module", default="Empty",
display_name="Display Name", scope=Scope.settings)
display_name = String(help="Display name for this module", default="Empty", display_name="Display Name", scope=Scope.settings)
show_chat = Boolean(help="Whether to show the chat widget in this course", default=False, scope=Scope.settings)
tabs = List(help="List of tabs to enable in this course", scope=Scope.settings)
end_of_course_survey_url = String(help="Url for the end-of-course survey", scope=Scope.settings)
discussion_blackouts = List(help="List of pairs of start/end dates for discussion blackouts", scope=Scope.settings)
@@ -373,7 +373,10 @@ class CourseDescriptor(CourseFields, SequenceDescriptor):
super(CourseDescriptor, self).__init__(*args, **kwargs)
if self.wiki_slug is None:
self.wiki_slug = self.location.course
if isinstance(self.location, Location):
self.wiki_slug = self.location.course
elif isinstance(self.location, CourseLocator):
self.wiki_slug = self.location.course_id or self.display_name
msg = None
@@ -407,7 +410,7 @@ class CourseDescriptor(CourseFields, SequenceDescriptor):
continue
# TODO check that this is still needed here and can't be by defaults.
if self.tabs is None:
if not self.tabs:
# When calling the various _tab methods, can omit the 'type':'blah' from the
# first arg, since that's only used for dispatch
tabs = []

8
common/lib/xmodule/xmodule/css/capa/display.scss
View File

@@ -3,6 +3,7 @@ h2 {
margin-bottom: 15px;
&.problem-header {
display: inline-block;
section.staff {
margin-top: 30px;
font-size: 80%;
@@ -28,6 +29,13 @@ iframe[seamless]{
color: darken($error-red, 11%);
}
section.problem-progress {
display: inline-block;
color: #999;
font-size: em(16);
font-weight: 100;
padding-left: 5px;
}
section.problem {
@media print {

3
common/lib/xmodule/xmodule/css/sequence/display.scss
View File

@@ -211,6 +211,8 @@ nav.sequence-nav {
@include transition(all .1s $ease-in-out-quart 0s);
white-space: pre;
z-index: 99;
visibility: hidden;
pointer-events: none;
&:empty {
background: none;
@@ -238,6 +240,7 @@ nav.sequence-nav {
display: block;
margin-top: 4px;
opacity: 1.0;
visibility: visible;
}
}
}

12
common/lib/xmodule/xmodule/discussion_module.py
View File

@@ -12,10 +12,14 @@ class DiscussionFields(object):
display_name = String(
display_name="Display Name",
help="Display name for this module",
default="Discussion Tag",
scope=Scope.settings)
data = String(help="XML data for the problem", scope=Scope.content,
default="<discussion></discussion>")
default="Discussion",
scope=Scope.settings
)
data = String(
help="XML data for the problem",
scope=Scope.content,
default="<discussion></discussion>"
)
discussion_category = String(
display_name="Category",
default="Week 1",

8
common/lib/xmodule/xmodule/error_module.py
View File

@@ -79,8 +79,10 @@ class ErrorDescriptor(ErrorFields, JSONEditingDescriptor):
@classmethod
def _construct(cls, system, contents, error_msg, location):
if location.name is None:
location = location._replace(
if isinstance(location, dict) and 'course' in location:
location = Location(location)
if isinstance(location, Location) and location.name is None:
location = location.replace(
category='error',
# Pick a unique url_name -- the sha1 hash of the contents.
# NOTE: We could try to pull out the url_name of the errored descriptor,
@@ -94,7 +96,7 @@ class ErrorDescriptor(ErrorFields, JSONEditingDescriptor):
model_data = {
'error_msg': str(error_msg),
'contents': contents,
'display_name': 'Error: ' + location.name,
'display_name': 'Error: ' + location.url(),
'location': location,
'category': 'error'
}

1
common/lib/xmodule/xmodule/fields.py
View File

@@ -80,6 +80,7 @@ class Date(ModelType):
TIMEDELTA_REGEX = re.compile(r'^((?P<days>\d+?) day(?:s?))?(\s)?((?P<hours>\d+?) hour(?:s?))?(\s)?((?P<minutes>\d+?) minute(?:s)?)?(\s)?((?P<seconds>\d+?) second(?:s)?)?$')
class Timedelta(ModelType):
def from_json(self, time_str):
"""

9
common/lib/xmodule/xmodule/foldit_module.py
View File

@@ -91,15 +91,18 @@ class FolditModule(FolditFields, XModule):
PuzzleComplete.completed_puzzles(self.system.anonymous_student_id),
key=lambda d: (d['set'], d['subset']))
def puzzle_leaders(self, n=10):
def puzzle_leaders(self, n=10, courses=None):
"""
Returns a list of n pairs (user, score) corresponding to the top
scores; the pairs are in descending order of score.
"""
from foldit.models import Score
leaders = [(e['username'], e['score']) for e in Score.get_tops_n(10)]
leaders.sort(key=lambda x:-x[1])
if courses is None:
courses = [self.location.course_id]
leaders = [(leader['username'], leader['score']) for leader in Score.get_tops_n(10, course_list=courses)]
leaders.sort(key=lambda x: -x[1])
return leaders

39
common/lib/xmodule/xmodule/gst_module.py
View File

@@ -19,10 +19,45 @@ from xblock.core import String, Scope
log = logging.getLogger(__name__)
DEFAULT_RENDER="""
<h2>Graphic slider tool: Dynamic range and implicit functions.</h2>
<p>You can make the range of the x axis (but not ticks of x axis) of
functions depend on a parameter value. This can be useful when the
function domain needs to be variable.</p>
<p>Implicit functions like a circle can be plotted as 2 separate
functions of the same color.</p>
<div style="height:50px;">
<slider var='r' style="width:400px;float:left;"/>
<textbox var='r' style="float:left;width:60px;margin-left:15px;"/>
</div>
<plot style="margin-top:15px;margin-bottom:15px;"/>
"""
DEFAULT_CONFIGURATION="""
<parameters>
<param var="r" min="5" max="25" step="0.5" initial="12.5" />
</parameters>
<functions>
<function color="red">Math.sqrt(r * r - x * x)</function>
<function color="red">-Math.sqrt(r * r - x * x)</function>
</functions>
<plot>
<xrange>
<!-- dynamic range -->
<min>-r</min>
<max>r</max>
</xrange>
<num_points>1000</num_points>
<xticks>-30, 6, 30</xticks>
<yticks>-30, 6, 30</yticks>
</plot>
"""
class GraphicalSliderToolFields(object):
render = String(scope=Scope.content)
configuration = String(scope=Scope.content)
render = String(scope=Scope.content, default=DEFAULT_RENDER)
configuration = String(scope=Scope.content, default=DEFAULT_CONFIGURATION)
class GraphicalSliderToolModule(GraphicalSliderToolFields, XModule):

4
common/lib/xmodule/xmodule/html_module.py
View File

@@ -25,9 +25,9 @@ class HtmlFields(object):
scope=Scope.settings,
# it'd be nice to have a useful default but it screws up other things; so,
# use display_name_with_default for those
default="Blank HTML Page"
default="Text"
)
data = String(help="Html contents to display for this module", default="", scope=Scope.content)
data = String(help="Html contents to display for this module", default=u"", scope=Scope.content)
source_code = String(help="Source code for LaTeX documents. This feature is not well-supported.", scope=Scope.settings)

2
common/lib/xmodule/xmodule/js/fixtures/problem.html
View File

@@ -1,6 +1,6 @@
<section class='xmodule_display xmodule_CapaModule' data-type='Problem'>
<section id='problem_1'
class='problems-wrapper'
class='problems-wrapper'
data-problem-id='i4x://edX/101/problem/Problem1'
data-url='/problem/Problem1'>
</section>

3
common/lib/xmodule/xmodule/js/fixtures/problem_content.html
View File

@@ -1,5 +1,8 @@
<h2 class="problem-header">Problem Header</h2>
<section class='problem-progress'>
</section>
<section class="problem">
<p>Problem Content</p>

19
common/lib/xmodule/xmodule/js/spec/capa/display_spec.coffee
View File

@@ -77,6 +77,25 @@ describe 'Problem', ->
[@problem.updateMathML, @stubbedJax, $('#input_example_1').get(0)]
]
describe 'renderProgressState', ->
beforeEach ->
@problem = new Problem($('.xmodule_display'))
#@renderProgressState = @problem.renderProgressState
describe 'with a status of "none"', ->
it 'reports the number of points possible', ->
@problem.el.data('progress_status', 'none')
@problem.el.data('progress_detail', '0/1')
@problem.renderProgressState()
expect(@problem.$('.problem-progress').html()).toEqual "(1 point possible)"
describe 'with any other valid status', ->
it 'reports the current score', ->
@problem.el.data('progress_status', 'foo')
@problem.el.data('progress_detail', '1/1')
@problem.renderProgressState()
expect(@problem.$('.problem-progress').html()).toEqual "(1/1 points)"
describe 'render', ->
beforeEach ->
@problem = new Problem($('.xmodule_display'))

27
common/lib/xmodule/xmodule/js/src/capa/display.coffee
View File

@@ -35,15 +35,34 @@ class @Problem
@$('input.math').each (index, element) =>
MathJax.Hub.Queue [@refreshMath, null, element]
renderProgressState: =>
detail = @el.data('progress_detail')
status = @el.data('progress_status')
# i18n
progress = "(#{detail} points)"
if status == 'none' and detail? and detail.indexOf('/') > 0
a = detail.split('/')
possible = parseInt(a[1])
if possible == 1
# i18n
progress = "(#{possible} point possible)"
else
# i18n
progress = "(#{possible} points possible)"
@$('.problem-progress').html(progress)
updateProgress: (response) =>
if response.progress_changed
@el.attr progress: response.progress_status
@el.data('progress_status', response.progress_status)
@el.data('progress_detail', response.progress_detail)
@el.trigger('progressChanged')
@renderProgressState()
forceUpdate: (response) =>
@el.attr progress: response.progress_status
@el.data('progress_status', response.progress_status)
@el.data('progress_detail', response.progress_detail)
@el.trigger('progressChanged')
@renderProgressState()
queueing: =>
@queued_items = @$(".xqueue")
@@ -113,7 +132,7 @@ class @Problem
@setupInputTypes()
@bind()
@queueing()
@forceUpdate response
# TODO add hooks for problem types here by inspecting response.html and doing
# stuff if a div w a class is found

2
common/lib/xmodule/xmodule/js/src/sequence/display.coffee
View File

@@ -45,7 +45,7 @@ class @Sequence
new_progress = "NA"
_this = this
$('.problems-wrapper').each (index) ->
progress = $(this).attr 'progress'
progress = $(this).data 'progress_status'
new_progress = _this.mergeProgress progress, new_progress
@progressTable[@position] = new_progress

18
common/lib/xmodule/xmodule/modulestore/exceptions.py
View File

@@ -7,10 +7,18 @@ class ItemNotFoundError(Exception):
pass
class ItemWriteConflictError(Exception):
pass
class InsufficientSpecificationError(Exception):
pass
class OverSpecificationError(Exception):
pass
class InvalidLocationError(Exception):
pass
@@ -21,3 +29,13 @@ class NoPathToItem(Exception):
class DuplicateItemError(Exception):
pass
class VersionConflictError(Exception):
"""
The caller asked for either draft or published head and gave a version which conflicted with it.
"""
def __init__(self, requestedLocation, currentHead):
super(VersionConflictError, self).__init__()
self.requestedLocation = requestedLocation
self.currentHead = currentHead

2
common/lib/xmodule/xmodule/modulestore/inheritance.py
View File

@@ -50,6 +50,8 @@ def inherit_metadata(descriptor, model_data):
def own_metadata(module):
# IN SPLIT MONGO this is just ['metadata'] as it keeps ['_inherited_metadata'] separate!
# FIXME move into kvs? will that work for xml mongo?
"""
Return a dictionary that contains only non-inherited field keys,
mapped to their values

465
common/lib/xmodule/xmodule/modulestore/locator.py Normal file
View File

@@ -0,0 +1,465 @@
"""
Created on Mar 13, 2013
@author: dmitchell
"""
from __future__ import absolute_import
import logging
import inspect
from abc import ABCMeta, abstractmethod
from urllib import quote
from bson.objectid import ObjectId
from bson.errors import InvalidId
from xmodule.modulestore.exceptions import InsufficientSpecificationError, OverSpecificationError
from .parsers import parse_url, parse_course_id, parse_block_ref
log = logging.getLogger(__name__)
class Locator(object):
"""
A locator is like a URL, it refers to a course resource.
Locator is an abstract base class: do not instantiate
"""
__metaclass__ = ABCMeta
@abstractmethod
def url(self):
"""
Return a string containing the URL for this location. Raises
InsufficientSpecificationError if the instance doesn't have a
complete enough specification to generate a url
"""
raise InsufficientSpecificationError()
def quoted_url(self):
return quote(self.url(), '@;#')
def __eq__(self, other):
return self.__dict__ == other.__dict__
def __repr__(self):
'''
repr(self) returns something like this: CourseLocator("edu.mit.eecs.6002x")
'''
classname = self.__class__.__name__
if classname.find('.') != -1:
classname = classname.split['.'][-1]
return '%s("%s")' % (classname, unicode(self))
def __str__(self):
'''
str(self) returns something like this: "edu.mit.eecs.6002x"
'''
return unicode(self).encode('utf8')
def __unicode__(self):
'''
unicode(self) returns something like this: "edu.mit.eecs.6002x"
'''
return self.url()
@abstractmethod
def version(self):
"""
Returns the ObjectId referencing this specific location.
Raises InsufficientSpecificationError if the instance
doesn't have a complete enough specification.
"""
raise InsufficientSpecificationError()
def set_property(self, property_name, new):
"""
Initialize property to new value.
If property has already been initialized to a different value, raise an exception.
"""
current = getattr(self, property_name)
if current and current != new:
raise OverSpecificationError('%s cannot be both %s and %s' %
(property_name, current, new))
setattr(self, property_name, new)
class CourseLocator(Locator):
"""
Examples of valid CourseLocator specifications:
CourseLocator(version_guid=ObjectId('519665f6223ebd6980884f2b'))
CourseLocator(course_id='edu.mit.eecs.6002x')
CourseLocator(course_id='edu.mit.eecs.6002x;published')
CourseLocator(course_id='edu.mit.eecs.6002x', revision='published')
CourseLocator(url='edx://@519665f6223ebd6980884f2b')
CourseLocator(url='edx://edu.mit.eecs.6002x')
CourseLocator(url='edx://edu.mit.eecs.6002x;published')
Should have at lease a specific course_id (id for the course as if it were a project w/
versions) with optional 'revision' (must be 'draft', 'published', or None),
or version_guid (which points to a specific version). Can contain both in which case
the persistence layer may raise exceptions if the given version != the current such version
of the course.
"""
# Default values
version_guid = None
course_id = None
revision = None
def __unicode__(self):
"""
Return a string representing this location.
"""
if self.course_id:
result = self.course_id
if self.revision:
result += ';' + self.revision
return result
elif self.version_guid:
return '@' + str(self.version_guid)
else:
# raise InsufficientSpecificationError("missing course_id or version_guid")
return '<InsufficientSpecificationError: missing course_id or version_guid>'
def url(self):
"""
Return a string containing the URL for this location.
"""
return 'edx://' + unicode(self)
# -- unused args which are used via inspect
# pylint: disable= W0613
def validate_args(self, url, version_guid, course_id, revision):
"""
Validate provided arguments.
"""
need_oneof = set(('url', 'version_guid', 'course_id'))
args, _, _, values = inspect.getargvalues(inspect.currentframe())
provided_args = [a for a in args if a != 'self' and values[a] is not None]
if len(need_oneof.intersection(provided_args)) == 0:
raise InsufficientSpecificationError("Must provide one of these args: %s " %
list(need_oneof))
def is_fully_specified(self):
"""
Returns True if either version_guid is specified, or course_id+revision
are specified.
This should always return True, since this should be validated in the constructor.
"""
return self.version_guid is not None \
or (self.course_id is not None and self.revision is not None)
def set_course_id(self, new):
"""
Initialize course_id to new value.
If course_id has already been initialized to a different value, raise an exception.
"""
self.set_property('course_id', new)
def set_revision(self, new):
"""
Initialize revision to new value.
If revision has already been initialized to a different value, raise an exception.
"""
self.set_property('revision', new)
def set_version_guid(self, new):
"""
Initialize version_guid to new value.
If version_guid has already been initialized to a different value, raise an exception.
"""
self.set_property('version_guid', new)
def as_course_locator(self):
"""
Returns a copy of itself (downcasting) as a CourseLocator.
The copy has the same CourseLocator fields as the original.
The copy does not include subclass information, such as
a usage_id (a property of BlockUsageLocator).
"""
return CourseLocator(course_id=self.course_id,
version_guid=self.version_guid,
revision=self.revision)
def __init__(self, url=None, version_guid=None, course_id=None, revision=None):
"""
Construct a CourseLocator
Caller may provide url (but no other parameters).
Caller may provide version_guid (but no other parameters).
Caller may provide course_id (optionally provide revision).
Resulting CourseLocator will have either a version_guid property
or a course_id (with optional revision) property, or both.
version_guid must be an instance of bson.objectid.ObjectId or None
url, course_id, and revision must be strings or None
"""
self.validate_args(url, version_guid, course_id, revision)
if url:
self.init_from_url(url)
if version_guid:
self.init_from_version_guid(version_guid)
if course_id or revision:
self.init_from_course_id(course_id, revision)
assert self.version_guid or self.course_id, \
"Either version_guid or course_id should be set."
@classmethod
def as_object_id(cls, value):
"""
Attempts to cast value as a bson.objectid.ObjectId.
If cast fails, raises ValueError
"""
if isinstance(value, ObjectId):
return value
try:
return ObjectId(value)
except InvalidId:
raise ValueError('"%s" is not a valid version_guid' % value)
def init_from_url(self, url):
"""
url must be a string beginning with 'edx://' and containing
either a valid version_guid or course_id (with optional revision)
If a block ('#HW3') is present, it is ignored.
"""
if isinstance(url, Locator):
url = url.url()
assert isinstance(url, basestring), \
'%s is not an instance of basestring' % url
parse = parse_url(url)
assert parse, 'Could not parse "%s" as a url' % url
if 'version_guid' in parse:
new_guid = parse['version_guid']
self.set_version_guid(self.as_object_id(new_guid))
else:
self.set_course_id(parse['id'])
self.set_revision(parse['revision'])
def init_from_version_guid(self, version_guid):
"""
version_guid must be an instance of bson.objectid.ObjectId,
or able to be cast as one.
If it's a string, attempt to cast it as an ObjectId first.
"""
version_guid = self.as_object_id(version_guid)
assert isinstance(version_guid, ObjectId), \
'%s is not an instance of ObjectId' % version_guid
self.set_version_guid(version_guid)
def init_from_course_id(self, course_id, explicit_revision=None):
"""
Course_id is a string like 'edu.mit.eecs.6002x' or 'edu.mit.eecs.6002x;published'.
Revision (optional) is a string like 'published'.
It may be provided explicitly (explicit_revision) or embedded into course_id.
If revision is part of course_id ("...;published"), parse it out separately.
If revision is provided both ways, that's ok as long as they are the same value.
If a block ('#HW3') is a part of course_id, it is ignored.
"""
if course_id:
if isinstance(course_id, CourseLocator):
course_id = course_id.course_id
assert course_id, "%s does not have a valid course_id"
parse = parse_course_id(course_id)
assert parse, 'Could not parse "%s" as a course_id' % course_id
self.set_course_id(parse['id'])
rev = parse['revision']
if rev:
self.set_revision(rev)
if explicit_revision:
self.set_revision(explicit_revision)
def version(self):
"""
Returns the ObjectId referencing this specific location.
"""
return self.version_guid
def html_id(self):
"""
Generate a discussion group id based on course
To make compatible with old Location object functionality. I don't believe this behavior fits at this
place, but I have no way to override. If this is really needed, it should probably use the pretty_id to seed
the name although that's mutable. We should also clearly define the purpose and restrictions of this
(e.g., I'm assuming periods are fine).
"""
return self.course_id
class BlockUsageLocator(CourseLocator):
"""
Encodes a location.
Locations address modules (aka blocks) which are definitions situated in a
course instance. Thus, a Location must identify the course and the occurrence of
the defined element in the course. Courses can be a version of an offering, the
current draft head, or the current production version.
Locators can contain both a version and a course_id w/ revision. The split mongo functions
may raise errors if these conflict w/ the current db state (i.e., the course's revision !=
the version_guid)
Locations can express as urls as well as dictionaries. They consist of
course_identifier: course_guid | version_guid
block : guid
revision : 'draft' | 'published' (optional)
"""
# Default value
usage_id = None
def __init__(self, url=None, version_guid=None, course_id=None,
revision=None, usage_id=None):
"""
Construct a BlockUsageLocator
Caller may provide url, version_guid, or course_id, and optionally provide revision.
The usage_id may be specified, either explictly or as part of
the url or course_id. If omitted, the locator is created but it
has not yet been initialized.
Resulting BlockUsageLocator will have a usage_id property.
It will have either a version_guid property or a course_id (with optional revision) property, or both.
version_guid must be an instance of bson.objectid.ObjectId or None
url, course_id, revision, and usage_id must be strings or None
"""
self.validate_args(url, version_guid, course_id, revision)
if url:
self.init_block_ref_from_url(url)
if course_id:
self.init_block_ref_from_course_id(course_id)
if usage_id:
self.init_block_ref(usage_id)
CourseLocator.__init__(self,
url=url,
version_guid=version_guid,
course_id=course_id,
revision=revision)
def is_initialized(self):
"""
Returns True if usage_id has been initialized, else returns False
"""
return self.usage_id is not None
def version_agnostic(self):
"""
Returns a copy of itself.
If both version_guid and course_id are known, use a blank course_id in the copy.
We don't care if the locator's version is not the current head; so, avoid version conflict
by reducing info.
:param block_locator:
"""
if self.course_id and self.version_guid:
return BlockUsageLocator(version_guid=self.version_guid,
revision=self.revision,
usage_id=self.usage_id)
else:
return BlockUsageLocator(course_id=self.course_id,
revision=self.revision,
usage_id=self.usage_id)
def set_usage_id(self, new):
"""
Initialize usage_id to new value.
If usage_id has already been initialized to a different value, raise an exception.
"""
self.set_property('usage_id', new)
def init_block_ref(self, block_ref):
parse = parse_block_ref(block_ref)
assert parse, 'Could not parse "%s" as a block_ref' % block_ref
self.set_usage_id(parse['block'])
def init_block_ref_from_url(self, url):
if isinstance(url, Locator):
url = url.url()
parse = parse_url(url)
assert parse, 'Could not parse "%s" as a url' % url
block = parse.get('block', None)
if block:
self.set_usage_id(block)
def init_block_ref_from_course_id(self, course_id):
if isinstance(course_id, CourseLocator):
course_id = course_id.course_id
assert course_id, "%s does not have a valid course_id"
parse = parse_course_id(course_id)
assert parse, 'Could not parse "%s" as a course_id' % course_id
block = parse.get('block', None)
if block:
self.set_usage_id(block)
def __unicode__(self):
"""
Return a string representing this location.
"""
rep = CourseLocator.__unicode__(self)
if self.usage_id is None:
# usage_id has not been initialized
return rep + '#NONE'
else:
return rep + '#' + self.usage_id
class DescriptionLocator(Locator):
"""
Container for how to locate a description
"""
def __init__(self, definition_id):
self.definition_id = definition_id
def __unicode__(self):
'''
Return a string representing this location.
unicode(self) returns something like this: "@519665f6223ebd6980884f2b"
'''
return '@' + str(self.definition_guid)
def url(self):
"""
Return a string containing the URL for this location.
url(self) returns something like this: 'edx://@519665f6223ebd6980884f2b'
"""
return 'edx://' + unicode(self)
def version(self):
"""
Returns the ObjectId referencing this specific location.
"""
return self.definition_guid
class VersionTree(object):
"""
Holds trees of Locators to represent version histories.
"""
def __init__(self, locator, tree_dict=None):
"""
:param locator: must be version specific (Course has version_guid or definition had id)
"""
assert isinstance(locator, Locator) and not inspect.isabstract(locator), \
"locator must be a concrete subclass of Locator"
assert locator.version(), \
"locator must be version specific (Course has version_guid or definition had id)"
self.locator = locator
if tree_dict is None:
self.children = []
else:
self.children = [VersionTree(child, tree_dict)
for child in tree_dict.get(locator.version(), [])]

9
common/lib/xmodule/xmodule/modulestore/mongo/base.py
View File

@@ -105,15 +105,6 @@ class MongoKeyValueStore(KeyValueStore):
else:
raise InvalidScopeError(key.scope)
def set_many(self, update_dict):
"""set_many method. Implementations should accept an `update_dict` of
key-value pairs, and set all the `keys` to the given `value`s."""
# `set` simply updates an in-memory db, rather than calling down to a real db,
# as mongo bulk save is handled elsewhere. A future improvement would be to pull
# the mongo-specific bulk save logic into this method.
for key, value in update_dict.iteritems():
self.set(key, value)
def delete(self, key):
if key.scope == Scope.children:
self._children = []

115
common/lib/xmodule/xmodule/modulestore/parsers.py Normal file
View File

@@ -0,0 +1,115 @@
import re
URL_RE = re.compile(r'^edx://(.+)$', re.IGNORECASE)
def parse_url(string):
"""
A url must begin with 'edx://' (case-insensitive match),
followed by either a version_guid or a course_id.
Examples:
'edx://@0123FFFF'
'edx://edu.mit.eecs.6002x'
'edx://edu.mit.eecs.6002x;published'
'edx://edu.mit.eecs.6002x;published#HW3'
This returns None if string cannot be parsed.
If it can be parsed as a version_guid, returns a dict
with key 'version_guid' and the value,
If it can be parsed as a course_id, returns a dict
with keys 'id' and 'revision' (value of 'revision' may be None),
"""
match = URL_RE.match(string)
if not match:
return None
path = match.group(1)
if path[0] == '@':
return parse_guid(path[1:])
return parse_course_id(path)
BLOCK_RE = re.compile(r'^\w+$', re.IGNORECASE)
def parse_block_ref(string):
r"""
A block_ref is a string of word_chars.
<word_chars> matches one or more Unicode word characters; this includes most
characters that can be part of a word in any language, as well as numbers
and the underscore. (see definition of \w in python regular expressions,
at http://docs.python.org/dev/library/re.html)
If string is a block_ref, returns a dict with key 'block_ref' and the value,
otherwise returns None.
"""
if len(string) > 0 and BLOCK_RE.match(string):
return {'block': string}
return None
GUID_RE = re.compile(r'^(?P<version_guid>[A-F0-9]+)(#(?P<block>\w+))?$', re.IGNORECASE)
def parse_guid(string):
"""
A version_guid is a string of hex digits (0-F).
If string is a version_guid, returns a dict with key 'version_guid' and the value,
otherwise returns None.
"""
m = GUID_RE.match(string)
if m is not None:
return m.groupdict()
else:
return None
COURSE_ID_RE = re.compile(r'^(?P<id>(\w+)(\.\w+\w*)*)(;(?P<revision>\w+))?(#(?P<block>\w+))?$', re.IGNORECASE)
def parse_course_id(string):
r"""
A course_id has a main id component.
There may also be an optional revision (;published or ;draft).
There may also be an optional block (#HW3 or #Quiz2).
Examples of valid course_ids:
'edu.mit.eecs.6002x'
'edu.mit.eecs.6002x;published'
'edu.mit.eecs.6002x#HW3'
'edu.mit.eecs.6002x;published#HW3'
Syntax:
course_id = main_id [; revision] [# block]
main_id = name [. name]*
revision = name
block = name
name = <word_chars>
<word_chars> matches one or more Unicode word characters; this includes most
characters that can be part of a word in any language, as well as numbers
and the underscore. (see definition of \w in python regular expressions,
at http://docs.python.org/dev/library/re.html)
If string is a course_id, returns a dict with keys 'id', 'revision', and 'block'.
Revision is optional: if missing returned_dict['revision'] is None.
Block is optional: if missing returned_dict['block'] is None.
Else returns None.
"""
match = COURSE_ID_RE.match(string)
if not match:
return None
return match.groupdict()

1
common/lib/xmodule/xmodule/modulestore/split_mongo/__init__.py Normal file
View File

@@ -0,0 +1 @@
from split import SplitMongoModuleStore

119
common/lib/xmodule/xmodule/modulestore/split_mongo/caching_descriptor_system.py Normal file
View File

@@ -0,0 +1,119 @@
import sys
import logging
from xmodule.mako_module import MakoDescriptorSystem
from xmodule.x_module import XModuleDescriptor
from xmodule.modulestore.locator import BlockUsageLocator
from xmodule.error_module import ErrorDescriptor
from xmodule.errortracker import exc_info_to_str
from xblock.runtime import DbModel
from ..exceptions import ItemNotFoundError
from .split_mongo_kvs import SplitMongoKVS, SplitMongoKVSid
log = logging.getLogger(__name__)
# TODO should this be here or w/ x_module or ???
class CachingDescriptorSystem(MakoDescriptorSystem):
"""
A system that has a cache of a course version's json that it will use to load modules
from, with a backup of calling to the underlying modulestore for more data.
Computes the metadata inheritance upon creation.
"""
def __init__(self, modulestore, course_entry, module_data, lazy,
default_class, error_tracker, render_template):
"""
Computes the metadata inheritance and sets up the cache.
modulestore: the module store that can be used to retrieve additional
modules
module_data: a dict mapping Location -> json that was cached from the
underlying modulestore
default_class: The default_class to use when loading an
XModuleDescriptor from the module_data
resources_fs: a filesystem, as per MakoDescriptorSystem
error_tracker: a function that logs errors for later display to users
render_template: a function for rendering templates, as per
MakoDescriptorSystem
"""
# TODO find all references to resources_fs and make handle None
super(CachingDescriptorSystem, self).__init__(
self._load_item, None, error_tracker, render_template)
self.modulestore = modulestore
self.course_entry = course_entry
self.lazy = lazy
self.module_data = module_data
self.default_class = default_class
# TODO see if self.course_id is needed: is already in course_entry but could be > 1 value
# Compute inheritance
modulestore.inherit_metadata(course_entry.get('blocks', {}),
course_entry.get('blocks', {})
.get(course_entry.get('root')))
def _load_item(self, usage_id, course_entry_override=None):
# TODO ensure all callers of system.load_item pass just the id
json_data = self.module_data.get(usage_id)
if json_data is None:
# deeper than initial descendant fetch or doesn't exist
self.modulestore.cache_items(self, [usage_id], lazy=self.lazy)
json_data = self.module_data.get(usage_id)
if json_data is None:
raise ItemNotFoundError
class_ = XModuleDescriptor.load_class(
json_data.get('category'),
self.default_class
)
return self.xblock_from_json(class_, usage_id, json_data, course_entry_override)
def xblock_from_json(self, class_, usage_id, json_data, course_entry_override=None):
if course_entry_override is None:
course_entry_override = self.course_entry
# most likely a lazy loader but not the id directly
definition = json_data.get('definition', {})
metadata = json_data.get('metadata', {})
block_locator = BlockUsageLocator(
version_guid=course_entry_override['_id'],
usage_id=usage_id,
course_id=course_entry_override.get('course_id'),
revision=course_entry_override.get('revision')
)
kvs = SplitMongoKVS(
definition,
json_data.get('children', []),
metadata,
json_data.get('_inherited_metadata'),
block_locator,
json_data.get('category'))
model_data = DbModel(kvs, class_, None,
SplitMongoKVSid(
# DbModel req's that these support .url()
block_locator,
self.modulestore.definition_locator(definition)))
try:
module = class_(self, model_data)
except Exception:
log.warning("Failed to load descriptor", exc_info=True)
if usage_id is None:
usage_id = "MISSING"
return ErrorDescriptor.from_json(
json_data,
self,
BlockUsageLocator(version_guid=course_entry_override['_id'],
usage_id=usage_id),
error_msg=exc_info_to_str(sys.exc_info())
)
module.edited_by = json_data.get('edited_by')
module.edited_on = json_data.get('edited_on')
module.previous_version = json_data.get('previous_version')
module.update_version = json_data.get('update_version')
module.definition_locator = self.modulestore.definition_locator(definition)
return module

26
common/lib/xmodule/xmodule/modulestore/split_mongo/definition_lazy_loader.py Normal file
View File

@@ -0,0 +1,26 @@
from xmodule.modulestore.locator import DescriptionLocator
class DefinitionLazyLoader(object):
"""
A placeholder to put into an xblock in place of its definition which
when accessed knows how to get its content. Only useful if the containing
object doesn't force access during init but waits until client wants the
definition. Only works if the modulestore is a split mongo store.
"""
def __init__(self, modulestore, definition_id):
"""
Simple placeholder for yet-to-be-fetched data
:param modulestore: the pymongo db connection with the definitions
:param definition_locator: the id of the record in the above to fetch
"""
self.modulestore = modulestore
self.definition_locator = DescriptionLocator(definition_id)
def fetch(self):
"""
Fetch the definition. Note, the caller should replace this lazy
loader pointer with the result so as not to fetch more than once
"""
return self.modulestore.definitions.find_one(
{'_id': self.definition_locator.definition_id})

1240
common/lib/xmodule/xmodule/modulestore/split_mongo/split.py Normal file
View File

@@ -0,0 +1,1240 @@
import threading
import datetime
import logging
import pymongo
import re
from importlib import import_module
from path import path
from xmodule.errortracker import null_error_tracker
from xmodule.x_module import XModuleDescriptor
from xmodule.modulestore.locator import BlockUsageLocator, DescriptionLocator, CourseLocator, VersionTree
from xmodule.modulestore.exceptions import InsufficientSpecificationError, VersionConflictError
from xmodule.modulestore import inheritance
from .. import ModuleStoreBase
from ..exceptions import ItemNotFoundError
from .definition_lazy_loader import DefinitionLazyLoader
from .caching_descriptor_system import CachingDescriptorSystem
log = logging.getLogger(__name__)
#==============================================================================
# Documentation is at
# https://edx-wiki.atlassian.net/wiki/display/ENG/Mongostore+Data+Structure
#
# Known issue:
# Inheritance for cached kvs doesn't work on edits. Use case.
# 1) attribute foo is inheritable
# 2) g.children = [p], p.children = [a]
# 3) g.foo = 1 on load
# 4) if g.foo > 0, if p.foo > 0, if a.foo > 0 all eval True
# 5) p.foo = -1
# 6) g.foo > 0, p.foo <= 0 all eval True BUT
# 7) BUG: a.foo > 0 still evals True but should be False
# 8) reread and everything works right
# 9) p.del(foo), p.foo > 0 is True! works
# 10) BUG: a.foo < 0!
# Local fix wont' permanently work b/c xblock may cache a.foo...
#
#==============================================================================
class SplitMongoModuleStore(ModuleStoreBase):
"""
A Mongodb backed ModuleStore supporting versions, inheritance,
and sharing.
"""
def __init__(self, host, db, collection, fs_root, render_template,
port=27017, default_class=None,
error_tracker=null_error_tracker,
user=None, password=None,
**kwargs):
ModuleStoreBase.__init__(self)
self.db = pymongo.database.Database(pymongo.MongoClient(
host=host,
port=port,
tz_aware=True,
**kwargs
), db)
# TODO add caching of structures to thread_cache to prevent repeated fetches (but not index b/c
# it changes w/o having a change in id)
self.course_index = self.db[collection + '.active_versions']
self.structures = self.db[collection + '.structures']
self.definitions = self.db[collection + '.definitions']
# ??? Code review question: those familiar w/ python threading. Should I instead
# use django cache? How should I expire entries?
# _add_cache could use a lru mechanism to control the cache size?
self.thread_cache = threading.local()
if user is not None and password is not None:
self.db.authenticate(user, password)
# every app has write access to the db (v having a flag to indicate r/o v write)
# Force mongo to report errors, at the expense of performance
# pymongo docs suck but explanation:
# http://api.mongodb.org/java/2.10.1/com/mongodb/WriteConcern.html
self.course_index.write_concern = {'w': 1}
self.structures.write_concern = {'w': 1}
self.definitions.write_concern = {'w': 1}
if default_class is not None:
module_path, _, class_name = default_class.rpartition('.')
class_ = getattr(import_module(module_path), class_name)
self.default_class = class_
else:
self.default_class = None
self.fs_root = path(fs_root)
self.error_tracker = error_tracker
self.render_template = render_template
def cache_items(self, system, base_usage_ids, depth=0, lazy=True):
'''
Handles caching of items once inheritance and any other one time
per course per fetch operations are done.
:param system: a CachingDescriptorSystem
:param base_usage_ids: list of usage_ids to fetch
:param depth: how deep below these to prefetch
:param lazy: whether to fetch definitions or use placeholders
'''
new_module_data = {}
for usage_id in base_usage_ids:
new_module_data = self.descendants(system.course_entry['blocks'],
usage_id,
depth,
new_module_data)
# remove any which were already in module_data (not sure if there's a better way)
for newkey in new_module_data.iterkeys():
if newkey in system.module_data:
del new_module_data[newkey]
if lazy:
for block in new_module_data.itervalues():
block['definition'] = DefinitionLazyLoader(self,
block['definition'])
else:
# Load all descendants by id
descendent_definitions = self.definitions.find({
'_id': {'$in': [block['definition']
for block in new_module_data.itervalues()]}})
# turn into a map
definitions = {definition['_id']: definition
for definition in descendent_definitions}
for block in new_module_data.itervalues():
if block['definition'] in definitions:
block['definition'] = definitions[block['definition']]
system.module_data.update(new_module_data)
return system.module_data
def _load_items(self, course_entry, usage_ids, depth=0, lazy=True):
'''
Load & cache the given blocks from the course. Prefetch down to the
given depth. Load the definitions into each block if lazy is False;
otherwise, use the lazy definition placeholder.
'''
system = self._get_cache(course_entry['_id'])
if system is None:
system = CachingDescriptorSystem(
self,
course_entry,
{},
lazy,
self.default_class,
self.error_tracker,
self.render_template
)
self._add_cache(course_entry['_id'], system)
self.cache_items(system, usage_ids, depth, lazy)
return [system.load_item(usage_id, course_entry) for usage_id in usage_ids]
def _get_cache(self, course_version_guid):
"""
Find the descriptor cache for this course if it exists
:param course_version_guid:
"""
if not hasattr(self.thread_cache, 'course_cache'):
self.thread_cache.course_cache = {}
system = self.thread_cache.course_cache
return system.get(course_version_guid)
def _add_cache(self, course_version_guid, system):
"""
Save this cache for subsequent access
:param course_version_guid:
:param system:
"""
if not hasattr(self.thread_cache, 'course_cache'):
self.thread_cache.course_cache = {}
self.thread_cache.course_cache[course_version_guid] = system
return system
def _clear_cache(self):
"""
Should only be used by testing or something which implements transactional boundary semantics
"""
self.thread_cache.course_cache = {}
def _lookup_course(self, course_locator):
'''
Decode the locator into the right series of db access. Does not
return the CourseDescriptor! It returns the actual db json from
structures.
Semantics: if course_id and revision given, then it will get that revision. If
also give a version_guid, it will see if the current head of that revision == that guid. If not
it raises VersionConflictError (the version now differs from what it was when you got your
reference)
:param course_locator: any subclass of CourseLocator
'''
# NOTE: if and when this uses cache, the update if changed logic will break if the cache
# holds the same objects as the descriptors!
if not course_locator.is_fully_specified():
raise InsufficientSpecificationError('Not fully specified: %s' % course_locator)
if course_locator.course_id is not None and course_locator.revision is not None:
# use the course_id
index = self.course_index.find_one({'_id': course_locator.course_id})
if index is None:
raise ItemNotFoundError(course_locator)
if course_locator.revision not in index['versions']:
raise ItemNotFoundError(course_locator)
version_guid = index['versions'][course_locator.revision]
if course_locator.version_guid is not None and version_guid != course_locator.version_guid:
# This may be a bit too touchy but it's hard to infer intent
raise VersionConflictError(course_locator, CourseLocator(course_locator, version_guid=version_guid))
else:
# TODO should this raise an exception if revision was provided?
version_guid = course_locator.version_guid
# cast string to ObjectId if necessary
version_guid = course_locator.as_object_id(version_guid)
entry = self.structures.find_one({'_id': version_guid})
# b/c more than one course can use same structure, the 'course_id' is not intrinsic to structure
# and the one assoc'd w/ it by another fetch may not be the one relevant to this fetch; so,
# fake it by explicitly setting it in the in memory structure.
if course_locator.course_id:
entry['course_id'] = course_locator.course_id
entry['revision'] = course_locator.revision
return entry
def get_courses(self, revision, qualifiers=None):
'''
Returns a list of course descriptors matching any given qualifiers.
qualifiers should be a dict of keywords matching the db fields or any
legal query for mongo to use against the active_versions collection.
Note, this is to find the current head of the named revision type
(e.g., 'draft'). To get specific versions via guid use get_course.
'''
if qualifiers is None:
qualifiers = {}
qualifiers.update({"versions.{}".format(revision): {"$exists": True}})
matching = self.course_index.find(qualifiers)
# collect ids and then query for those
version_guids = []
id_version_map = {}
for course_entry in matching:
version_guid = course_entry['versions'][revision]
version_guids.append(version_guid)
id_version_map[version_guid] = course_entry['_id']
course_entries = self.structures.find({'_id': {'$in': version_guids}})
# get the block for the course element (s/b the root)
result = []
for entry in course_entries:
# structures are course agnostic but the caller wants to know course, so add it in here
entry['course_id'] = id_version_map[entry['_id']]
root = entry['root']
result.extend(self._load_items(entry, [root], 0, lazy=True))
return result
def get_course(self, course_locator):
'''
Gets the course descriptor for the course identified by the locator
which may or may not be a blockLocator.
raises InsufficientSpecificationError
'''
course_entry = self._lookup_course(course_locator)
root = course_entry['root']
result = self._load_items(course_entry, [root], 0, lazy=True)
return result[0]
def get_course_for_item(self, location):
'''
Provided for backward compatibility. Is equivalent to calling get_course
:param location:
'''
return self.get_course(location)
def has_item(self, block_location):
"""
Returns True if location exists in its course. Returns false if
the course or the block w/in the course do not exist for the given version.
raises InsufficientSpecificationError if the locator does not id a block
"""
if block_location.usage_id is None:
raise InsufficientSpecificationError(block_location)
try:
course_structure = self._lookup_course(block_location)
except ItemNotFoundError:
# this error only occurs if the course does not exist
return False
return course_structure['blocks'].get(block_location.usage_id) is not None
def get_item(self, location, depth=0):
"""
depth (int): An argument that some module stores may use to prefetch
descendants of the queried modules for more efficient results later
in the request. The depth is counted in the number of
calls to get_children() to cache. None indicates to cache all
descendants.
raises InsufficientSpecificationError or ItemNotFoundError
"""
assert isinstance(location, BlockUsageLocator)
if not location.is_initialized():
raise InsufficientSpecificationError("Not yet initialized: %s" % location)
course = self._lookup_course(location)
items = self._load_items(course, [location.usage_id], depth, lazy=True)
if len(items) == 0:
raise ItemNotFoundError(location)
return items[0]
# TODO refactor this and get_courses to use a constructed query
def get_items(self, locator, qualifiers):
'''
Get all of the modules in the given course matching the qualifiers. The
qualifiers should only be fields in the structures collection (sorry).
There will be a separate search method for searching through
definitions.
Common qualifiers are category, definition (provide definition id),
metadata: {display_name ..}, children (return
block if its children includes the one given value). If you want
substring matching use {$regex: /acme.*corp/i} type syntax.
Although these
look like mongo queries, it is all done in memory; so, you cannot
try arbitrary queries.
:param locator: CourseLocator or BlockUsageLocator restricting search scope
:param qualifiers: a dict restricting which elements should match
'''
# TODO extend to only search a subdag of the course?
course = self._lookup_course(locator)
items = []
for usage_id, value in course['blocks'].iteritems():
if self._block_matches(value, qualifiers):
items.append(usage_id)
if len(items) > 0:
return self._load_items(course, items, 0, lazy=True)
else:
return []
# What's the use case for usage_id being separate?
def get_parent_locations(self, locator, usage_id=None):
'''
Return the locations (Locators w/ usage_ids) for the parents of this location in this
course. Could use get_items(location, {'children': usage_id}) but this is slightly faster.
NOTE: does not actually ensure usage_id exists
If usage_id is None, then the locator must specify the usage_id
'''
if usage_id is None:
usage_id = locator.usage_id
course = self._lookup_course(locator)
items = []
for parent_id, value in course['blocks'].iteritems():
for child_id in value['children']:
if usage_id == child_id:
locator = locator.as_course_locator()
items.append(BlockUsageLocator(url=locator, usage_id=parent_id))
return items
def get_course_index_info(self, course_locator):
"""
The index records the initial creation of the indexed course and tracks the current version
heads. This function is primarily for test verification but may serve some
more general purpose.
:param course_locator: must have a course_id set
:return {'org': , 'prettyid': ,
versions: {'draft': the head draft version id,
'published': the head published version id if any,
},
'edited_by': who created the course originally (named edited for consistency),
'edited_on': when the course was originally created
}
"""
if course_locator.course_id is None:
return None
index = self.course_index.find_one({'_id': course_locator.course_id})
return index
# TODO figure out a way to make this info accessible from the course descriptor
def get_course_history_info(self, course_locator):
"""
Because xblocks doesn't give a means to separate the course structure's meta information from
the course xblock's, this method will get that info for the structure as a whole.
:param course_locator:
:return {'original_version': the version guid of the original version of this course,
'previous_version': the version guid of the previous version,
'edited_by': who made the last change,
'edited_on': when the change was made
}
"""
course = self._lookup_course(course_locator)
return {'original_version': course['original_version'],
'previous_version': course['previous_version'],
'edited_by': course['edited_by'],
'edited_on': course['edited_on']
}
def get_definition_history_info(self, definition_locator):
"""
Because xblocks doesn't give a means to separate the definition's meta information from
the usage xblock's, this method will get that info for the definition
:return {'original_version': the version guid of the original version of this course,
'previous_version': the version guid of the previous version,
'edited_by': who made the last change,
'edited_on': when the change was made
}
"""
definition = self.definitions.find_one({'_id': definition_locator.definition_id})
if definition is None:
return None
return {'original_version': definition['original_version'],
'previous_version': definition['previous_version'],
'edited_by': definition['edited_by'],
'edited_on': definition['edited_on']
}
def get_course_successors(self, course_locator, version_history_depth=1):
'''
Find the version_history_depth next versions of this course. Return as a VersionTree
Mostly makes sense when course_locator uses a version_guid, but because it finds all relevant
next versions, these do include those created for other courses.
:param course_locator:
'''
if version_history_depth < 1:
return None
if course_locator.version_guid is None:
course = self._lookup_course(course_locator)
version_guid = course.version_guid
else:
version_guid = course_locator.version_guid
# TODO if depth is significant, it may make sense to get all that have the same original_version
# and reconstruct the subtree from version_guid
next_entries = self.structures.find({'previous_version' : version_guid})
# must only scan cursor's once
next_versions = [struct for struct in next_entries]
result = {version_guid: [CourseLocator(version_guid=struct['_id']) for struct in next_versions]}
depth = 1
while depth < version_history_depth and len(next_versions) > 0:
depth += 1
next_entries = self.structures.find({'previous_version':
{'$in': [struct['_id'] for struct in next_versions]}})
next_versions = [struct for struct in next_entries]
for course_structure in next_versions:
result.setdefault(course_structure['previous_version'], []).append(
CourseLocator(version_guid=struct['_id']))
return VersionTree(CourseLocator(course_locator, version_guid=version_guid), result)
def get_block_generations(self, block_locator):
'''
Find the history of this block. Return as a VersionTree of each place the block changed (except
deletion).
The block's history tracks its explicit changes; so, changes in descendants won't be reflected
as new iterations.
'''
block_locator = block_locator.version_agnostic()
course_struct = self._lookup_course(block_locator)
usage_id = block_locator.usage_id
update_version_field = 'blocks.{}.update_version'.format(usage_id)
all_versions_with_block = self.structures.find({'original_version': course_struct['original_version'],
update_version_field: {'$exists': True}})
# find (all) root versions and build map previous: [successors]
possible_roots = []
result = {}
for version in all_versions_with_block:
if version['_id'] == version['blocks'][usage_id]['update_version']:
if version['blocks'][usage_id].get('previous_version') is None:
possible_roots.append(version['blocks'][usage_id]['update_version'])
else:
result.setdefault(version['blocks'][usage_id]['previous_version'], set()).add(
version['blocks'][usage_id]['update_version'])
# more than one possible_root means usage was added and deleted > 1x.
if len(possible_roots) > 1:
# find the history segment including block_locator's version
element_to_find = course_struct['blocks'][usage_id]['update_version']
if element_to_find in possible_roots:
possible_roots = [element_to_find]
for possibility in possible_roots:
if self._find_local_root(element_to_find, possibility, result):
possible_roots = [possibility]
break
elif len(possible_roots) == 0:
return None
# convert the results value sets to locators
for k, versions in result.iteritems():
result[k] = [BlockUsageLocator(version_guid=version, usage_id=usage_id)
for version in versions]
return VersionTree(BlockUsageLocator(version_guid=possible_roots[0], usage_id=usage_id), result)
def get_definition_successors(self, definition_locator, version_history_depth=1):
'''
Find the version_history_depth next versions of this definition. Return as a VersionTree
'''
# TODO implement
pass
def create_definition_from_data(self, new_def_data, category, user_id):
"""
Pull the definition fields out of descriptor and save to the db as a new definition
w/o a predecessor and return the new id.
:param user_id: request.user object
"""
document = {"category" : category,
"data": new_def_data,
"edited_by": user_id,
"edited_on": datetime.datetime.utcnow(),
"previous_version": None,
"original_version": None}
new_id = self.definitions.insert(document)
definition_locator = DescriptionLocator(new_id)
document['original_version'] = new_id
self.definitions.update({'_id': new_id}, {'$set': {"original_version": new_id}})
return definition_locator
def update_definition_from_data(self, definition_locator, new_def_data, user_id):
"""
See if new_def_data differs from the persisted version. If so, update
the persisted version and return the new id.
:param user_id: request.user
"""
def needs_saved():
if isinstance(new_def_data, dict):
for key, value in new_def_data.iteritems():
if key not in old_definition['data'] or value != old_definition['data'][key]:
return True
for key, value in old_definition['data'].iteritems():
if key not in new_def_data:
return True
else:
return new_def_data != old_definition['data']
# if this looks in cache rather than fresh fetches, then it will probably not detect
# actual change b/c the descriptor and cache probably point to the same objects
old_definition = self.definitions.find_one({'_id': definition_locator.definition_id})
if old_definition is None:
raise ItemNotFoundError(definition_locator.url())
del old_definition['_id']
if needs_saved():
old_definition['data'] = new_def_data
old_definition['edited_by'] = user_id
old_definition['edited_on'] = datetime.datetime.utcnow()
old_definition['previous_version'] = definition_locator.definition_id
new_id = self.definitions.insert(old_definition)
return DescriptionLocator(new_id), True
else:
return definition_locator, False
def _generate_usage_id(self, course_blocks, category):
"""
Generate a somewhat readable block id unique w/in this course using the category
:param course_blocks: the current list of blocks.
:param category:
"""
# NOTE: a potential bug is that a block is deleted and another created which gets the old
# block's id. a possible fix is to cache the last serial in a dict in the structure
# {category: last_serial...}
# A potential confusion is if the name incorporates the parent's name, then if the child
# moves, its id won't change and will be confusing
serial = 1
while category + str(serial) in course_blocks:
serial += 1
return category + str(serial)
def _generate_course_id(self, id_root):
"""
Generate a somewhat readable course id unique w/in this db using the id_root
:param course_blocks: the current list of blocks.
:param category:
"""
existing_uses = self.course_index.find({"_id": {"$regex": id_root}})
if existing_uses.count() > 0:
max_found = 0
matcher = re.compile(id_root + r'(\d+)')
for entry in existing_uses:
serial = re.search(matcher, entry['_id'])
if serial is not None and serial.groups > 0:
value = int(serial.group(1))
if value > max_found:
max_found = value
return id_root + str(max_found + 1)
else:
return id_root
# TODO I would love to write this to take a real descriptor and persist it BUT descriptors, kvs, and dbmodel
# all assume locators are set and unique! Having this take the model contents piecemeal breaks the separation
# of model from persistence layer
def create_item(self, course_or_parent_locator, category, user_id, definition_locator=None, new_def_data=None,
metadata=None, force=False):
"""
Add a descriptor to persistence as the last child of the optional parent_location or just as an element
of the course (if no parent provided). Return the resulting post saved version with populated locators.
If the locator is a BlockUsageLocator, then it's assumed to be the parent. If it's a CourseLocator, then it's
merely the containing course.
raises InsufficientSpecificationError if there is no course locator.
raises VersionConflictError if course_id and version_guid given and the current version head != version_guid
and force is not True.
force: fork the structure and don't update the course draftVersion if the above
The incoming definition_locator should either be None to indicate this is a brand new definition or
a pointer to the existing definition to which this block should point or from which this was derived.
If new_def_data is None, then definition_locator must have a value meaning that this block points
to the existing definition. If new_def_data is not None and definition_location is not None, then
new_def_data is assumed to be a new payload for definition_location.
Creates a new version of the course structure, creates and inserts the new block, makes the block point
to the definition which may be new or a new version of an existing or an existing.
Rules for course locator:
* If the course locator specifies a course_id and either it doesn't
specify version_guid or the one it specifies == the current draft, it progresses the course to point
to the new draft and sets the active version to point to the new draft
* If the locator has a course_id but its version_guid != current draft, it raises VersionConflictError.
NOTE: using a version_guid will end up creating a new version of the course. Your new item won't be in
the course id'd by version_guid but instead in one w/ a new version_guid. Ensure in this case that you get
the new version_guid from the locator in the returned object!
"""
# find course_index entry if applicable and structures entry
index_entry = self._get_index_if_valid(course_or_parent_locator, force)
structure = self._lookup_course(course_or_parent_locator)
# persist the definition if persisted != passed
if (definition_locator is None or definition_locator.definition_id is None):
definition_locator = self.create_definition_from_data(new_def_data, category, user_id)
elif new_def_data is not None:
definition_locator, _ = self.update_definition_from_data(definition_locator, new_def_data, user_id)
# copy the structure and modify the new one
new_structure = self._version_structure(structure, user_id)
# generate an id
new_usage_id = self._generate_usage_id(new_structure['blocks'], category)
update_version_keys = ['blocks.{}.update_version'.format(new_usage_id)]
if isinstance(course_or_parent_locator, BlockUsageLocator) and course_or_parent_locator.usage_id is not None:
parent = new_structure['blocks'][course_or_parent_locator.usage_id]
parent['children'].append(new_usage_id)
parent['edited_on'] = datetime.datetime.utcnow()
parent['edited_by'] = user_id
parent['previous_version'] = parent['update_version']
update_version_keys.append('blocks.{}.update_version'.format(course_or_parent_locator.usage_id))
new_structure['blocks'][new_usage_id] = {
"children": [],
"category": category,
"definition": definition_locator.definition_id,
"metadata": metadata if metadata else {},
'edited_on': datetime.datetime.utcnow(),
'edited_by': user_id,
'previous_version': None
}
new_id = self.structures.insert(new_structure)
update_version_payload = {key: new_id for key in update_version_keys}
self.structures.update({'_id': new_id},
{'$set': update_version_payload})
# update the index entry if appropriate
if index_entry is not None:
self._update_head(index_entry, course_or_parent_locator.revision, new_id)
course_parent = course_or_parent_locator.as_course_locator()
else:
course_parent = None
# fetch and return the new item--fetching is unnecessary but a good qc step
return self.get_item(BlockUsageLocator(course_id=course_parent,
usage_id=new_usage_id,
version_guid=new_id))
def create_course(self, org, prettyid, user_id, id_root=None, metadata=None, course_data=None,
master_version='draft', versions_dict=None, root_category='course'):
"""
Create a new entry in the active courses index which points to an existing or new structure. Returns
the course root of the resulting entry (the location has the course id)
id_root: allows the caller to specify the course_id. It's a root in that, if it's already taken,
this method will append things to the root to make it unique. (defaults to org)
metadata: if provided, will set the metadata of the root course object in the new draft course. If both
metadata and a starting version are provided, it will generate a successor version to the given version,
and update the metadata with any provided values (via update not setting).
course_data: if provided, will update the data of the new course xblock definition to this. Like metadata,
if provided, this will cause a new version of any given version as well as a new version of the
definition (which will point to the existing one if given a version). If not provided and given
a draft_version, it will reuse the same definition as the draft course (obvious since it's reusing the draft
course). If not provided and no draft is given, it will be empty and get the field defaults (hopefully) when
loaded.
master_version: the tag (key) for the version name in the dict which is the 'draft' version. Not the actual
version guid, but what to call it.
versions_dict: the starting version ids where the keys are the tags such as 'draft' and 'published'
and the values are structure guids. If provided, the new course will reuse this version (unless you also
provide any overrides such as metadata, see above). if not provided, will create a mostly empty course
structure with just a category course root xblock.
"""
if metadata is None:
metadata = {}
# build from inside out: definition, structure, index entry
# if building a wholly new structure
if versions_dict is None or master_version not in versions_dict:
# create new definition and structure
if course_data is None:
course_data = {}
definition_entry = {
'category': root_category,
'data': course_data,
'edited_by': user_id,
'edited_on': datetime.datetime.utcnow(),
'previous_version': None,
}
definition_id = self.definitions.insert(definition_entry)
definition_entry['original_version'] = definition_id
self.definitions.update({'_id': definition_id}, {'$set': {"original_version": definition_id}})
draft_structure = {
'root': 'course',
'previous_version': None,
'edited_by': user_id,
'edited_on': datetime.datetime.utcnow(),
'blocks': {
'course': {
'children':[],
'category': 'course',
'definition': definition_id,
'metadata': metadata,
'edited_on': datetime.datetime.utcnow(),
'edited_by': user_id,
'previous_version': None}}}
new_id = self.structures.insert(draft_structure)
draft_structure['original_version'] = new_id
self.structures.update({'_id': new_id},
{'$set': {"original_version": new_id,
'blocks.course.update_version': new_id}})
if versions_dict is None:
versions_dict = {master_version: new_id}
else:
versions_dict[master_version] = new_id
else:
# just get the draft_version structure
draft_version = CourseLocator(version_guid=versions_dict[master_version])
draft_structure = self._lookup_course(draft_version)
if course_data is not None or metadata:
draft_structure = self._version_structure(draft_structure, user_id)
root_block = draft_structure['blocks'][draft_structure['root']]
if metadata is not None:
root_block['metadata'].update(metadata)
if course_data is not None:
definition = self.definitions.find_one({'_id': root_block['definition']})
definition['data'].update(course_data)
definition['previous_version'] = definition['_id']
definition['edited_by'] = user_id
definition['edited_on'] = datetime.datetime.utcnow()
del definition['_id']
root_block['definition'] = self.definitions.insert(definition)
root_block['edited_on'] = datetime.datetime.utcnow()
root_block['edited_by'] = user_id
root_block['previous_version'] = root_block.get('update_version')
# insert updates the '_id' in draft_structure
new_id = self.structures.insert(draft_structure)
versions_dict[master_version] = new_id
self.structures.update({'_id': new_id},
{'$set': {'blocks.{}.update_version'.format(draft_structure['root']): new_id}})
# create the index entry
if id_root is None:
id_root = org
new_id = self._generate_course_id(id_root)
index_entry = {
'_id': new_id,
'org': org,
'prettyid': prettyid,
'edited_by': user_id,
'edited_on': datetime.datetime.utcnow(),
'versions': versions_dict}
new_id = self.course_index.insert(index_entry)
return self.get_course(CourseLocator(course_id=new_id, revision=master_version))
def update_item(self, descriptor, user_id, force=False):
"""
Save the descriptor's definition, metadata, & children references (i.e., it doesn't descend the tree).
Return the new descriptor (updated location).
raises ItemNotFoundError if the location does not exist.
Creates a new course version. If the descriptor's location has a course_id, it moves the course head
pointer. If the version_guid of the descriptor points to a non-head version and there's been an intervening
change to this item, it raises a VersionConflictError unless force is True. In the force case, it forks
the course but leaves the head pointer where it is (this change will not be in the course head).
The implementation tries to detect which, if any changes, actually need to be saved and thus won't version
the definition, structure, nor course if they didn't change.
"""
original_structure = self._lookup_course(descriptor.location)
index_entry = self._get_index_if_valid(descriptor.location, force)
descriptor.definition_locator, is_updated = self.update_definition_from_data(
descriptor.definition_locator, descriptor.xblock_kvs.get_data(), user_id)
# check children
original_entry = original_structure['blocks'][descriptor.location.usage_id]
if (not is_updated and descriptor.has_children
and not self._xblock_lists_equal(original_entry['children'], descriptor.children)):
is_updated = True
# check metadata
if not is_updated:
is_updated = self._compare_metadata(descriptor.xblock_kvs.get_own_metadata(), original_entry['metadata'])
# if updated, rev the structure
if is_updated:
new_structure = self._version_structure(original_structure, user_id)
block_data = new_structure['blocks'][descriptor.location.usage_id]
if descriptor.has_children:
block_data["children"] = [self._usage_id(child) for child in descriptor.children]
block_data["definition"] = descriptor.definition_locator.definition_id
block_data["metadata"] = descriptor.xblock_kvs.get_own_metadata()
block_data['edited_on'] = datetime.datetime.utcnow()
block_data['edited_by'] = user_id
block_data['previous_version'] = block_data['update_version']
new_id = self.structures.insert(new_structure)
self.structures.update({'_id': new_id},
{'$set': {'blocks.{}.update_version'.format(descriptor.location.usage_id): new_id}})
# update the index entry if appropriate
if index_entry is not None:
self._update_head(index_entry, descriptor.location.revision, new_id)
# fetch and return the new item--fetching is unnecessary but a good qc step
return self.get_item(BlockUsageLocator(descriptor.location, version_guid=new_id))
else:
# nothing changed, just return the one sent in
return descriptor
def persist_xblock_dag(self, xblock, user_id, force=False):
"""
create or update the xblock and all of its children. The xblock's location must specify a course.
If it doesn't specify a usage_id, then it's presumed to be new and need creation. This function
descends the children performing the same operation for any that are xblocks. Any children which
are usage_ids just update the children pointer.
All updates go into the same course version (bulk updater).
Updates the objects which came in w/ updated location and definition_location info.
returns the post-persisted version of the incoming xblock. Note that its children will be ids not
objects.
:param xblock:
:param user_id:
"""
# find course_index entry if applicable and structures entry
index_entry = self._get_index_if_valid(xblock.location, force)
structure = self._lookup_course(xblock.location)
new_structure = self._version_structure(structure, user_id)
changed_blocks = self._persist_subdag(xblock, user_id, new_structure['blocks'])
if changed_blocks:
new_id = self.structures.insert(new_structure)
update_command = {}
for usage_id in changed_blocks:
update_command['blocks.{}.update_version'.format(usage_id)] = new_id
self.structures.update({'_id': new_id}, {'$set': update_command})
# update the index entry if appropriate
if index_entry is not None:
self._update_head(index_entry, xblock.location.revision, new_id)
# fetch and return the new item--fetching is unnecessary but a good qc step
return self.get_item(BlockUsageLocator(xblock.location, version_guid=new_id))
else:
return xblock
def _persist_subdag(self, xblock, user_id, structure_blocks):
# persist the definition if persisted != passed
new_def_data = xblock.xblock_kvs.get_data()
if (xblock.definition_locator is None or xblock.definition_locator.definition_id is None):
xblock.definition_locator = self.create_definition_from_data(new_def_data,
xblock.category, user_id)
is_updated = True
elif new_def_data is not None:
xblock.definition_locator, is_updated = self.update_definition_from_data(xblock.definition_locator,
new_def_data, user_id)
if xblock.location.usage_id is None:
# generate an id
is_new = True
is_updated = True
usage_id = self._generate_usage_id(structure_blocks, xblock.category)
xblock.location.usage_id = usage_id
else:
is_new = False
usage_id = xblock.location.usage_id
if (not is_updated and xblock.has_children
and not self._xblock_lists_equal(structure_blocks[usage_id]['children'], xblock.children)):
is_updated = True
children = []
updated_blocks = []
if xblock.has_children:
for child in xblock.children:
if isinstance(child, XModuleDescriptor):
updated_blocks += self._persist_subdag(child, user_id, structure_blocks)
children.append(child.location.usage_id)
else:
children.append(child)
is_updated = is_updated or updated_blocks
metadata = xblock.xblock_kvs.get_own_metadata()
if not is_new and not is_updated:
is_updated = self._compare_metadata(metadata, structure_blocks[usage_id]['metadata'])
if is_updated:
structure_blocks[usage_id] = {
"children": children,
"category": xblock.category,
"definition": xblock.definition_locator.definition_id,
"metadata": metadata if metadata else {},
'previous_version': structure_blocks.get(usage_id, {}).get('update_version'),
'edited_by': user_id,
'edited_on': datetime.datetime.utcnow()
}
updated_blocks.append(usage_id)
return updated_blocks
def _compare_metadata(self, metadata, original_metadata):
original_keys = original_metadata.keys()
if len(metadata) != len(original_keys):
return True
else:
new_keys = metadata.keys()
for key in original_keys:
if key not in new_keys or original_metadata[key] != metadata[key]:
return True
# TODO change all callers to update_item
def update_children(self, course_id, location, children):
raise NotImplementedError()
# TODO change all callers to update_item
def update_metadata(self, course_id, location, metadata):
raise NotImplementedError()
def update_course_index(self, course_locator, new_values_dict, update_versions=False):
"""
Change the given course's index entry for the given fields. new_values_dict
should be a subset of the dict returned by get_course_index_info.
It cannot include '_id' (will raise IllegalArgument).
Provide update_versions=True if you intend this to replace the versions hash.
Note, this operation can be dangerous and break running courses.
If the dict includes versions and not update_versions, it will raise an exception.
If the dict includes edited_on or edited_by, it will raise an exception
Does not return anything useful.
"""
# TODO how should this log the change? edited_on and edited_by for this entry
# has the semantic of who created the course and when; so, changing those will lose
# that information.
if '_id' in new_values_dict:
raise ValueError("Cannot override _id")
if 'edited_on' in new_values_dict or 'edited_by' in new_values_dict:
raise ValueError("Cannot set edited_on or edited_by")
if not update_versions and 'versions' in new_values_dict:
raise ValueError("Cannot override versions without setting update_versions")
self.course_index.update({'_id': course_locator.course_id},
{'$set': new_values_dict})
def delete_item(self, usage_locator, user_id, force=False):
"""
Delete the tree rooted at block and any references w/in the course to the block
from a new version of the course structure.
returns CourseLocator for new version
raises ItemNotFoundError if the location does not exist.
raises ValueError if usage_locator points to the structure root
Creates a new course version. If the descriptor's location has a course_id, it moves the course head
pointer. If the version_guid of the descriptor points to a non-head version and there's been an intervening
change to this item, it raises a VersionConflictError unless force is True. In the force case, it forks
the course but leaves the head pointer where it is (this change will not be in the course head).
"""
assert isinstance(usage_locator, BlockUsageLocator) and usage_locator.is_initialized()
original_structure = self._lookup_course(usage_locator)
if original_structure['root'] == usage_locator.usage_id:
raise ValueError("Cannot delete the root of a course")
index_entry = self._get_index_if_valid(usage_locator, force)
new_structure = self._version_structure(original_structure, user_id)
new_blocks = new_structure['blocks']
parents = self.get_parent_locations(usage_locator)
update_version_keys = []
for parent in parents:
parent_block = new_blocks[parent.usage_id]
parent_block['children'].remove(usage_locator.usage_id)
parent_block['edited_on'] = datetime.datetime.utcnow()
parent_block['edited_by'] = user_id
parent_block['previous_version'] = parent_block['update_version']
update_version_keys.append('blocks.{}.update_version'.format(parent.usage_id))
# remove subtree
def remove_subtree(usage_id):
for child in new_blocks[usage_id]['children']:
remove_subtree(child)
del new_blocks[usage_id]
remove_subtree(usage_locator.usage_id)
# update index if appropriate and structures
new_id = self.structures.insert(new_structure)
if update_version_keys:
update_version_payload = {key: new_id for key in update_version_keys}
self.structures.update({'_id': new_id}, {'$set': update_version_payload})
result = CourseLocator(version_guid=new_id)
# update the index entry if appropriate
if index_entry is not None:
self._update_head(index_entry, usage_locator.revision, new_id)
result.course_id = usage_locator.course_id
result.revision = usage_locator.revision
return result
def delete_course(self, course_id):
"""
Remove the given course from the course index.
Only removes the course from the index. The data remains. You can use create_course
with a versions hash to restore the course; however, the edited_on and
edited_by won't reflect the originals, of course.
:param course_id: uses course_id rather than locator to emphasize its global effect
"""
index = self.course_index.find_one({'_id': course_id})
if index is None:
raise ItemNotFoundError(course_id)
# this is the only real delete in the system. should it do something else?
self.course_index.remove(index['_id'])
# TODO remove all callers and then this
def get_errored_courses(self):
"""
This function doesn't make sense for the mongo modulestore, as structures
are loaded on demand, rather than up front
"""
return {}
def inherit_metadata(self, block_map, block, inheriting_metadata=None):
"""
Updates block with any value
that exist in inheriting_metadata and don't appear in block['metadata'],
and then inherits block['metadata'] to all of the children in
block['children']. Filters by inheritance.INHERITABLE_METADATA
"""
if block is None:
return
if inheriting_metadata is None:
inheriting_metadata = {}
# the currently passed down values take precedence over any previously cached ones
# NOTE: this should show the values which all fields would have if inherited: i.e.,
# not set to the locally defined value but to value set by nearest ancestor who sets it
block.setdefault('_inherited_metadata', {}).update(inheriting_metadata)
# update the inheriting w/ what should pass to children
inheriting_metadata = block['_inherited_metadata'].copy()
for field in inheritance.INHERITABLE_METADATA:
if field in block['metadata']:
inheriting_metadata[field] = block['metadata'][field]
for child in block.get('children', []):
self.inherit_metadata(block_map, block_map[child], inheriting_metadata)
def descendants(self, block_map, usage_id, depth, descendent_map):
"""
adds block and its descendants out to depth to descendent_map
Depth specifies the number of levels of descendants to return
(0 => this usage only, 1 => this usage and its children, etc...)
A depth of None returns all descendants
"""
if usage_id not in block_map:
return descendent_map
if usage_id not in descendent_map:
descendent_map[usage_id] = block_map[usage_id]
if depth is None or depth > 0:
depth = depth - 1 if depth is not None else None
for child in block_map[usage_id].get('children', []):
descendent_map = self.descendants(block_map, child, depth,
descendent_map)
return descendent_map
def definition_locator(self, definition):
'''
Pull the id out of the definition w/ correct semantics for its
representation
'''
if isinstance(definition, DefinitionLazyLoader):
return definition.definition_locator
elif '_id' not in definition:
return None
else:
return DescriptionLocator(definition['_id'])
def _block_matches(self, value, qualifiers):
'''
Return True or False depending on whether the value (block contents)
matches the qualifiers as per get_items
:param value:
:param qualifiers:
'''
for key, criteria in qualifiers.iteritems():
if key in value:
target = value[key]
if not self._value_matches(target, criteria):
return False
elif criteria is not None:
return False
return True
def _value_matches(self, target, criteria):
''' helper for _block_matches '''
if isinstance(target, list):
return any(self._value_matches(ele, criteria)
for ele in target)
elif isinstance(criteria, dict):
if '$regex' in criteria:
return re.search(criteria['$regex'], target) is not None
elif not isinstance(target, dict):
return False
else:
return (isinstance(target, dict) and
self._block_matches(target, criteria))
else:
return criteria == target
def _xblock_lists_equal(self, lista, listb):
"""
Do the 2 lists refer to the same xblocks in the same order (presumes they're from the
same course)
:param lista:
:param listb:
"""
if len(lista) != len(listb):
return False
for idx in enumerate(lista):
if lista[idx] != listb[idx]:
itema = self._usage_id(lista[idx])
if itema != self._usage_id(listb[idx]):
return False
return True
def _usage_id(self, xblock_or_id):
"""
arg is either an xblock or an id. If an xblock, get the usage_id from its location. Otherwise, return itself.
:param xblock_or_id:
"""
if isinstance(xblock_or_id, XModuleDescriptor):
return xblock_or_id.location.usage_id
else:
return xblock_or_id
def _get_index_if_valid(self, locator, force=False):
"""
If the locator identifies a course and points to its draft (or plausibly its draft),
then return the index entry.
raises VersionConflictError if not the right version
:param locator:
"""
if locator.course_id is None or locator.revision is None:
return None
else:
index_entry = self.course_index.find_one({'_id': locator.course_id})
if (locator.version_guid is not None
and index_entry['versions'][locator.revision] != locator.version_guid
and not force):
raise VersionConflictError(
locator,
CourseLocator(
course_id=index_entry['_id'],
version_guid=index_entry['versions'][locator.revision],
revision=locator.revision))
else:
return index_entry
def _version_structure(self, structure, user_id):
"""
Copy the structure and update the history info (edited_by, edited_on, previous_version)
:param structure:
:param user_id:
"""
new_structure = structure.copy()
new_structure['blocks'] = new_structure['blocks'].copy()
del new_structure['_id']
new_structure['previous_version'] = structure['_id']
new_structure['edited_by'] = user_id
new_structure['edited_on'] = datetime.datetime.utcnow()
return new_structure
def _find_local_root(self, element_to_find, possibility, tree):
if possibility not in tree:
return False
if element_to_find in tree[possibility]:
return True
for subtree in tree[possibility]:
if self._find_local_root(element_to_find, subtree, tree):
return True
return False
def _update_head(self, index_entry, revision, new_id):
"""
Update the active index for the given course's revision to point to new_id
:param index_entry:
:param course_locator:
:param new_id:
"""
self.course_index.update(
{"_id": index_entry["_id"]},
{"$set": {"versions.{}".format(revision): new_id}})

163
common/lib/xmodule/xmodule/modulestore/split_mongo/split_mongo_kvs.py Normal file
View File

@@ -0,0 +1,163 @@
import copy
from xblock.core import Scope
from collections import namedtuple
from xblock.runtime import KeyValueStore, InvalidScopeError
from .definition_lazy_loader import DefinitionLazyLoader
# id is a BlockUsageLocator, def_id is the definition's guid
SplitMongoKVSid = namedtuple('SplitMongoKVSid', 'id, def_id')
# TODO should this be here or w/ x_module or ???
class SplitMongoKVS(KeyValueStore):
"""
A KeyValueStore that maps keyed data access to one of the 3 data areas
known to the MongoModuleStore (data, children, and metadata)
"""
def __init__(self, definition, children, metadata, _inherited_metadata, location, category):
"""
:param definition:
:param children:
:param metadata: the locally defined value for each metadata field
:param _inherited_metadata: the value of each inheritable field from above this.
Note, metadata may override and disagree w/ this b/c this says what the value
should be if metadata is undefined for this field.
"""
# ensure kvs's don't share objects w/ others so that changes can't appear in separate ones
# the particular use case was that changes to kvs's were polluting caches. My thinking was
# that kvs's should be independent thus responsible for the isolation.
if isinstance(definition, DefinitionLazyLoader):
self._definition = definition
else:
self._definition = copy.copy(definition)
self._children = copy.copy(children)
self._metadata = copy.copy(metadata)
self._inherited_metadata = _inherited_metadata
self._location = location
self._category = category
def get(self, key):
if key.scope == Scope.children:
return self._children
elif key.scope == Scope.parent:
return None
elif key.scope == Scope.settings:
if key.field_name in self._metadata:
return self._metadata[key.field_name]
elif key.field_name in self._inherited_metadata:
return self._inherited_metadata[key.field_name]
else:
raise KeyError()
elif key.scope == Scope.content:
if key.field_name == 'location':
return self._location
elif key.field_name == 'category':
return self._category
else:
if isinstance(self._definition, DefinitionLazyLoader):
self._definition = self._definition.fetch()
if (key.field_name == 'data' and
not isinstance(self._definition.get('data'), dict)):
return self._definition.get('data')
elif 'data' not in self._definition or key.field_name not in self._definition['data']:
raise KeyError()
else:
return self._definition['data'][key.field_name]
else:
raise InvalidScopeError(key.scope)
def set(self, key, value):
# TODO cache db update implications & add method to invoke
if key.scope == Scope.children:
self._children = value
# TODO remove inheritance from any orphaned exchildren
# TODO add inheritance to any new children
elif key.scope == Scope.settings:
# TODO if inheritable, push down to children who don't override
self._metadata[key.field_name] = value
elif key.scope == Scope.content:
if key.field_name == 'location':
self._location = value
elif key.field_name == 'category':
self._category = value
else:
if isinstance(self._definition, DefinitionLazyLoader):
self._definition = self._definition.fetch()
if (key.field_name == 'data' and
not isinstance(self._definition.get('data'), dict)):
self._definition.get('data')
else:
self._definition.setdefault('data', {})[key.field_name] = value
else:
raise InvalidScopeError(key.scope)
def delete(self, key):
# TODO cache db update implications & add method to invoke
if key.scope == Scope.children:
self._children = []
elif key.scope == Scope.settings:
# TODO if inheritable, ensure _inherited_metadata has value from above and
# revert children to that value
if key.field_name in self._metadata:
del self._metadata[key.field_name]
elif key.scope == Scope.content:
# don't allow deletion of location nor category
if key.field_name == 'location':
pass
elif key.field_name == 'category':
pass
else:
if isinstance(self._definition, DefinitionLazyLoader):
self._definition = self._definition.fetch()
if (key.field_name == 'data' and
not isinstance(self._definition.get('data'), dict)):
self._definition.setdefault('data', None)
else:
try:
del self._definition['data'][key.field_name]
except KeyError:
pass
else:
raise InvalidScopeError(key.scope)
def has(self, key):
if key.scope in (Scope.children, Scope.parent):
return True
elif key.scope == Scope.settings:
return key.field_name in self._metadata or key.field_name in self._inherited_metadata
elif key.scope == Scope.content:
if key.field_name == 'location':
return True
elif key.field_name == 'category':
return self._category is not None
else:
if isinstance(self._definition, DefinitionLazyLoader):
self._definition = self._definition.fetch()
if (key.field_name == 'data' and
not isinstance(self._definition.get('data'), dict)):
return self._definition.get('data') is not None
else:
return key.field_name in self._definition.get('data', {})
else:
return False
def get_data(self):
"""
Intended only for use by persistence layer to get the native definition['data'] rep
"""
if isinstance(self._definition, DefinitionLazyLoader):
self._definition = self._definition.fetch()
return self._definition.get('data')
def get_own_metadata(self):
"""
Get the metadata explicitly set on this element.
"""
return self._metadata
def get_inherited_metadata(self):
"""
Get the metadata set by the ancestors (which own metadata may override or not)
"""
return self._inherited_metadata

10
common/lib/xmodule/xmodule/modulestore/tests/factories.py
View File

@@ -38,16 +38,6 @@ class XModuleCourseFactory(Factory):
new_course.display_name = display_name
new_course.lms.start = datetime.datetime.now(UTC).replace(microsecond=0)
new_course.tabs = kwargs.pop(
'tabs',
[
{"type": "courseware"},
{"type": "course_info", "name": "Course Info"},
{"type": "discussion", "name": "Discussion"},
{"type": "wiki", "name": "Wiki"},
{"type": "progress", "name": "Progress"}
]
)
# The rest of kwargs become attributes on the course:
for k, v in kwargs.iteritems():

96
common/lib/xmodule/xmodule/modulestore/tests/persistent_factories.py Normal file
View File

@@ -0,0 +1,96 @@
from xmodule.modulestore.django import modulestore
from xmodule.course_module import CourseDescriptor
from xmodule.x_module import XModuleDescriptor
import factory
# [dhm] I'm not sure why we're using factory_boy if we're not following its pattern. If anyone
# assumes they can call build, it will completely fail, for example.
# pylint: disable=W0232
class PersistentCourseFactory(factory.Factory):
"""
Create a new course (not a new version of a course, but a whole new index entry).
keywords:
* org: defaults to textX
* prettyid: defaults to 999
* display_name
* user_id
* data (optional) the data payload to save in the course item
* metadata (optional) the metadata payload. If display_name is in the metadata, that takes
precedence over any display_name provided directly.
"""
FACTORY_FOR = CourseDescriptor
org = 'testX'
prettyid = '999'
display_name = 'Robot Super Course'
user_id = "test_user"
data = None
metadata = None
master_version = 'draft'
# pylint: disable=W0613
@classmethod
def _create(cls, target_class, *args, **kwargs):
org = kwargs.get('org')
prettyid = kwargs.get('prettyid')
display_name = kwargs.get('display_name')
user_id = kwargs.get('user_id')
data = kwargs.get('data')
metadata = kwargs.get('metadata', {})
if metadata is None:
metadata = {}
if 'display_name' not in metadata:
metadata['display_name'] = display_name
# Write the data to the mongo datastore
new_course = modulestore('split').create_course(
org, prettyid, user_id, metadata=metadata, course_data=data, id_root=prettyid,
master_version=kwargs.get('master_version'))
return new_course
@classmethod
def _build(cls, target_class, *args, **kwargs):
raise NotImplementedError()
class ItemFactory(factory.Factory):
FACTORY_FOR = XModuleDescriptor
category = 'chapter'
user_id = 'test_user'
display_name = factory.LazyAttributeSequence(lambda o, n: "{} {}".format(o.category, n))
# pylint: disable=W0613
@classmethod
def _create(cls, target_class, *args, **kwargs):
"""
Uses *kwargs*:
*parent_location* (required): the location of the course & possibly parent
*category* (defaults to 'chapter')
*data* (optional): the data for the item
definition_locator (optional): the DescriptorLocator for the definition this uses or branches
*display_name* (optional): the display name of the item
*metadata* (optional): dictionary of metadata attributes (display_name here takes
precedence over the above attr)
"""
metadata = kwargs.get('metadata', {})
if 'display_name' not in metadata and 'display_name' in kwargs:
metadata['display_name'] = kwargs['display_name']
return modulestore('split').create_item(kwargs['parent_location'], kwargs['category'],
kwargs['user_id'], definition_locator=kwargs.get('definition_locator'),
new_def_data=kwargs.get('data'), metadata=metadata)
@classmethod
def _build(cls, target_class, *args, **kwargs):
raise NotImplementedError()

539
common/lib/xmodule/xmodule/modulestore/tests/test_locators.py Normal file
View File

@@ -0,0 +1,539 @@
'''
Created on Mar 14, 2013
@author: dmitchell
'''
from unittest import TestCase
from nose.plugins.skip import SkipTest
from bson.objectid import ObjectId
from xmodule.modulestore.locator import Locator, CourseLocator, BlockUsageLocator
from xmodule.modulestore.exceptions import InvalidLocationError, \
InsufficientSpecificationError, OverSpecificationError
class LocatorTest(TestCase):
def test_cant_instantiate_abstract_class(self):
self.assertRaises(TypeError, Locator)
def test_course_constructor_overspecified(self):
self.assertRaises(
OverSpecificationError,
CourseLocator,
url='edx://edu.mit.eecs.6002x',
course_id='edu.harvard.history',
revision='published',
version_guid=ObjectId())
self.assertRaises(
OverSpecificationError,
CourseLocator,
url='edx://edu.mit.eecs.6002x',
course_id='edu.harvard.history',
version_guid=ObjectId())
self.assertRaises(
OverSpecificationError,
CourseLocator,
url='edx://edu.mit.eecs.6002x;published',
revision='draft')
self.assertRaises(
OverSpecificationError,
CourseLocator,
course_id='edu.mit.eecs.6002x;published',
revision='draft')
def test_course_constructor_underspecified(self):
self.assertRaises(InsufficientSpecificationError, CourseLocator)
self.assertRaises(InsufficientSpecificationError, CourseLocator, revision='published')
def test_course_constructor_bad_version_guid(self):
self.assertRaises(ValueError, CourseLocator, version_guid="012345")
self.assertRaises(InsufficientSpecificationError, CourseLocator, version_guid=None)
def test_course_constructor_version_guid(self):
# generate a random location
test_id_1 = ObjectId()
test_id_1_loc = str(test_id_1)
testobj_1 = CourseLocator(version_guid=test_id_1)
self.check_course_locn_fields(testobj_1, 'version_guid', version_guid=test_id_1)
self.assertEqual(str(testobj_1.version_guid), test_id_1_loc)
self.assertEqual(str(testobj_1), '@' + test_id_1_loc)
self.assertEqual(testobj_1.url(), 'edx://@' + test_id_1_loc)
# Test using a given string
test_id_2_loc = '519665f6223ebd6980884f2b'
test_id_2 = ObjectId(test_id_2_loc)
testobj_2 = CourseLocator(version_guid=test_id_2)
self.check_course_locn_fields(testobj_2, 'version_guid', version_guid=test_id_2)
self.assertEqual(str(testobj_2.version_guid), test_id_2_loc)
self.assertEqual(str(testobj_2), '@' + test_id_2_loc)
self.assertEqual(testobj_2.url(), 'edx://@' + test_id_2_loc)
def test_course_constructor_bad_course_id(self):
"""
Test all sorts of badly-formed course_ids (and urls with those course_ids)
"""
for bad_id in ('edu.mit.',
' edu.mit.eecs',
'edu.mit.eecs ',
'@edu.mit.eecs',
'#edu.mit.eecs',
'edu.mit.ee cs',
'edu.mit.ee,cs',
'edu.mit.ee/cs',
'edu.mit.ee$cs',
'edu.mit.ee&cs',
'edu.mit.ee()cs',
';this',
'edu.mit.eecs;',
'edu.mit.eecs;this;that',
'edu.mit.eecs;this;',
'edu.mit.eecs;this ',
'edu.mit.eecs;th%is ',
):
self.assertRaises(AssertionError, CourseLocator, course_id=bad_id)
self.assertRaises(AssertionError, CourseLocator, url='edx://' + bad_id)
def test_course_constructor_bad_url(self):
for bad_url in ('edx://',
'edx:/edu.mit.eecs',
'http://edu.mit.eecs',
'edu.mit.eecs',
'edx//edu.mit.eecs'):
self.assertRaises(AssertionError, CourseLocator, url=bad_url)
def test_course_constructor_redundant_001(self):
testurn = 'edu.mit.eecs.6002x'
testobj = CourseLocator(course_id=testurn, url='edx://' + testurn)
self.check_course_locn_fields(testobj, 'course_id', course_id=testurn)
def test_course_constructor_redundant_002(self):
testurn = 'edu.mit.eecs.6002x;published'
expected_urn = 'edu.mit.eecs.6002x'
expected_rev = 'published'
testobj = CourseLocator(course_id=testurn, url='edx://' + testurn)
self.check_course_locn_fields(testobj, 'course_id',
course_id=expected_urn,
revision=expected_rev)
def test_course_constructor_course_id_no_revision(self):
testurn = 'edu.mit.eecs.6002x'
testobj = CourseLocator(course_id=testurn)
self.check_course_locn_fields(testobj, 'course_id', course_id=testurn)
self.assertEqual(testobj.course_id, testurn)
self.assertEqual(str(testobj), testurn)
self.assertEqual(testobj.url(), 'edx://' + testurn)
def test_course_constructor_course_id_with_revision(self):
testurn = 'edu.mit.eecs.6002x;published'
expected_id = 'edu.mit.eecs.6002x'
expected_revision = 'published'
testobj = CourseLocator(course_id=testurn)
self.check_course_locn_fields(testobj, 'course_id with revision',
course_id=expected_id,
revision=expected_revision,
)
self.assertEqual(testobj.course_id, expected_id)
self.assertEqual(testobj.revision, expected_revision)
self.assertEqual(str(testobj), testurn)
self.assertEqual(testobj.url(), 'edx://' + testurn)
def test_course_constructor_course_id_separate_revision(self):
test_id = 'edu.mit.eecs.6002x'
test_revision = 'published'
expected_urn = 'edu.mit.eecs.6002x;published'
testobj = CourseLocator(course_id=test_id, revision=test_revision)
self.check_course_locn_fields(testobj, 'course_id with separate revision',
course_id=test_id,
revision=test_revision,
)
self.assertEqual(testobj.course_id, test_id)
self.assertEqual(testobj.revision, test_revision)
self.assertEqual(str(testobj), expected_urn)
self.assertEqual(testobj.url(), 'edx://' + expected_urn)
def test_course_constructor_course_id_repeated_revision(self):
"""
The same revision appears in the course_id and the revision field.
"""
test_id = 'edu.mit.eecs.6002x;published'
test_revision = 'published'
expected_id = 'edu.mit.eecs.6002x'
expected_urn = 'edu.mit.eecs.6002x;published'
testobj = CourseLocator(course_id=test_id, revision=test_revision)
self.check_course_locn_fields(testobj, 'course_id with repeated revision',
course_id=expected_id,
revision=test_revision,
)
self.assertEqual(testobj.course_id, expected_id)
self.assertEqual(testobj.revision, test_revision)
self.assertEqual(str(testobj), expected_urn)
self.assertEqual(testobj.url(), 'edx://' + expected_urn)
def test_block_constructor(self):
testurn = 'edu.mit.eecs.6002x;published#HW3'
expected_id = 'edu.mit.eecs.6002x'
expected_revision = 'published'
expected_block_ref = 'HW3'
testobj = BlockUsageLocator(course_id=testurn)
self.check_block_locn_fields(testobj, 'test_block constructor',
course_id=expected_id,
revision=expected_revision,
block=expected_block_ref)
self.assertEqual(str(testobj), testurn)
self.assertEqual(testobj.url(), 'edx://' + testurn)
# ------------------------------------------------------------
# Disabled tests
def test_course_urls(self):
'''
Test constructor and property accessors.
'''
raise SkipTest()
self.assertRaises(TypeError, CourseLocator, 'empty constructor')
# url inits
testurn = 'edx://org/course/category/name'
self.assertRaises(InvalidLocationError, CourseLocator, url=testurn)
testurn = 'unknown/versionid/blockid'
self.assertRaises(InvalidLocationError, CourseLocator, url=testurn)
testurn = 'cvx/versionid'
testobj = CourseLocator(testurn)
self.check_course_locn_fields(testobj, testurn, 'versionid')
self.assertEqual(testobj, CourseLocator(testobj),
'initialization from another instance')
testurn = 'cvx/versionid/'
testobj = CourseLocator(testurn)
self.check_course_locn_fields(testobj, testurn, 'versionid')
testurn = 'cvx/versionid/blockid'
testobj = CourseLocator(testurn)
self.check_course_locn_fields(testobj, testurn, 'versionid')
testurn = 'cvx/versionid/blockid/extraneousstuff?including=args'
testobj = CourseLocator(testurn)
self.check_course_locn_fields(testobj, testurn, 'versionid')
testurn = 'cvx://versionid/blockid'
testobj = CourseLocator(testurn)
self.check_course_locn_fields(testobj, testurn, 'versionid')
testurn = 'crx/courseid/blockid'
testobj = CourseLocator(testurn)
self.check_course_locn_fields(testobj, testurn, course_id='courseid')
testurn = 'crx/courseid@revision/blockid'
testobj = CourseLocator(testurn)
self.check_course_locn_fields(testobj, testurn, course_id='courseid',
revision='revision')
self.assertEqual(testobj, CourseLocator(testobj),
'run initialization from another instance')
def test_course_keyword_setters(self):
raise SkipTest()
# arg list inits
testobj = CourseLocator(version_guid='versionid')
self.check_course_locn_fields(testobj, 'versionid arg', 'versionid')
testobj = CourseLocator(course_id='courseid')
self.check_course_locn_fields(testobj, 'courseid arg',
course_id='courseid')
testobj = CourseLocator(course_id='courseid', revision='rev')
self.check_course_locn_fields(testobj, 'rev arg',
course_id='courseid',
revision='rev')
# ignores garbage
testobj = CourseLocator(course_id='courseid', revision='rev',
potato='spud')
self.check_course_locn_fields(testobj, 'extra keyword arg',
course_id='courseid',
revision='rev')
# url w/ keyword override
testurn = 'crx/courseid@revision/blockid'
testobj = CourseLocator(testurn, revision='rev')
self.check_course_locn_fields(testobj, 'rev override',
course_id='courseid',
revision='rev')
def test_course_dict(self):
raise SkipTest()
# dict init w/ keyword overwrites
testobj = CourseLocator({"version_guid": 'versionid'})
self.check_course_locn_fields(testobj, 'versionid dict', 'versionid')
testobj = CourseLocator({"course_id": 'courseid'})
self.check_course_locn_fields(testobj, 'courseid dict',
course_id='courseid')
testobj = CourseLocator({"course_id": 'courseid', "revision": 'rev'})
self.check_course_locn_fields(testobj, 'rev dict',
course_id='courseid',
revision='rev')
# ignores garbage
testobj = CourseLocator({"course_id": 'courseid', "revision": 'rev',
"potato": 'spud'})
self.check_course_locn_fields(testobj, 'extra keyword dict',
course_id='courseid',
revision='rev')
testobj = CourseLocator({"course_id": 'courseid', "revision": 'rev'},
revision='alt')
self.check_course_locn_fields(testobj, 'rev dict',
course_id='courseid',
revision='alt')
# urn init w/ dict & keyword overwrites
testobj = CourseLocator('crx/notcourse@notthis',
{"course_id": 'courseid'},
revision='alt')
self.check_course_locn_fields(testobj, 'rev dict',
course_id='courseid',
revision='alt')
def test_url(self):
'''
Ensure CourseLocator generates expected urls.
'''
raise SkipTest()
testobj = CourseLocator(version_guid='versionid')
self.assertEqual(testobj.url(), 'cvx/versionid', 'versionid')
self.assertEqual(testobj, CourseLocator(testobj.url()),
'versionid conversion through url')
testobj = CourseLocator(course_id='courseid')
self.assertEqual(testobj.url(), 'crx/courseid', 'courseid')
self.assertEqual(testobj, CourseLocator(testobj.url()),
'courseid conversion through url')
testobj = CourseLocator(course_id='courseid', revision='rev')
self.assertEqual(testobj.url(), 'crx/courseid@rev', 'rev')
self.assertEqual(testobj, CourseLocator(testobj.url()),
'rev conversion through url')
def test_html(self):
'''
Ensure CourseLocator generates expected urls.
'''
raise SkipTest()
testobj = CourseLocator(version_guid='versionid')
self.assertEqual(testobj.html_id(), 'cvx/versionid', 'versionid')
self.assertEqual(testobj, CourseLocator(testobj.html_id()),
'versionid conversion through html_id')
testobj = CourseLocator(course_id='courseid')
self.assertEqual(testobj.html_id(), 'crx/courseid', 'courseid')
self.assertEqual(testobj, CourseLocator(testobj.html_id()),
'courseid conversion through html_id')
testobj = CourseLocator(course_id='courseid', revision='rev')
self.assertEqual(testobj.html_id(), 'crx/courseid%40rev', 'rev')
self.assertEqual(testobj, CourseLocator(testobj.html_id()),
'rev conversion through html_id')
def test_block_locator(self):
'''
Test constructor and property accessors.
'''
raise SkipTest()
self.assertIsInstance(BlockUsageLocator(), BlockUsageLocator,
'empty constructor')
# url inits
testurn = 'edx://org/course/category/name'
self.assertRaises(InvalidLocationError, BlockUsageLocator, testurn)
testurn = 'unknown/versionid/blockid'
self.assertRaises(InvalidLocationError, BlockUsageLocator, testurn)
testurn = 'cvx/versionid'
testobj = BlockUsageLocator(testurn)
self.check_block_locn_fields(testobj, testurn, 'versionid')
self.assertEqual(testobj, BlockUsageLocator(testobj),
'initialization from another instance')
testurn = 'cvx/versionid/'
testobj = BlockUsageLocator(testurn)
self.check_block_locn_fields(testobj, testurn, 'versionid')
testurn = 'cvx/versionid/blockid'
testobj = BlockUsageLocator(testurn)
self.check_block_locn_fields(testobj, testurn, 'versionid',
block='blockid')
testurn = 'cvx/versionid/blockid/extraneousstuff?including=args'
testobj = BlockUsageLocator(testurn)
self.check_block_locn_fields(testobj, testurn, 'versionid',
block='blockid')
testurn = 'cvx://versionid/blockid'
testobj = BlockUsageLocator(testurn)
self.check_block_locn_fields(testobj, testurn, 'versionid',
block='blockid')
testurn = 'crx/courseid/blockid'
testobj = BlockUsageLocator(testurn)
self.check_block_locn_fields(testobj, testurn, course_id='courseid',
block='blockid')
testurn = 'crx/courseid@revision/blockid'
testobj = BlockUsageLocator(testurn)
self.check_block_locn_fields(testobj, testurn, course_id='courseid',
revision='revision', block='blockid')
self.assertEqual(testobj, BlockUsageLocator(testobj),
'run initialization from another instance')
def test_block_keyword_init(self):
# arg list inits
raise SkipTest()
testobj = BlockUsageLocator(version_guid='versionid')
self.check_block_locn_fields(testobj, 'versionid arg', 'versionid')
testobj = BlockUsageLocator(version_guid='versionid', usage_id='myblock')
self.check_block_locn_fields(testobj, 'versionid arg', 'versionid',
block='myblock')
testobj = BlockUsageLocator(course_id='courseid')
self.check_block_locn_fields(testobj, 'courseid arg',
course_id='courseid')
testobj = BlockUsageLocator(course_id='courseid', revision='rev')
self.check_block_locn_fields(testobj, 'rev arg',
course_id='courseid',
revision='rev')
# ignores garbage
testobj = BlockUsageLocator(course_id='courseid', revision='rev',
usage_id='this_block', potato='spud')
self.check_block_locn_fields(testobj, 'extra keyword arg',
course_id='courseid', block='this_block', revision='rev')
# url w/ keyword override
testurn = 'crx/courseid@revision/blockid'
testobj = BlockUsageLocator(testurn, revision='rev')
self.check_block_locn_fields(testobj, 'rev override',
course_id='courseid', block='blockid',
revision='rev')
def test_block_keywords(self):
# dict init w/ keyword overwrites
raise SkipTest()
testobj = BlockUsageLocator({"version_guid": 'versionid',
'usage_id': 'dictblock'})
self.check_block_locn_fields(testobj, 'versionid dict', 'versionid',
block='dictblock')
testobj = BlockUsageLocator({"course_id": 'courseid',
'usage_id': 'dictblock'})
self.check_block_locn_fields(testobj, 'courseid dict',
block='dictblock', course_id='courseid')
testobj = BlockUsageLocator({"course_id": 'courseid', "revision": 'rev',
'usage_id': 'dictblock'})
self.check_block_locn_fields(testobj, 'rev dict',
course_id='courseid', block='dictblock',
revision='rev')
# ignores garbage
testobj = BlockUsageLocator({"course_id": 'courseid', "revision": 'rev',
'usage_id': 'dictblock', "potato": 'spud'})
self.check_block_locn_fields(testobj, 'extra keyword dict',
course_id='courseid', block='dictblock',
revision='rev')
testobj = BlockUsageLocator({"course_id": 'courseid', "revision": 'rev',
'usage_id': 'dictblock'}, revision='alt', usage_id='anotherblock')
self.check_block_locn_fields(testobj, 'rev dict',
course_id='courseid', block='anotherblock',
revision='alt')
# urn init w/ dict & keyword overwrites
testobj = BlockUsageLocator('crx/notcourse@notthis/northis',
{"course_id": 'courseid'}, revision='alt', usage_id='anotherblock')
self.check_block_locn_fields(testobj, 'rev dict',
course_id='courseid', block='anotherblock',
revision='alt')
def test_ensure_fully_specd(self):
'''
Test constructor and property accessors.
'''
raise SkipTest()
self.assertRaises(InsufficientSpecificationError,
BlockUsageLocator.ensure_fully_specified, BlockUsageLocator())
# url inits
testurn = 'edx://org/course/category/name'
self.assertRaises(InvalidLocationError,
BlockUsageLocator.ensure_fully_specified, testurn)
testurn = 'unknown/versionid/blockid'
self.assertRaises(InvalidLocationError,
BlockUsageLocator.ensure_fully_specified, testurn)
testurn = 'cvx/versionid'
self.assertRaises(InsufficientSpecificationError,
BlockUsageLocator.ensure_fully_specified, testurn)
testurn = 'cvx/versionid/'
self.assertRaises(InsufficientSpecificationError,
BlockUsageLocator.ensure_fully_specified, testurn)
testurn = 'cvx/versionid/blockid'
self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn),
BlockUsageLocator, testurn)
testurn = 'cvx/versionid/blockid/extraneousstuff?including=args'
self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn),
BlockUsageLocator, testurn)
testurn = 'cvx://versionid/blockid'
self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn),
BlockUsageLocator, testurn)
testurn = 'crx/courseid/blockid'
self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn),
BlockUsageLocator, testurn)
testurn = 'crx/courseid@revision/blockid'
self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn),
BlockUsageLocator, testurn)
def test_ensure_fully_via_keyword(self):
# arg list inits
raise SkipTest()
testobj = BlockUsageLocator(version_guid='versionid')
self.assertRaises(InsufficientSpecificationError,
BlockUsageLocator.ensure_fully_specified, testobj)
testurn = 'crx/courseid@revision/blockid'
testobj = BlockUsageLocator(version_guid='versionid', usage_id='myblock')
self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn),
BlockUsageLocator, testurn)
testobj = BlockUsageLocator(course_id='courseid')
self.assertRaises(InsufficientSpecificationError,
BlockUsageLocator.ensure_fully_specified, testobj)
testobj = BlockUsageLocator(course_id='courseid', revision='rev')
self.assertRaises(InsufficientSpecificationError,
BlockUsageLocator.ensure_fully_specified, testobj)
testobj = BlockUsageLocator(course_id='courseid', revision='rev',
usage_id='this_block')
self.assertIsInstance(BlockUsageLocator.ensure_fully_specified(testurn),
BlockUsageLocator, testurn)
# ------------------------------------------------------------------
# Utilities
def check_course_locn_fields(self, testobj, msg, version_guid=None,
course_id=None, revision=None):
self.assertEqual(testobj.version_guid, version_guid, msg)
self.assertEqual(testobj.course_id, course_id, msg)
self.assertEqual(testobj.revision, revision, msg)
def check_block_locn_fields(self, testobj, msg, version_guid=None,
course_id=None, revision=None, block=None):
self.check_course_locn_fields(testobj, msg, version_guid, course_id,
revision)
self.assertEqual(testobj.usage_id, block)

992
common/lib/xmodule/xmodule/modulestore/tests/test_split_modulestore.py Normal file
View File

@@ -0,0 +1,992 @@
'''
Created on Mar 25, 2013
@author: dmitchell
'''
import datetime
import subprocess
import unittest
import uuid
from importlib import import_module
from xblock.core import Scope
from xmodule.course_module import CourseDescriptor
from xmodule.modulestore.exceptions import InsufficientSpecificationError, ItemNotFoundError, VersionConflictError
from xmodule.modulestore.locator import CourseLocator, BlockUsageLocator, VersionTree, DescriptionLocator
from pytz import UTC
from path import path
import re
class SplitModuleTest(unittest.TestCase):
'''
The base set of tests manually populates a db w/ courses which have
versions. It creates unique collection names and removes them after all
tests finish.
'''
# Snippet of what would be in the django settings envs file
modulestore_options = {
'default_class': 'xmodule.raw_module.RawDescriptor',
'host': 'localhost',
'db': 'test_xmodule',
'collection': 'modulestore{0}'.format(uuid.uuid4().hex),
'fs_root': '',
}
MODULESTORE = {
'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore',
'OPTIONS': modulestore_options
}
# don't create django dependency; so, duplicates common.py in envs
match = re.search(r'(.*?/common)(?:$|/)', path(__file__))
COMMON_ROOT = match.group(1)
modulestore = None
# These version_guids correspond to values hard-coded in fixture files
# used for these tests. The files live in mitx/fixtures/splitmongo_json/*
GUID_D0 = "1d00000000000000dddd0000" # v12345d
GUID_D1 = "1d00000000000000dddd1111" # v12345d1
GUID_D2 = "1d00000000000000dddd2222" # v23456d
GUID_D3 = "1d00000000000000dddd3333" # v12345d0
GUID_D4 = "1d00000000000000dddd4444" # v23456d0
GUID_D5 = "1d00000000000000dddd5555" # v345679d
GUID_P = "1d00000000000000eeee0000" # v23456p
@staticmethod
def bootstrapDB():
'''
Loads the initial data into the db ensuring the collection name is
unique.
'''
collection_prefix = SplitModuleTest.MODULESTORE['OPTIONS']['collection'] + '.'
dbname = SplitModuleTest.MODULESTORE['OPTIONS']['db']
processes = [
subprocess.Popen([
'mongoimport', '-d', dbname, '-c',
collection_prefix + collection, '--jsonArray',
'--file',
SplitModuleTest.COMMON_ROOT + '/test/data/splitmongo_json/' + collection + '.json'
])
for collection in ('active_versions', 'structures', 'definitions')]
for p in processes:
if p.wait() != 0:
raise Exception("DB did not init correctly")
@classmethod
def tearDownClass(cls):
collection_prefix = SplitModuleTest.MODULESTORE['OPTIONS']['collection'] + '.'
if SplitModuleTest.modulestore:
for collection in ('active_versions', 'structures', 'definitions'):
modulestore().db.drop_collection(collection_prefix + collection)
# drop the modulestore to force re init
SplitModuleTest.modulestore = None
def findByIdInResult(self, collection, _id):
"""
Result is a collection of descriptors. Find the one whose block id
matches the _id.
"""
for element in collection:
if element.location.usage_id == _id:
return element
class SplitModuleCourseTests(SplitModuleTest):
'''
Course CRUD operation tests
'''
def test_get_courses(self):
courses = modulestore().get_courses('draft')
# should have gotten 3 draft courses
self.assertEqual(len(courses), 3, "Wrong number of courses")
# check metadata -- NOTE no promised order
course = self.findByIdInResult(courses, "head12345")
self.assertEqual(course.location.course_id, "GreekHero")
self.assertEqual(
str(course.location.version_guid), self.GUID_D0,
"course version mismatch"
)
self.assertEqual(course.category, 'course', 'wrong category')
self.assertEqual(len(course.tabs), 6, "wrong number of tabs")
self.assertEqual(
course.display_name, "The Ancient Greek Hero",
"wrong display name"
)
self.assertEqual(
course.advertised_start, "Fall 2013",
"advertised_start"
)
self.assertEqual(
len(course.children), 3,
"children")
self.assertEqual(course.definition_locator.definition_id, "head12345_12")
# check dates and graders--forces loading of descriptor
self.assertEqual(course.edited_by, "testassist@edx.org")
self.assertEqual(str(course.previous_version), self.GUID_D1)
self.assertDictEqual(course.grade_cutoffs, {"Pass": 0.45})
def test_revision_requests(self):
# query w/ revision qualifier (both draft and published)
courses_published = modulestore().get_courses('published')
self.assertEqual(len(courses_published), 1, len(courses_published))
course = self.findByIdInResult(courses_published, "head23456")
self.assertIsNotNone(course, "published courses")
self.assertEqual(course.location.course_id, "wonderful")
self.assertEqual(str(course.location.version_guid), self.GUID_P,
course.location.version_guid)
self.assertEqual(course.category, 'course', 'wrong category')
self.assertEqual(len(course.tabs), 4, "wrong number of tabs")
self.assertEqual(course.display_name, "The most wonderful course",
course.display_name)
self.assertIsNone(course.advertised_start)
self.assertEqual(len(course.children), 0,
"children")
def test_search_qualifiers(self):
# query w/ search criteria
courses = modulestore().get_courses('draft', qualifiers={'org': 'testx'})
self.assertEqual(len(courses), 2)
self.assertIsNotNone(self.findByIdInResult(courses, "head12345"))
self.assertIsNotNone(self.findByIdInResult(courses, "head23456"))
courses = modulestore().get_courses(
'draft',
qualifiers={'edited_on': {"$lt": datetime.datetime(2013, 3, 28, 15)}})
self.assertEqual(len(courses), 2)
courses = modulestore().get_courses(
'draft',
qualifiers={'org': 'testx', "prettyid": "test_course"})
self.assertEqual(len(courses), 1)
self.assertIsNotNone(self.findByIdInResult(courses, "head12345"))
def test_get_course(self):
'''
Test the various calling forms for get_course
'''
locator = CourseLocator(version_guid=self.GUID_D1)
course = modulestore().get_course(locator)
self.assertIsNone(course.location.course_id)
self.assertEqual(str(course.location.version_guid), self.GUID_D1)
self.assertEqual(course.category, 'course')
self.assertEqual(len(course.tabs), 6)
self.assertEqual(course.display_name, "The Ancient Greek Hero")
self.assertIsNone(course.advertised_start)
self.assertEqual(len(course.children), 0)
self.assertEqual(course.definition_locator.definition_id, "head12345_11")
# check dates and graders--forces loading of descriptor
self.assertEqual(course.edited_by, "testassist@edx.org")
self.assertDictEqual(course.grade_cutoffs, {"Pass": 0.55})
locator = CourseLocator(course_id='GreekHero', revision='draft')
course = modulestore().get_course(locator)
self.assertEqual(course.location.course_id, "GreekHero")
self.assertEqual(str(course.location.version_guid), self.GUID_D0)
self.assertEqual(course.category, 'course')
self.assertEqual(len(course.tabs), 6)
self.assertEqual(course.display_name, "The Ancient Greek Hero")
self.assertEqual(course.advertised_start, "Fall 2013")
self.assertEqual(len(course.children), 3)
# check dates and graders--forces loading of descriptor
self.assertEqual(course.edited_by, "testassist@edx.org")
self.assertDictEqual(course.grade_cutoffs, {"Pass": 0.45})
locator = CourseLocator(course_id='wonderful', revision='published')
course = modulestore().get_course(locator)
self.assertEqual(course.location.course_id, "wonderful")
self.assertEqual(str(course.location.version_guid), self.GUID_P)
locator = CourseLocator(course_id='wonderful', revision='draft')
course = modulestore().get_course(locator)
self.assertEqual(str(course.location.version_guid), self.GUID_D2)
def test_get_course_negative(self):
# Now negative testing
self.assertRaises(InsufficientSpecificationError,
modulestore().get_course, CourseLocator(course_id='edu.meh.blah'))
self.assertRaises(ItemNotFoundError,
modulestore().get_course, CourseLocator(course_id='nosuchthing', revision='draft'))
self.assertRaises(ItemNotFoundError,
modulestore().get_course,
CourseLocator(course_id='GreekHero', revision='published'))
def test_course_successors(self):
"""
get_course_successors(course_locator, version_history_depth=1)
"""
locator = CourseLocator(version_guid=self.GUID_D3)
result = modulestore().get_course_successors(locator)
self.assertIsInstance(result, VersionTree)
self.assertIsNone(result.locator.course_id)
self.assertEqual(str(result.locator.version_guid), self.GUID_D3)
self.assertEqual(len(result.children), 1)
self.assertEqual(str(result.children[0].locator.version_guid), self.GUID_D1)
self.assertEqual(len(result.children[0].children), 0, "descended more than one level")
result = modulestore().get_course_successors(locator, version_history_depth=2)
self.assertEqual(len(result.children), 1)
self.assertEqual(str(result.children[0].locator.version_guid), self.GUID_D1)
self.assertEqual(len(result.children[0].children), 1)
result = modulestore().get_course_successors(locator, version_history_depth=99)
self.assertEqual(len(result.children), 1)
self.assertEqual(str(result.children[0].locator.version_guid), self.GUID_D1)
self.assertEqual(len(result.children[0].children), 1)
class SplitModuleItemTests(SplitModuleTest):
'''
Item read tests including inheritance
'''
def test_has_item(self):
'''
has_item(BlockUsageLocator)
'''
# positive tests of various forms
locator = BlockUsageLocator(version_guid=self.GUID_D1, usage_id='head12345')
self.assertTrue(modulestore().has_item(locator),
"couldn't find in %s" % self.GUID_D1)
locator = BlockUsageLocator(course_id='GreekHero', usage_id='head12345', revision='draft')
self.assertTrue(
modulestore().has_item(locator),
"couldn't find in 12345"
)
self.assertTrue(
modulestore().has_item(BlockUsageLocator(
course_id=locator.course_id,
revision='draft',
usage_id=locator.usage_id
)),
"couldn't find in draft 12345"
)
self.assertFalse(
modulestore().has_item(BlockUsageLocator(
course_id=locator.course_id,
revision='published',
usage_id=locator.usage_id)),
"found in published 12345"
)
locator.revision = 'draft'
self.assertTrue(
modulestore().has_item(locator),
"not found in draft 12345"
)
# not a course obj
locator = BlockUsageLocator(course_id='GreekHero', usage_id='chapter1', revision='draft')
self.assertTrue(
modulestore().has_item(locator),
"couldn't find chapter1"
)
# in published course
locator = BlockUsageLocator(course_id="wonderful", usage_id="head23456", revision='draft')
self.assertTrue(modulestore().has_item(BlockUsageLocator(course_id=locator.course_id,
usage_id=locator.usage_id,
revision='published')),
"couldn't find in 23456")
locator.revision = 'published'
self.assertTrue(modulestore().has_item(locator), "couldn't find in 23456")
def test_negative_has_item(self):
# negative tests--not found
# no such course or block
locator = BlockUsageLocator(course_id="doesnotexist", usage_id="head23456", revision='draft')
self.assertFalse(modulestore().has_item(locator))
locator = BlockUsageLocator(course_id="wonderful", usage_id="doesnotexist", revision='draft')
self.assertFalse(modulestore().has_item(locator))
# negative tests--insufficient specification
self.assertRaises(InsufficientSpecificationError, BlockUsageLocator)
self.assertRaises(InsufficientSpecificationError,
modulestore().has_item, BlockUsageLocator(version_guid=self.GUID_D1))
self.assertRaises(InsufficientSpecificationError,
modulestore().has_item, BlockUsageLocator(course_id='GreekHero'))
def test_get_item(self):
'''
get_item(blocklocator)
'''
# positive tests of various forms
locator = BlockUsageLocator(version_guid=self.GUID_D1, usage_id='head12345')
block = modulestore().get_item(locator)
self.assertIsInstance(block, CourseDescriptor)
locator = BlockUsageLocator(course_id='GreekHero', usage_id='head12345', revision='draft')
block = modulestore().get_item(locator)
self.assertEqual(block.location.course_id, "GreekHero")
# look at this one in detail
self.assertEqual(len(block.tabs), 6, "wrong number of tabs")
self.assertEqual(block.display_name, "The Ancient Greek Hero")
self.assertEqual(block.advertised_start, "Fall 2013")
self.assertEqual(len(block.children), 3)
self.assertEqual(block.definition_locator.definition_id, "head12345_12")
# check dates and graders--forces loading of descriptor
self.assertEqual(block.edited_by, "testassist@edx.org")
self.assertDictEqual(
block.grade_cutoffs, {"Pass": 0.45},
)
# try to look up other revisions
self.assertRaises(ItemNotFoundError,
modulestore().get_item,
BlockUsageLocator(course_id=locator.as_course_locator(),
usage_id=locator.usage_id,
revision='published'))
locator.revision = 'draft'
self.assertIsInstance(
modulestore().get_item(locator),
CourseDescriptor
)
def test_get_non_root(self):
# not a course obj
locator = BlockUsageLocator(course_id='GreekHero', usage_id='chapter1', revision='draft')
block = modulestore().get_item(locator)
self.assertEqual(block.location.course_id, "GreekHero")
self.assertEqual(block.category, 'chapter')
self.assertEqual(block.definition_locator.definition_id, "chapter12345_1")
self.assertEqual(block.display_name, "Hercules")
self.assertEqual(block.edited_by, "testassist@edx.org")
# in published course
locator = BlockUsageLocator(course_id="wonderful", usage_id="head23456", revision='published')
self.assertIsInstance(
modulestore().get_item(locator),
CourseDescriptor
)
# negative tests--not found
# no such course or block
locator = BlockUsageLocator(course_id="doesnotexist", usage_id="head23456", revision='draft')
with self.assertRaises(ItemNotFoundError):
modulestore().get_item(locator)
locator = BlockUsageLocator(course_id="wonderful", usage_id="doesnotexist", revision='draft')
with self.assertRaises(ItemNotFoundError):
modulestore().get_item(locator)
# negative tests--insufficient specification
with self.assertRaises(InsufficientSpecificationError):
modulestore().get_item(BlockUsageLocator(version_guid=self.GUID_D1))
with self.assertRaises(InsufficientSpecificationError):
modulestore().get_item(BlockUsageLocator(course_id='GreekHero', revision='draft'))
# pylint: disable=W0212
def test_matching(self):
'''
test the block and value matches help functions
'''
self.assertTrue(modulestore()._value_matches('help', 'help'))
self.assertFalse(modulestore()._value_matches('help', 'Help'))
self.assertTrue(modulestore()._value_matches(['distract', 'help', 'notme'], 'help'))
self.assertFalse(modulestore()._value_matches(['distract', 'Help', 'notme'], 'help'))
self.assertFalse(modulestore()._value_matches({'field': ['distract', 'Help', 'notme']}, {'field': 'help'}))
self.assertFalse(modulestore()._value_matches(['distract', 'Help', 'notme'], {'field': 'help'}))
self.assertTrue(modulestore()._value_matches(
{'field': ['distract', 'help', 'notme'],
'irrelevant': 2},
{'field': 'help'}))
self.assertTrue(modulestore()._value_matches('I need some help', {'$regex': 'help'}))
self.assertTrue(modulestore()._value_matches(['I need some help', 'today'], {'$regex': 'help'}))
self.assertFalse(modulestore()._value_matches('I need some help', {'$regex': 'Help'}))
self.assertFalse(modulestore()._value_matches(['I need some help', 'today'], {'$regex': 'Help'}))
self.assertTrue(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 1}))
self.assertTrue(modulestore()._block_matches({'a': 1, 'b': 2}, {'c': None}))
self.assertTrue(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 1, 'c': None}))
self.assertFalse(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 2}))
self.assertFalse(modulestore()._block_matches({'a': 1, 'b': 2}, {'c': 1}))
self.assertFalse(modulestore()._block_matches({'a': 1, 'b': 2}, {'a': 1, 'c': 1}))
def test_get_items(self):
'''
get_items(locator, qualifiers, [revision])
'''
locator = CourseLocator(version_guid=self.GUID_D0)
# get all modules
matches = modulestore().get_items(locator, {})
self.assertEqual(len(matches), 6)
matches = modulestore().get_items(locator, {'category': 'chapter'})
self.assertEqual(len(matches), 3)
matches = modulestore().get_items(locator, {'category': 'garbage'})
self.assertEqual(len(matches), 0)
matches = modulestore().get_items(
locator,
{
'category': 'chapter',
'metadata': {'display_name': {'$regex': 'Hera'}}
}
)
self.assertEqual(len(matches), 2)
matches = modulestore().get_items(locator, {'children': 'chapter2'})
self.assertEqual(len(matches), 1)
self.assertEqual(matches[0].location.usage_id, 'head12345')
def test_get_parents(self):
'''
get_parent_locations(locator, [usage_id], [revision]): [BlockUsageLocator]
'''
locator = CourseLocator(course_id="GreekHero", revision='draft')
parents = modulestore().get_parent_locations(locator, usage_id='chapter1')
self.assertEqual(len(parents), 1)
self.assertEqual(parents[0].usage_id, 'head12345')
self.assertEqual(parents[0].course_id, "GreekHero")
locator.usage_id = 'chapter2'
parents = modulestore().get_parent_locations(locator)
self.assertEqual(len(parents), 1)
self.assertEqual(parents[0].usage_id, 'head12345')
parents = modulestore().get_parent_locations(locator, usage_id='nosuchblock')
self.assertEqual(len(parents), 0)
def test_get_children(self):
"""
Test the existing get_children method on xdescriptors
"""
locator = BlockUsageLocator(course_id="GreekHero", usage_id="head12345", revision='draft')
block = modulestore().get_item(locator)
children = block.get_children()
expected_ids = [
"chapter1", "chapter2", "chapter3"
]
for child in children:
self.assertEqual(child.category, "chapter")
self.assertIn(child.location.usage_id, expected_ids)
expected_ids.remove(child.location.usage_id)
self.assertEqual(len(expected_ids), 0)
class TestItemCrud(SplitModuleTest):
"""
Test create update and delete of items
"""
# TODO do I need to test this case which I believe won't work:
# 1) fetch a course and some of its blocks
# 2) do a series of CRUD operations on those previously fetched elements
# The problem here will be that the version_guid of the items will be the version at time of fetch.
# Each separate save will change the head version; so, the 2nd piecemeal change will flag the version
# conflict. That is, if versions are v0..vn and start as v0 in initial fetch, the first CRUD op will
# say it's changing an object from v0, splitMongo will process it and make the current head v1, the next
# crud op will pass in its v0 element and splitMongo will flag the version conflict.
# What I don't know is how realistic this test is and whether to wrap the modulestore with a higher level
# transactional operation which manages the version change or make the threading cache reason out whether or
# not the changes are independent and additive and thus non-conflicting.
# A use case I expect is
# (client) change this metadata
# (server) done, here's the new info which, btw, updates the course version to v1
# (client) add these children to this other node (which says it came from v0 or
# will the client have refreshed the version before doing the op?)
# In this case, having a server side transactional model won't help b/c the bug is a long-transaction on the
# on the client where it would be a mistake for the server to assume anything about client consistency. The best
# the server could do would be to see if the parent's children changed at all since v0.
def test_create_minimal_item(self):
"""
create_item(course_or_parent_locator, category, user, definition_locator=None, new_def_data=None,
metadata=None): new_desciptor
"""
# grab link to course to ensure new versioning works
locator = CourseLocator(course_id="GreekHero", revision='draft')
premod_course = modulestore().get_course(locator)
premod_time = datetime.datetime.now(UTC) - datetime.timedelta(seconds=1)
# add minimal one w/o a parent
category = 'sequential'
new_module = modulestore().create_item(
locator, category, 'user123',
metadata={'display_name': 'new sequential'}
)
# check that course version changed and course's previous is the other one
self.assertEqual(new_module.location.course_id, "GreekHero")
self.assertNotEqual(new_module.location.version_guid, premod_course.location.version_guid)
self.assertIsNone(locator.version_guid, "Version inadvertently filled in")
current_course = modulestore().get_course(locator)
self.assertEqual(new_module.location.version_guid, current_course.location.version_guid)
history_info = modulestore().get_course_history_info(current_course.location)
self.assertEqual(history_info['previous_version'], premod_course.location.version_guid)
self.assertEqual(str(history_info['original_version']), self.GUID_D3)
self.assertEqual(history_info['edited_by'], "user123")
self.assertGreaterEqual(history_info['edited_on'], premod_time)
self.assertLessEqual(history_info['edited_on'], datetime.datetime.now(UTC))
# check block's info: category, definition_locator, and display_name
self.assertEqual(new_module.category, 'sequential')
self.assertIsNotNone(new_module.definition_locator)
self.assertEqual(new_module.display_name, 'new sequential')
# check that block does not exist in previous version
locator = BlockUsageLocator(
version_guid=premod_course.location.version_guid,
usage_id=new_module.location.usage_id
)
self.assertRaises(ItemNotFoundError, modulestore().get_item, locator)
def test_create_parented_item(self):
"""
Test create_item w/ specifying the parent of the new item
"""
locator = BlockUsageLocator(course_id="wonderful", usage_id="head23456", revision='draft')
premod_course = modulestore().get_course(locator)
category = 'chapter'
new_module = modulestore().create_item(
locator, category, 'user123',
metadata={'display_name': 'new chapter'},
definition_locator=DescriptionLocator("chapter12345_2")
)
# check that course version changed and course's previous is the other one
self.assertNotEqual(new_module.location.version_guid, premod_course.location.version_guid)
parent = modulestore().get_item(locator)
self.assertIn(new_module.location.usage_id, parent.children)
self.assertEqual(new_module.definition_locator.definition_id, "chapter12345_2")
def test_unique_naming(self):
"""
Check that 2 modules of same type get unique usage_ids. Also check that if creation provides
a definition id and new def data that it branches the definition in the db.
Actually, this tries to test all create_item features not tested above.
"""
locator = BlockUsageLocator(course_id="contender", usage_id="head345679", revision='draft')
category = 'problem'
premod_time = datetime.datetime.now(UTC) - datetime.timedelta(seconds=1)
new_payload = "<problem>empty</problem>"
new_module = modulestore().create_item(
locator, category, 'anotheruser',
metadata={'display_name': 'problem 1'},
new_def_data=new_payload
)
another_payload = "<problem>not empty</problem>"
another_module = modulestore().create_item(
locator, category, 'anotheruser',
metadata={'display_name': 'problem 2'},
definition_locator=DescriptionLocator("problem12345_3_1"),
new_def_data=another_payload
)
# check that course version changed and course's previous is the other one
parent = modulestore().get_item(locator)
self.assertNotEqual(new_module.location.usage_id, another_module.location.usage_id)
self.assertIn(new_module.location.usage_id, parent.children)
self.assertIn(another_module.location.usage_id, parent.children)
self.assertEqual(new_module.data, new_payload)
self.assertEqual(another_module.data, another_payload)
# check definition histories
new_history = modulestore().get_definition_history_info(new_module.definition_locator)
self.assertIsNone(new_history['previous_version'])
self.assertEqual(new_history['original_version'], new_module.definition_locator.definition_id)
self.assertEqual(new_history['edited_by'], "anotheruser")
self.assertLessEqual(new_history['edited_on'], datetime.datetime.now(UTC))
self.assertGreaterEqual(new_history['edited_on'], premod_time)
another_history = modulestore().get_definition_history_info(another_module.definition_locator)
self.assertEqual(another_history['previous_version'], 'problem12345_3_1')
# TODO check that default fields are set
def test_update_metadata(self):
"""
test updating an items metadata ensuring the definition doesn't version but the course does if it should
"""
locator = BlockUsageLocator(course_id="GreekHero", usage_id="problem3_2", revision='draft')
problem = modulestore().get_item(locator)
pre_def_id = problem.definition_locator.definition_id
pre_version_guid = problem.location.version_guid
self.assertIsNotNone(pre_def_id)
self.assertIsNotNone(pre_version_guid)
premod_time = datetime.datetime.now(UTC) - datetime.timedelta(seconds=1)
self.assertNotEqual(problem.max_attempts, 4, "Invalidates rest of test")
problem.max_attempts = 4
updated_problem = modulestore().update_item(problem, 'changeMaven')
# check that course version changed and course's previous is the other one
self.assertEqual(updated_problem.definition_locator.definition_id, pre_def_id)
self.assertNotEqual(updated_problem.location.version_guid, pre_version_guid)
self.assertEqual(updated_problem.max_attempts, 4)
# refetch to ensure original didn't change
original_location = BlockUsageLocator(
version_guid=pre_version_guid,
usage_id=problem.location.usage_id
)
problem = modulestore().get_item(original_location)
self.assertNotEqual(problem.max_attempts, 4, "original changed")
current_course = modulestore().get_course(locator)
self.assertEqual(updated_problem.location.version_guid, current_course.location.version_guid)
history_info = modulestore().get_course_history_info(current_course.location)
self.assertEqual(history_info['previous_version'], pre_version_guid)
self.assertEqual(str(history_info['original_version']), self.GUID_D3)
self.assertEqual(history_info['edited_by'], "changeMaven")
self.assertGreaterEqual(history_info['edited_on'], premod_time)
self.assertLessEqual(history_info['edited_on'], datetime.datetime.now(UTC))
def test_update_children(self):
"""
test updating an item's children ensuring the definition doesn't version but the course does if it should
"""
locator = BlockUsageLocator(course_id="GreekHero", usage_id="chapter3", revision='draft')
block = modulestore().get_item(locator)
pre_def_id = block.definition_locator.definition_id
pre_version_guid = block.location.version_guid
# reorder children
self.assertGreater(len(block.children), 0, "meaningless test")
moved_child = block.children.pop()
updated_problem = modulestore().update_item(block, 'childchanger')
# check that course version changed and course's previous is the other one
self.assertEqual(updated_problem.definition_locator.definition_id, pre_def_id)
self.assertNotEqual(updated_problem.location.version_guid, pre_version_guid)
self.assertEqual(updated_problem.children, block.children)
self.assertNotIn(moved_child, updated_problem.children)
locator.usage_id = "chapter1"
other_block = modulestore().get_item(locator)
other_block.children.append(moved_child)
other_updated = modulestore().update_item(other_block, 'childchanger')
self.assertIn(moved_child, other_updated.children)
def test_update_definition(self):
"""
test updating an item's definition: ensure it gets versioned as well as the course getting versioned
"""
locator = BlockUsageLocator(course_id="GreekHero", usage_id="head12345", revision='draft')
block = modulestore().get_item(locator)
pre_def_id = block.definition_locator.definition_id
pre_version_guid = block.location.version_guid
block.grading_policy['GRADER'][0]['min_count'] = 13
updated_block = modulestore().update_item(block, 'definition_changer')
self.assertNotEqual(updated_block.definition_locator.definition_id, pre_def_id)
self.assertNotEqual(updated_block.location.version_guid, pre_version_guid)
self.assertEqual(updated_block.grading_policy['GRADER'][0]['min_count'], 13)
def test_update_manifold(self):
"""
Test updating metadata, children, and definition in a single call ensuring all the versioning occurs
"""
# first add 2 children to the course for the update to manipulate
locator = BlockUsageLocator(course_id="contender", usage_id="head345679", revision='draft')
category = 'problem'
new_payload = "<problem>empty</problem>"
modulestore().create_item(
locator, category, 'test_update_manifold',
metadata={'display_name': 'problem 1'},
new_def_data=new_payload
)
another_payload = "<problem>not empty</problem>"
modulestore().create_item(
locator, category, 'test_update_manifold',
metadata={'display_name': 'problem 2'},
definition_locator=DescriptionLocator("problem12345_3_1"),
new_def_data=another_payload
)
# pylint: disable=W0212
modulestore()._clear_cache()
# now begin the test
block = modulestore().get_item(locator)
pre_def_id = block.definition_locator.definition_id
pre_version_guid = block.location.version_guid
self.assertNotEqual(block.grading_policy['GRADER'][0]['min_count'], 13)
block.grading_policy['GRADER'][0]['min_count'] = 13
block.children = block.children[1:] + [block.children[0]]
block.advertised_start = "Soon"
updated_block = modulestore().update_item(block, "test_update_manifold")
self.assertNotEqual(updated_block.definition_locator.definition_id, pre_def_id)
self.assertNotEqual(updated_block.location.version_guid, pre_version_guid)
self.assertEqual(updated_block.grading_policy['GRADER'][0]['min_count'], 13)
self.assertEqual(updated_block.children[0], block.children[0])
self.assertEqual(updated_block.advertised_start, "Soon")
def test_delete_item(self):
course = self.create_course_for_deletion()
self.assertRaises(ValueError,
modulestore().delete_item,
course.location,
'deleting_user')
reusable_location = BlockUsageLocator(
course_id=course.location.course_id,
usage_id=course.location.usage_id,
revision='draft')
# delete a leaf
problems = modulestore().get_items(reusable_location, {'category': 'problem'})
locn_to_del = problems[0].location
new_course_loc = modulestore().delete_item(locn_to_del, 'deleting_user')
deleted = BlockUsageLocator(course_id=reusable_location.course_id,
revision=reusable_location.revision,
usage_id=locn_to_del.usage_id)
self.assertFalse(modulestore().has_item(deleted))
self.assertRaises(VersionConflictError, modulestore().has_item, locn_to_del)
locator = BlockUsageLocator(
version_guid=locn_to_del.version_guid,
usage_id=locn_to_del.usage_id
)
self.assertTrue(modulestore().has_item(locator))
self.assertNotEqual(new_course_loc.version_guid, course.location.version_guid)
# delete a subtree
nodes = modulestore().get_items(reusable_location, {'category': 'chapter'})
new_course_loc = modulestore().delete_item(nodes[0].location, 'deleting_user')
# check subtree
def check_subtree(node):
if node:
node_loc = node.location
self.assertFalse(modulestore().has_item(
BlockUsageLocator(
course_id=node_loc.course_id,
revision=node_loc.revision,
usage_id=node.location.usage_id)))
locator = BlockUsageLocator(
version_guid=node.location.version_guid,
usage_id=node.location.usage_id)
self.assertTrue(modulestore().has_item(locator))
if node.has_children:
for sub in node.get_children():
check_subtree(sub)
check_subtree(nodes[0])
def create_course_for_deletion(self):
course = modulestore().create_course('nihilx', 'deletion', 'deleting_user')
root = BlockUsageLocator(
course_id=course.location.course_id,
usage_id=course.location.usage_id,
revision='draft')
for _ in range(4):
self.create_subtree_for_deletion(root, ['chapter', 'vertical', 'problem'])
return modulestore().get_item(root)
def create_subtree_for_deletion(self, parent, category_queue):
if not category_queue:
return
node = modulestore().create_item(parent, category_queue[0], 'deleting_user')
node_loc = BlockUsageLocator(parent.as_course_locator(), usage_id=node.location.usage_id)
for _ in range(4):
self.create_subtree_for_deletion(node_loc, category_queue[1:])
class TestCourseCreation(SplitModuleTest):
"""
Test create_course, duh :-)
"""
def test_simple_creation(self):
"""
The simplest case but probing all expected results from it.
"""
# Oddly getting differences of 200nsec
pre_time = datetime.datetime.now(UTC) - datetime.timedelta(milliseconds=1)
new_course = modulestore().create_course('test_org', 'test_course', 'create_user')
new_locator = new_course.location
# check index entry
index_info = modulestore().get_course_index_info(new_locator)
self.assertEqual(index_info['org'], 'test_org')
self.assertEqual(index_info['prettyid'], 'test_course')
self.assertGreaterEqual(index_info["edited_on"], pre_time)
self.assertLessEqual(index_info["edited_on"], datetime.datetime.now(UTC))
self.assertEqual(index_info['edited_by'], 'create_user')
# check structure info
structure_info = modulestore().get_course_history_info(new_locator)
self.assertEqual(structure_info['original_version'], index_info['versions']['draft'])
self.assertIsNone(structure_info['previous_version'])
self.assertGreaterEqual(structure_info["edited_on"], pre_time)
self.assertLessEqual(structure_info["edited_on"], datetime.datetime.now(UTC))
self.assertEqual(structure_info['edited_by'], 'create_user')
# check the returned course object
self.assertIsInstance(new_course, CourseDescriptor)
self.assertEqual(new_course.category, 'course')
self.assertFalse(new_course.show_calculator)
self.assertTrue(new_course.allow_anonymous)
self.assertEqual(len(new_course.children), 0)
self.assertEqual(new_course.edited_by, "create_user")
self.assertEqual(len(new_course.grading_policy['GRADER']), 4)
self.assertDictEqual(new_course.grade_cutoffs, {"Pass": 0.5})
def test_cloned_course(self):
"""
Test making a course which points to an existing draft and published but not making any changes to either.
"""
pre_time = datetime.datetime.now(UTC)
original_locator = CourseLocator(course_id="wonderful", revision='draft')
original_index = modulestore().get_course_index_info(original_locator)
new_draft = modulestore().create_course(
'leech', 'best_course', 'leech_master', id_root='best',
versions_dict=original_index['versions'])
new_draft_locator = new_draft.location
self.assertRegexpMatches(new_draft_locator.course_id, r'best.*')
# the edited_by and other meta fields on the new course will be the original author not this one
self.assertEqual(new_draft.edited_by, 'test@edx.org')
self.assertLess(new_draft.edited_on, pre_time)
self.assertEqual(new_draft.location.version_guid, original_index['versions']['draft'])
# however the edited_by and other meta fields on course_index will be this one
new_index = modulestore().get_course_index_info(new_draft_locator)
self.assertGreaterEqual(new_index["edited_on"], pre_time)
self.assertLessEqual(new_index["edited_on"], datetime.datetime.now(UTC))
self.assertEqual(new_index['edited_by'], 'leech_master')
new_published_locator = CourseLocator(course_id=new_draft_locator.course_id, revision='published')
new_published = modulestore().get_course(new_published_locator)
self.assertEqual(new_published.edited_by, 'test@edx.org')
self.assertLess(new_published.edited_on, pre_time)
self.assertEqual(new_published.location.version_guid, original_index['versions']['published'])
# changing this course will not change the original course
# using new_draft.location will insert the chapter under the course root
new_item = modulestore().create_item(
new_draft.location, 'chapter', 'leech_master',
metadata={'display_name': 'new chapter'}
)
new_draft_locator.version_guid = None
new_index = modulestore().get_course_index_info(new_draft_locator)
self.assertNotEqual(new_index['versions']['draft'], original_index['versions']['draft'])
new_draft = modulestore().get_course(new_draft_locator)
self.assertEqual(new_item.edited_by, 'leech_master')
self.assertGreaterEqual(new_item.edited_on, pre_time)
self.assertNotEqual(new_item.location.version_guid, original_index['versions']['draft'])
self.assertNotEqual(new_draft.location.version_guid, original_index['versions']['draft'])
structure_info = modulestore().get_course_history_info(new_draft_locator)
self.assertGreaterEqual(structure_info["edited_on"], pre_time)
self.assertLessEqual(structure_info["edited_on"], datetime.datetime.now(UTC))
self.assertEqual(structure_info['edited_by'], 'leech_master')
original_course = modulestore().get_course(original_locator)
self.assertEqual(original_course.location.version_guid, original_index['versions']['draft'])
self.assertFalse(
modulestore().has_item(BlockUsageLocator(
original_locator,
usage_id=new_item.location.usage_id
))
)
def test_derived_course(self):
"""
Create a new course which overrides metadata and course_data
"""
pre_time = datetime.datetime.now(UTC)
original_locator = CourseLocator(course_id="contender", revision='draft')
original = modulestore().get_course(original_locator)
original_index = modulestore().get_course_index_info(original_locator)
data_payload = {}
metadata_payload = {}
for field in original.fields:
if field.scope == Scope.content and field.name != 'location':
data_payload[field.name] = getattr(original, field.name)
elif field.scope == Scope.settings:
metadata_payload[field.name] = getattr(original, field.name)
data_payload['grading_policy']['GRADE_CUTOFFS'] = {'A': .9, 'B': .8, 'C': .65}
metadata_payload['display_name'] = 'Derivative'
new_draft = modulestore().create_course(
'leech', 'derivative', 'leech_master', id_root='counter',
versions_dict={'draft': original_index['versions']['draft']},
course_data=data_payload,
metadata=metadata_payload
)
new_draft_locator = new_draft.location
self.assertRegexpMatches(new_draft_locator.course_id, r'counter.*')
# the edited_by and other meta fields on the new course will be the original author not this one
self.assertEqual(new_draft.edited_by, 'leech_master')
self.assertGreaterEqual(new_draft.edited_on, pre_time)
self.assertNotEqual(new_draft.location.version_guid, original_index['versions']['draft'])
# however the edited_by and other meta fields on course_index will be this one
new_index = modulestore().get_course_index_info(new_draft_locator)
self.assertGreaterEqual(new_index["edited_on"], pre_time)
self.assertLessEqual(new_index["edited_on"], datetime.datetime.now(UTC))
self.assertEqual(new_index['edited_by'], 'leech_master')
self.assertEqual(new_draft.display_name, metadata_payload['display_name'])
self.assertDictEqual(
new_draft.grading_policy['GRADE_CUTOFFS'],
data_payload['grading_policy']['GRADE_CUTOFFS']
)
def test_update_course_index(self):
"""
Test changing the org, pretty id, etc of a course. Test that it doesn't allow changing the id, etc.
"""
locator = CourseLocator(course_id="GreekHero", revision='draft')
modulestore().update_course_index(locator, {'org': 'funkyU'})
course_info = modulestore().get_course_index_info(locator)
self.assertEqual(course_info['org'], 'funkyU')
modulestore().update_course_index(locator, {'org': 'moreFunky', 'prettyid': 'Ancient Greek Demagods'})
course_info = modulestore().get_course_index_info(locator)
self.assertEqual(course_info['org'], 'moreFunky')
self.assertEqual(course_info['prettyid'], 'Ancient Greek Demagods')
self.assertRaises(ValueError, modulestore().update_course_index, locator, {'_id': 'funkygreeks'})
with self.assertRaises(ValueError):
modulestore().update_course_index(
locator,
{'edited_on': datetime.datetime.now(UTC)}
)
with self.assertRaises(ValueError):
modulestore().update_course_index(
locator,
{'edited_by': 'sneak'}
)
self.assertRaises(ValueError, modulestore().update_course_index, locator,
{'versions': {'draft': self.GUID_D1}})
# an allowed but not necessarily recommended way to revert the draft version
versions = course_info['versions']
versions['draft'] = self.GUID_D1
modulestore().update_course_index(locator, {'versions': versions}, update_versions=True)
course = modulestore().get_course(locator)
self.assertEqual(str(course.location.version_guid), self.GUID_D1)
# an allowed but not recommended way to publish a course
versions['published'] = self.GUID_D1
modulestore().update_course_index(locator, {'versions': versions}, update_versions=True)
course = modulestore().get_course(CourseLocator(course_id=locator.course_id, revision="published"))
self.assertEqual(str(course.location.version_guid), self.GUID_D1)
class TestInheritance(SplitModuleTest):
"""
Test the metadata inheritance mechanism.
"""
def test_inheritance(self):
"""
The actual test
"""
# Note, not testing value where defined (course) b/c there's no
# defined accessor for it on CourseDescriptor.
locator = BlockUsageLocator(course_id="GreekHero", usage_id="problem3_2", revision='draft')
node = modulestore().get_item(locator)
# inherited
self.assertEqual(node.graceperiod, datetime.timedelta(hours=2))
locator = BlockUsageLocator(course_id="GreekHero", usage_id="problem1", revision='draft')
node = modulestore().get_item(locator)
# overridden
self.assertEqual(node.graceperiod, datetime.timedelta(hours=4))
# TODO test inheritance after set and delete of attrs
#===========================================
# This mocks the django.modulestore() function and is intended purely to disentangle
# the tests from django
def modulestore():
def load_function(path):
module_path, _, name = path.rpartition('.')
return getattr(import_module(module_path), name)
if SplitModuleTest.modulestore is None:
SplitModuleTest.bootstrapDB()
class_ = load_function(SplitModuleTest.MODULESTORE['ENGINE'])
options = {}
options.update(SplitModuleTest.MODULESTORE['OPTIONS'])
options['render_template'] = render_to_template_mock
# pylint: disable=W0142
SplitModuleTest.modulestore = class_(**options)
return SplitModuleTest.modulestore
# pylint: disable=W0613
def render_to_template_mock(*args):
pass

26
common/lib/xmodule/xmodule/modulestore/xml_importer.py
View File

@@ -138,7 +138,7 @@ def import_module_from_xml(modulestore, static_content_store, course_data_path,
# For example, what I'm seeing is <img src='foo.jpg' /> -> <img src='bar.jpg'>
# Note the dropped element closing tag. This causes the LMS to fail when rendering modules - that's
# no good, so we have to do this kludge
if isinstance(module.data, str) or isinstance(module.data, unicode): # some module 'data' fields are non strings which blows up the link traversal code
if isinstance(module.data, str) or isinstance(module.data, unicode): # some module 'data' fields are non strings which blows up the link traversal code
lxml_rewrite_links(module.data, lambda link: verify_content_links(module, course_data_path, static_content_store, link, remap_dict))
for key in remap_dict.keys():
@@ -315,7 +315,7 @@ def import_module(module, store, course_data_path, static_content_store, allow_n
# For example, what I'm seeing is <img src='foo.jpg' /> -> <img src='bar.jpg'>
# Note the dropped element closing tag. This causes the LMS to fail when rendering modules - that's
# no good, so we have to do this kludge
if isinstance(module_data, str) or isinstance(module_data, unicode): # some module 'data' fields are non strings which blows up the link traversal code
if isinstance(module_data, str) or isinstance(module_data, unicode): # some module 'data' fields are non strings which blows up the link traversal code
lxml_rewrite_links(module_data, lambda link: verify_content_links(module, course_data_path, static_content_store, link, remap_dict))
for key in remap_dict.keys():
@@ -523,6 +523,26 @@ def validate_data_source_paths(data_dir, course_dir):
return err_cnt, warn_cnt
def validate_course_policy(module_store, course_id):
"""
Validate that the course explicitly sets values for any fields whose defaults may have changed between
the export and the import.
Does not add to error count as these are just warnings.
"""
# is there a reliable way to get the module location just given the course_id?
warn_cnt = 0
for module in module_store.modules[course_id].itervalues():
if module.location.category == 'course':
if not 'rerandomize' in module._model_data:
warn_cnt += 1
print 'WARN: course policy does not specify value for "rerandomize" whose default is now "never". The behavior of your course may change.'
if not 'showanswer' in module._model_data:
warn_cnt += 1
print 'WARN: course policy does not specify value for "showanswer" whose default is now "finished". The behavior of your course may change.'
return warn_cnt
def perform_xlint(data_dir, course_dirs,
default_class='xmodule.raw_module.RawDescriptor',
load_error_modules=True):
@@ -568,6 +588,8 @@ def perform_xlint(data_dir, course_dirs,
err_cnt += validate_category_hierarchy(module_store, course_id, "chapter", "sequential")
# constrain that sequentials only have 'verticals'
err_cnt += validate_category_hierarchy(module_store, course_id, "sequential", "vertical")
# validate the course policy overrides any defaults which have changed over time
warn_cnt += validate_course_policy(module_store, course_id)
# don't allow metadata on verticals, since we can't edit them in studio
err_cnt += validate_no_non_editable_metadata(module_store, course_id, "vertical")
# don't allow metadata on chapters, since we can't edit them in studio

5
common/lib/xmodule/xmodule/open_ended_grading_classes/open_ended_module.py
View File

@@ -19,6 +19,7 @@ import openendedchild
from numpy import median
from datetime import datetime
from pytz import UTC
from .combined_open_ended_rubric import CombinedOpenEndedRubric
@@ -170,7 +171,7 @@ class OpenEndedModule(openendedchild.OpenEndedChild):
if xqueue is None:
return {'success': False, 'msg': "Couldn't submit feedback."}
qinterface = xqueue['interface']
qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
anonymous_student_id = system.anonymous_student_id
queuekey = xqueue_interface.make_hashkey(str(system.seed) + qtime +
anonymous_student_id +
@@ -224,7 +225,7 @@ class OpenEndedModule(openendedchild.OpenEndedChild):
if xqueue is None:
return False
qinterface = xqueue['interface']
qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
anonymous_student_id = system.anonymous_student_id

10
common/lib/xmodule/xmodule/open_ended_grading_classes/openendedchild.py
View File

@@ -5,6 +5,7 @@ import re
import open_ended_image_submission
from xmodule.progress import Progress
import capa.xqueue_interface as xqueue_interface
from capa.util import *
from .peer_grading_service import PeerGradingService, MockPeerGradingService
import controller_query_service
@@ -334,12 +335,15 @@ class OpenEndedChild(object):
log.exception("Could not create image and check it.")
if image_ok:
image_key = image_data.name + datetime.now().strftime("%Y%m%d%H%M%S")
image_key = image_data.name + datetime.now(UTC).strftime(
xqueue_interface.dateformat
)
try:
image_data.seek(0)
success, s3_public_url = open_ended_image_submission.upload_to_s3(image_data, image_key,
self.s3_interface)
success, s3_public_url = open_ended_image_submission.upload_to_s3(
image_data, image_key, self.s3_interface
)
except:
log.exception("Could not upload image to S3.")

4
common/lib/xmodule/xmodule/progress.py
View File

@@ -146,7 +146,7 @@ class Progress(object):
sending Progress objects to js to limit dependencies.
'''
if progress is None:
return "NA"
return "0"
return progress.ternary_str()
@staticmethod
@@ -157,5 +157,5 @@ class Progress(object):
passing Progress objects to js to limit dependencies.
'''
if progress is None:
return "NA"
return "0"
return str(progress)

11
common/lib/xmodule/xmodule/schematic_module.py
View File

@@ -1,11 +0,0 @@
from .x_module import XModule, XModuleDescriptor
class ModuleDescriptor(XModuleDescriptor):
pass
class Module(XModule):
def get_html(self):
return '<input type="hidden" class="schematic" name="{item_id}" height="480" width="640">'.format(item_id=self.item_id)

31
common/lib/xmodule/xmodule/tests/test_capa_module.py
View File

@@ -1233,6 +1233,37 @@ class CapaModuleTest(unittest.TestCase):
mock_log.exception.assert_called_once_with('Got bad progress')
mock_log.reset_mock()
@patch('xmodule.capa_module.Progress')
def test_get_progress_calculate_progress_fraction(self, mock_progress):
"""
Check that score and total are calculated correctly for the progress fraction.
"""
module = CapaFactory.create()
module.weight = 1
module.get_progress()
mock_progress.assert_called_with(0, 1)
other_module = CapaFactory.create(correct=True)
other_module.weight = 1
other_module.get_progress()
mock_progress.assert_called_with(1, 1)
def test_get_html(self):
"""
Check that get_html() calls get_progress() with no arguments.
"""
module = CapaFactory.create()
module.get_progress = Mock(wraps=module.get_progress)
module.get_html()
module.get_progress.assert_called_once_with()
def test_get_problem(self):
"""
Check that get_problem() returns the expected dictionary.
"""
module = CapaFactory.create()
self.assertEquals(module.get_problem("data"), {'html': module.get_problem_html(encapsulate=False)})
class ComplexEncoderTest(unittest.TestCase):
def test_default(self):

6
common/lib/xmodule/xmodule/tests/test_combined_open_ended.py
View File

@@ -14,6 +14,7 @@ from xmodule.modulestore import Location
from lxml import etree
import capa.xqueue_interface as xqueue_interface
from datetime import datetime
from pytz import UTC
import logging
log = logging.getLogger(__name__)
@@ -212,7 +213,7 @@ class OpenEndedModuleTest(unittest.TestCase):
'submission_id': '1',
'grader_id': '1',
'score': 3}
qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
student_info = {'anonymous_student_id': self.test_system.anonymous_student_id,
'submission_time': qtime}
contents = {
@@ -233,7 +234,7 @@ class OpenEndedModuleTest(unittest.TestCase):
def test_send_to_grader(self):
submission = "This is a student submission"
qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
student_info = {'anonymous_student_id': self.test_system.anonymous_student_id,
'submission_time': qtime}
contents = self.openendedmodule.payload.copy()
@@ -632,6 +633,7 @@ class OpenEndedModuleXmlTest(unittest.TestCase, DummyModulestore):
module.handle_ajax("reset", {})
self.assertEqual(module.state, "initial")
class OpenEndedModuleXmlAttemptTest(unittest.TestCase, DummyModulestore):
"""
Test if student is able to reset the problem

6
common/lib/xmodule/xmodule/tests/test_progress.py
View File

@@ -90,15 +90,15 @@ class ProgressTest(unittest.TestCase):
self.assertEqual(Progress.to_js_status_str(self.not_started), "none")
self.assertEqual(Progress.to_js_status_str(self.half_done), "in_progress")
self.assertEqual(Progress.to_js_status_str(self.done), "done")
self.assertEqual(Progress.to_js_status_str(None), "NA")
self.assertEqual(Progress.to_js_status_str(None), "0")
def test_to_js_detail_str(self):
'''Test the Progress.to_js_detail_str() method'''
f = Progress.to_js_detail_str
for p in (self.not_started, self.half_done, self.done):
self.assertEqual(f(p), str(p))
# But None should be encoded as NA
self.assertEqual(f(None), "NA")
# But None should be encoded as 0
self.assertEqual(f(None), "0")
def test_add(self):
'''Test the Progress.add_counts() method'''

160
common/lib/xmodule/xmodule/tests/test_xblock_wrappers.py Normal file
View File

@@ -0,0 +1,160 @@
"""
Tests for the wrapping layer that provides the XBlock API using XModule/Descriptor
functionality
"""
from nose.tools import assert_equal
from unittest.case import SkipTest
from mock import Mock
from xmodule.annotatable_module import AnnotatableDescriptor
from xmodule.capa_module import CapaDescriptor
from xmodule.course_module import CourseDescriptor
from xmodule.combined_open_ended_module import CombinedOpenEndedDescriptor
from xmodule.discussion_module import DiscussionDescriptor
from xmodule.gst_module import GraphicalSliderToolDescriptor
from xmodule.html_module import HtmlDescriptor
from xmodule.peer_grading_module import PeerGradingDescriptor
from xmodule.poll_module import PollDescriptor
from xmodule.video_module import VideoDescriptor
from xmodule.word_cloud_module import WordCloudDescriptor
from xmodule.crowdsource_hinter import CrowdsourceHinterDescriptor
from xmodule.videoalpha_module import VideoAlphaDescriptor
from xmodule.seq_module import SequenceDescriptor
from xmodule.conditional_module import ConditionalDescriptor
from xmodule.randomize_module import RandomizeDescriptor
from xmodule.vertical_module import VerticalDescriptor
from xmodule.wrapper_module import WrapperDescriptor
LEAF_XMODULES = (
AnnotatableDescriptor,
CapaDescriptor,
CombinedOpenEndedDescriptor,
DiscussionDescriptor,
GraphicalSliderToolDescriptor,
HtmlDescriptor,
PeerGradingDescriptor,
PollDescriptor,
VideoDescriptor,
# This is being excluded because it has dependencies on django
#VideoAlphaDescriptor,
WordCloudDescriptor,
)
CONTAINER_XMODULES = (
CrowdsourceHinterDescriptor,
CourseDescriptor,
SequenceDescriptor,
ConditionalDescriptor,
RandomizeDescriptor,
VerticalDescriptor,
WrapperDescriptor,
CourseDescriptor,
)
class TestXBlockWrapper(object):
@property
def leaf_module_runtime(self):
runtime = Mock()
runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs))
runtime.anonymous_student_id = 'anonymous_student_id'
runtime.open_ended_grading_interface = {}
runtime.seed = 5
runtime.get = lambda x: getattr(runtime, x)
runtime.position = 2
runtime.ajax_url = 'ajax_url'
runtime.xblock_model_data = lambda d: d._model_data
return runtime
@property
def leaf_descriptor_runtime(self):
runtime = Mock()
runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs))
return runtime
def leaf_descriptor(self, descriptor_cls):
return descriptor_cls(
self.leaf_descriptor_runtime,
{'location': 'i4x://org/course/catagory/name'}
)
def leaf_module(self, descriptor_cls):
return self.leaf_descriptor(descriptor_cls).xmodule(self.leaf_module_runtime)
def container_module_runtime(self, depth):
runtime = self.leaf_module_runtime
if depth == 0:
runtime.get_module.side_effect = lambda x: self.leaf_module(HtmlDescriptor)
else:
runtime.get_module.side_effect = lambda x: self.container_module(VerticalDescriptor, depth-1)
return runtime
@property
def container_descriptor_runtime(self):
runtime = Mock()
runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs))
return runtime
def container_descriptor(self, descriptor_cls):
return descriptor_cls(
self.container_descriptor_runtime,
{
'location': 'i4x://org/course/catagory/name',
'children': range(3)
}
)
def container_module(self, descriptor_cls, depth):
return self.container_descriptor(descriptor_cls).xmodule(self.container_module_runtime(depth))
class TestStudentView(TestXBlockWrapper):
# Test that for all of the leaf XModule Descriptors,
# the student_view wrapper returns the same thing in its content
# as get_html returns
def test_student_view_leaf_node(self):
for descriptor_cls in LEAF_XMODULES:
yield self.check_student_view_leaf_node, descriptor_cls
# Check that when an xmodule is instantiated from descriptor_cls
# it generates the same thing from student_view that it does from get_html
def check_student_view_leaf_node(self, descriptor_cls):
xmodule = self.leaf_module(descriptor_cls)
assert_equal(xmodule.get_html(), xmodule.student_view(None).content)
# Test that for all container XModule Descriptors,
# their corresponding XModule renders the same thing using student_view
# as it does using get_html, under the following conditions:
# a) All of its descendents are xmodules
# b) Some of its descendents are xmodules and some are xblocks
# c) All of its descendents are xblocks
def test_student_view_container_node(self):
for descriptor_cls in CONTAINER_XMODULES:
yield self.check_student_view_container_node_xmodules_only, descriptor_cls
yield self.check_student_view_container_node_mixed, descriptor_cls
yield self.check_student_view_container_node_xblocks_only, descriptor_cls
# Check that when an xmodule is generated from descriptor_cls
# with only xmodule children, it generates the same html from student_view
# as it does using get_html
def check_student_view_container_node_xmodules_only(self, descriptor_cls):
xmodule = self.container_module(descriptor_cls, 2)
assert_equal(xmodule.get_html(), xmodule.student_view(None).content)
# Check that when an xmodule is generated from descriptor_cls
# with mixed xmodule and xblock children, it generates the same html from student_view
# as it does using get_html
def check_student_view_container_node_mixed(self, descriptor_cls):
raise SkipTest("XBlock support in XDescriptor not yet fully implemented")
# Check that when an xmodule is generated from descriptor_cls
# with only xblock children, it generates the same html from student_view
# as it does using get_html
def check_student_view_container_node_xblocks_only(self, descriptor_cls):
raise SkipTest("XBlock support in XModules not yet fully implemented")

14
common/lib/xmodule/xmodule/video_module.py
View File

@@ -27,11 +27,13 @@ class VideoFields(object):
scope=Scope.settings,
# it'd be nice to have a useful default but it screws up other things; so,
# use display_name_with_default for those
default="Video Title"
default="Video"
)
data = String(help="XML data for the problem",
data = String(
help="XML data for the problem",
default='',
scope=Scope.content)
scope=Scope.content
)
position = Integer(help="Current position in the video", scope=Scope.user_state, default=0)
show_captions = Boolean(help="This controls whether or not captions are shown by default.", display_name="Show Captions", scope=Scope.settings, default=True)
youtube_id_1_0 = String(help="This is the Youtube ID reference for the normal speed video.", display_name="Default Speed", scope=Scope.settings, default="OEoXaMPEzfM")
@@ -125,7 +127,7 @@ class VideoDescriptor(VideoFields,
url identifiers
"""
video = super(VideoDescriptor, cls).from_xml(xml_data, system, org, course)
_parse_video_xml(video, xml_data)
_parse_video_xml(video, video.data)
return video
def definition_to_xml(self, resource_fs):
@@ -146,10 +148,6 @@ def _parse_video_xml(video, xml_data):
display_name = xml.get('display_name')
if display_name:
video.display_name = display_name
elif video.url_name is not None:
# copies the logic of display_name_with_default in order that studio created videos will have an
# initial non guid name
video.display_name = video.url_name.replace('_', ' ')
youtube = xml.get('youtube')
if youtube:

116
common/lib/xmodule/xmodule/x_module.py
View File

@@ -8,9 +8,11 @@ from collections import namedtuple
from pkg_resources import resource_listdir, resource_string, resource_isdir
from xmodule.modulestore import inheritance, Location
from xmodule.modulestore.exceptions import ItemNotFoundError, InsufficientSpecificationError
from xmodule.modulestore.exceptions import ItemNotFoundError, InsufficientSpecificationError, InvalidLocationError
from xblock.core import XBlock, Scope, String, Integer, Float, ModelType
from xblock.fragment import Fragment
from xmodule.modulestore.locator import BlockUsageLocator
log = logging.getLogger(__name__)
@@ -27,7 +29,13 @@ class LocationField(ModelType):
"""
Parse the json value as a Location
"""
return Location(value)
try:
return Location(value)
except InvalidLocationError:
if isinstance(value, BlockUsageLocator):
return value
else:
return BlockUsageLocator(value)
def to_json(self, value):
"""
@@ -166,6 +174,10 @@ class XModule(XModuleFields, HTMLSnippet, XBlock):
self.url_name = self.location.name
if not hasattr(self, 'category'):
self.category = self.location.category
elif isinstance(self.location, BlockUsageLocator):
self.url_name = self.location.usage_id
if not hasattr(self, 'category'):
raise InsufficientSpecificationError()
else:
raise InsufficientSpecificationError()
self._loaded_children = None
@@ -191,6 +203,13 @@ class XModule(XModuleFields, HTMLSnippet, XBlock):
'''
if self._loaded_children is None:
child_descriptors = self.get_child_descriptors()
# This deliberately uses system.get_module, rather than runtime.get_block,
# because we're looking at XModule children, rather than XModuleDescriptor children.
# That means it can use the deprecated XModule apis, rather than future XBlock apis
# TODO: Once we're in a system where this returns a mix of XModuleDescriptors
# and XBlocks, we're likely to have to change this more
children = [self.system.get_module(descriptor) for descriptor in child_descriptors]
# get_module returns None if the current user doesn't have access
# to the location.
@@ -296,6 +315,19 @@ class XModule(XModuleFields, HTMLSnippet, XBlock):
return ""
# ~~~~~~~~~~~~~~~ XBlock API Wrappers ~~~~~~~~~~~~~~~~
def student_view(self, context):
"""
Return a fragment with the html from this XModule
Doesn't yet add any of the javascript to the fragment, nor the css.
Also doesn't expect any javascript binding, yet.
Makes no use of the context parameter
"""
return Fragment(self.get_html())
def policy_key(location):
"""
Get the key for a location in a policy file. (Since the policy file is
@@ -436,8 +468,17 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
self.url_name = self.location.name
if not hasattr(self, 'category'):
self.category = self.location.category
elif isinstance(self.location, BlockUsageLocator):
self.url_name = self.location.usage_id
if not hasattr(self, 'category'):
raise InsufficientSpecificationError()
else:
raise InsufficientSpecificationError()
# update_version is the version which last updated this xblock v prev being the penultimate updater
# leaving off original_version since it complicates creation w/o any obv value yet and is computable
# by following previous until None
# definition_locator is only used by mongostores which separate definitions from blocks
self.edited_by = self.edited_on = self.previous_version = self.update_version = self.definition_locator = None
self._child_instances = None
@property
@@ -473,7 +514,7 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
child = child_loc
else:
try:
child = self.system.load_item(child_loc)
child = self.runtime.get_block(child_loc)
except ItemNotFoundError:
log.exception('Unable to load item {loc}, skipping'.format(loc=child_loc))
continue
@@ -514,22 +555,30 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
# ================================= JSON PARSING ===========================
@staticmethod
def load_from_json(json_data, system, default_class=None):
def load_from_json(json_data, system, default_class=None, parent_xblock=None):
"""
This method instantiates the correct subclass of XModuleDescriptor based
on the contents of json_data.
on the contents of json_data. It does not persist it and can create one which
has no usage id.
json_data must contain a 'location' element, and must be suitable to be
passed into the subclasses `from_json` method as model_data
parent_xblock is used to compute inherited metadata as well as to append the new xblock.
json_data:
- 'location' : must have this field
- 'category': the xmodule category (required or location must be a Location)
- 'metadata': a dict of locally set metadata (not inherited)
- 'children': a list of children's usage_ids w/in this course
- 'definition':
- '_id' (optional): the usage_id of this. Will generate one if not given one.
"""
class_ = XModuleDescriptor.load_class(
json_data['location']['category'],
json_data.get('category', json_data.get('location', {}).get('category')),
default_class
)
return class_.from_json(json_data, system)
return class_.from_json(json_data, system, parent_xblock)
@classmethod
def from_json(cls, json_data, system):
def from_json(cls, json_data, system, parent_xblock=None):
"""
Creates an instance of this descriptor from the supplied json_data.
This may be overridden by subclasses
@@ -547,28 +596,25 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
Otherwise, it contains the single field 'data'
4) Any value later in this list overrides a value earlier in this list
system: A DescriptorSystem for interacting with external resources
json_data:
- 'category': the xmodule category (required)
- 'metadata': a dict of locally set metadata (not inherited)
- 'children': a list of children's usage_ids w/in this course
- 'definition':
- '_id' (optional): the usage_id of this. Will generate one if not given one.
"""
model_data = {}
usage_id = json_data.get('_id', None)
if not '_inherited_metadata' in json_data and parent_xblock is not None:
json_data['_inherited_metadata'] = parent_xblock.xblock_kvs.get_inherited_metadata().copy()
json_metadata = json_data.get('metadata', {})
for field in inheritance.INHERITABLE_METADATA:
if field in json_metadata:
json_data['_inherited_metadata'][field] = json_metadata[field]
for key, value in json_data.get('metadata', {}).items():
model_data[cls._translate(key)] = value
model_data.update(json_data.get('metadata', {}))
definition = json_data.get('definition', {})
if 'children' in definition:
model_data['children'] = definition['children']
if 'data' in definition:
if isinstance(definition['data'], dict):
model_data.update(definition['data'])
else:
model_data['data'] = definition['data']
model_data['location'] = json_data['location']
return cls(system, model_data)
new_block = system.xblock_from_json(cls, usage_id, json_data)
if parent_xblock is not None:
parent_xblock.children.append(new_block)
return new_block
@classmethod
def _translate(cls, key):
@@ -649,6 +695,8 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
"""
Use w/ caution. Really intended for use by the persistence layer.
"""
# if caller wants kvs, caller's assuming it's up to date; so, decache it
self.save()
return self._model_data._kvs
# =============================== BUILTIN METHODS ==========================
@@ -780,6 +828,10 @@ class DescriptorSystem(object):
self.resources_fs = resources_fs
self.error_tracker = error_tracker
def get_block(self, block_id):
"""See documentation for `xblock.runtime:Runtime.get_block`"""
return self.load_item(block_id)
class XMLParsingSystem(DescriptorSystem):
def __init__(self, load_item, resources_fs, error_tracker, process_xml, policy, **kwargs):
@@ -868,8 +920,8 @@ class ModuleSystem(object):
publish(event) - A function that allows XModules to publish events (such as grade changes)
xblock_model_data - A dict-like object containing the all data available to this
xblock
xblock_model_data - A function that constructs a model_data for an xblock from its
corresponding descriptor
cache - A cache object with two methods:
.get(key) returns an object from the cache or None.

4
common/lib/xmodule/xmodule/xml_module.py
View File

@@ -306,6 +306,7 @@ class XmlDescriptor(XModuleDescriptor):
org and course are optional strings that will be used in the generated modules
url identifiers
"""
xml_object = etree.fromstring(xml_data)
# VS[compat] -- just have the url_name lookup, once translation is done
url_name = xml_object.get('url_name', xml_object.get('slug'))
@@ -318,7 +319,8 @@ class XmlDescriptor(XModuleDescriptor):
filepath = cls._format_filepath(xml_object.tag, name_to_pathname(url_name))
definition_xml = cls.load_file(filepath, system.resources_fs, location)
else:
definition_xml = xml_object # this is just a pointer, not the real definition content
definition_xml = xml_object
filepath = None
definition, children = cls.load_definition(definition_xml, system, location) # note this removes metadata

27
common/test/data/splitmongo_json/active_versions.json Normal file
View File

@@ -0,0 +1,27 @@
[{"_id" : "GreekHero",
"org" : "testx",
"prettyid" : "test_course",
"versions" : {
"draft" : { "$oid" : "1d00000000000000dddd0000" }
},
"edited_on" : {"$date" : 1364481713238},
"edited_by" : "test@edx.org"},
{"_id" : "wonderful",
"org" : "testx",
"prettyid" : "another_course",
"versions" : {
"draft" : { "$oid" : "1d00000000000000dddd2222" },
"published" : { "$oid" : "1d00000000000000eeee0000" }
},
"edited_on" : {"$date" : 1364481313238},
"edited_by" : "test@edx.org"},
{"_id" : "contender",
"org" : "guestx",
"prettyid" : "test_course",
"versions" : {
"draft" : { "$oid" : "1d00000000000000dddd5555" }},
"edited_on" : {"$date" : 1364491313238},
"edited_by" : "test@guestx.edu"}
]

Some files were not shown because too many files have changed in this diff Show More