diff --git a/AUTHORS b/AUTHORS index bf4bf4ff83..1af7349491 100644 --- a/AUTHORS +++ b/AUTHORS @@ -76,3 +76,4 @@ Jonah Stanley Slater Victoroff Peter Fogg Bethany LaPenta +Renzo Lucioni diff --git a/CHANGELOG.rst b/CHANGELOG.rst new file mode 100644 index 0000000000..cac4757218 --- /dev/null +++ b/CHANGELOG.rst @@ -0,0 +1,110 @@ +Change Log +---------- + +These are notable changes in edx-platform. This is a rolling list of changes, +in roughly chronological order, most recent first. Add your entries at or near +the top. Include a label indicating the component affected. + +LMS: Forums. Added handling for case where discussion module can get `None` as +value of lms.start in `lms/djangoapps/django_comment_client/utils.py` + +Studio, LMS: Make ModelTypes more strict about their expected content (for +instance, Boolean, Integer, String), but also allow them to hold either the +typed value, or a String that can be converted to their typed value. For example, +an Integer can contain 3 or '3'. This changed an update to the xblock library. + +LMS: Courses whose id matches a regex in the COURSES_WITH_UNSAFE_CODE Django +setting now run entirely outside the Python sandbox. + +Blades: Added tests for Video Alpha player. + +Blades: Video Alpha bug fix for speed changing to 1.0 in Firefox. + +Blades: Additional event tracking added to Video Alpha: fullscreen switch, show/hide +captions. + +CMS: Allow editors to delete uploaded files/assets + +XModules: `XModuleDescriptor.__init__` and `XModule.__init__` dropped the +`location` parameter (and added it as a field), and renamed `system` to `runtime`, +to accord more closely to `XBlock.__init__` + +LMS: Some errors handling Non-ASCII data in XML courses have been fixed. + +LMS: Add page-load tracking using segment-io (if SEGMENT_IO_LMS_KEY and +SEGMENT_IO_LMS feature flag is on) + +Blades: Simplify calc.py (which is used for the Numerical/Formula responses); add trig/other functions. + +LMS: Background colors on login, register, and courseware have been corrected +back to white. + +LMS: Accessibility improvements have been made to several courseware and +navigation elements. + +LMS: Small design/presentation changes to login and register views. + +LMS: Functionality added to instructor enrollment tab in LMS such that invited +student can be auto-enrolled in course or when activating if not current +student. + +Blades: Staff debug info is now accessible for Graphical Slider Tool problems. + +Blades: For Video Alpha the events ready, play, pause, seek, and speed change +are logged on the server (in the logs). + +Common: all dates and times are not time zone aware datetimes. No code should create or use struct_times nor naive +datetimes. + +Common: Developers can now have private Django settings files. + +Common: Safety code added to prevent anything above the vertical level in the +course tree from being marked as version='draft'. It will raise an exception if +the code tries to so mark a node. We need the backtraces to figure out where +this very infrequent intermittent marking was occurring. It was making courses +look different in Studio than in LMS. + +Deploy: MKTG_URLS is now read from env.json. + +Common: Theming makes it possible to change the look of the site, from +Stanford. + +Common: Accessibility UI fixes. + +Common: The "duplicate email" error message is more informative. + +Studio: Component metadata settings editor. + +Studio: Autoplay for Video Alpha is disabled (only in Studio). + +Studio: Single-click creation for video and discussion components. + +Studio: fixed a bad link in the activation page. + +LMS: Changed the help button text. + +LMS: Fixed failing numeric response (decimal but no trailing digits). + +LMS: XML Error module no longer shows students a stack trace. + +Blades: Videoalpha. + +XModules: Added partial credit for foldit module. + +XModules: Added "randomize" XModule to list of XModule types. + +XModules: Show errors with full descriptors. + +XQueue: Fixed (hopefully) worker crash when the connection to RabbitMQ is +dropped suddenly. + +XQueue: Upload file submissions to a specially named bucket in S3. + +Common: Removed request debugger. + +Common: Updated Django to version 1.4.5. + +Common: Updated CodeJail. + +Common: Allow setting of authentication session cookie name. + diff --git a/README.md b/README.md index 3a6236ea70..92a4116354 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ CMS templates. Fortunately, `rake` will do all of this for you! Just run: If you are running these commands using the [`zsh`](http://www.zsh.org/) shell, zsh will assume that you are doing -[shell globbing](https://en.wikipedia.org/wiki/Glob_(programming)), search for +[shell globbing](https://en.wikipedia.org/wiki/Glob_%28programming%29), search for a file in your directory named `django-adminsyncdb` or `django-adminmigrate`, and fail. To fix this, just surround the argument with quotation marks, so that you're running `rake "django-admin[syncdb]"`. diff --git a/cms/CHANGELOG.md b/cms/CHANGELOG.md deleted file mode 100644 index d21d08d23c..0000000000 --- a/cms/CHANGELOG.md +++ /dev/null @@ -1,21 +0,0 @@ -Instructions -============ -For each pull request, add one or more lines to the bottom of the change list. When -code is released to production, change the `Upcoming` entry to todays date, and add -a new block at the bottom of the file. - - Upcoming - -------- - -Change log entries should be targeted at end users. A good place to start is the -user story that instigated the pull request. - - -Changes -======= - -Upcoming --------- -* Fix: Deleting last component in a unit does not work -* Fix: Unit name is editable when a unit is public -* Fix: Visual feedback inconsistent when saving a unit name change diff --git a/cms/djangoapps/contentstore/features/advanced-settings.feature b/cms/djangoapps/contentstore/features/advanced-settings.feature index 558294e890..13600f2086 100644 --- a/cms/djangoapps/contentstore/features/advanced-settings.feature +++ b/cms/djangoapps/contentstore/features/advanced-settings.feature @@ -28,11 +28,18 @@ Feature: Advanced (manual) course policy Scenario: Test how multi-line input appears Given I am on the Advanced Course Settings page in Studio - When I create a JSON object as a value + When I create a JSON object as a value for "discussion_topics" Then it is displayed as formatted And I reload the page Then it is displayed as formatted + Scenario: Test error if value supplied is of the wrong type + Given I am on the Advanced Course Settings page in Studio + When I create a JSON object as a value for "display_name" + Then I get an error on save + And I reload the page + Then the policy key value is unchanged + Scenario: Test automatic quoting of non-JSON values Given I am on the Advanced Course Settings page in Studio When I create a non-JSON value not in quotes diff --git a/cms/djangoapps/contentstore/features/advanced-settings.py b/cms/djangoapps/contentstore/features/advanced-settings.py index eb00c06ba9..4995f3505d 100644 --- a/cms/djangoapps/contentstore/features/advanced-settings.py +++ b/cms/djangoapps/contentstore/features/advanced-settings.py @@ -2,13 +2,8 @@ #pylint: disable=W0621 from lettuce import world, step -from common import * -from nose.tools import assert_false, assert_equal - -""" -http://selenium.googlecode.com/svn/trunk/docs/api/py/webdriver/selenium.webdriver.common.keys.html -""" -from selenium.webdriver.common.keys import Keys +from nose.tools import assert_false, assert_equal, assert_regexp_matches +from common import type_in_codemirror KEY_CSS = '.key input.policy-key' VALUE_CSS = 'textarea.json' @@ -38,13 +33,7 @@ def press_the_notification_button(step, name): @step(u'I edit the value of a policy key$') def edit_the_value_of_a_policy_key(step): - """ - It is hard to figure out how to get into the CodeMirror - area, so cheat and do it from the policy key field :) - """ - world.css_find(".CodeMirror")[get_index_of(DISPLAY_NAME_KEY)].click() - g = world.css_find("div.CodeMirror.CodeMirror-focused > div > textarea") - g._element.send_keys(Keys.ARROW_LEFT, ' ', 'X') + type_in_codemirror(get_index_of(DISPLAY_NAME_KEY), 'X') @step(u'I edit the value of a policy key and save$') @@ -52,9 +41,9 @@ def edit_the_value_of_a_policy_key_and_save(step): change_display_name_value(step, '"foo"') -@step('I create a JSON object as a value$') -def create_JSON_object(step): - change_display_name_value(step, '{"key": "value", "key_2": "value_2"}') +@step('I create a JSON object as a value for "(.*)"$') +def create_JSON_object(step, key): + change_value(step, key, '{"key": "value", "key_2": "value_2"}') @step('I create a non-JSON value not in quotes$') @@ -82,7 +71,12 @@ def they_are_alphabetized(step): @step('it is displayed as formatted$') def it_is_formatted(step): - assert_policy_entries([DISPLAY_NAME_KEY], ['{\n "key": "value",\n "key_2": "value_2"\n}']) + assert_policy_entries(['discussion_topics'], ['{\n "key": "value",\n "key_2": "value_2"\n}']) + + +@step('I get an error on save$') +def error_on_save(step): + assert_regexp_matches(world.css_text('#notification-error-description'), 'Incorrect setting format') @step('it is displayed as a string') @@ -124,12 +118,9 @@ def get_display_name_value(): def change_display_name_value(step, new_value): + change_value(step, DISPLAY_NAME_KEY, new_value) - world.css_find(".CodeMirror")[get_index_of(DISPLAY_NAME_KEY)].click() - g = world.css_find("div.CodeMirror.CodeMirror-focused > div > textarea") - display_name = get_display_name_value() - for count in range(len(display_name)): - g._element.send_keys(Keys.END, Keys.BACK_SPACE) - # Must delete "" before typing the JSON value - g._element.send_keys(Keys.END, Keys.BACK_SPACE, Keys.BACK_SPACE, new_value) + +def change_value(step, key, new_value): + type_in_codemirror(get_index_of(key), new_value) press_the_notification_button(step, "Save") diff --git a/cms/djangoapps/contentstore/features/common.py b/cms/djangoapps/contentstore/features/common.py index 494192ad06..c28b35b1c2 100644 --- a/cms/djangoapps/contentstore/features/common.py +++ b/cms/djangoapps/contentstore/features/common.py @@ -169,3 +169,14 @@ def open_new_unit(step): step.given('I have added a new subsection') step.given('I expand the first section') world.css_click('a.new-unit-item') + + +def type_in_codemirror(index, text): + world.css_click(".CodeMirror", index=index) + g = world.css_find("div.CodeMirror.CodeMirror-focused > div > textarea") + if world.is_mac(): + g._element.send_keys(Keys.COMMAND + 'a') + else: + g._element.send_keys(Keys.CONTROL + 'a') + g._element.send_keys(Keys.DELETE) + g._element.send_keys(text) diff --git a/cms/djangoapps/contentstore/features/problem-editor.feature b/cms/djangoapps/contentstore/features/problem-editor.feature index bde350d8a3..cc1d766d2e 100644 --- a/cms/djangoapps/contentstore/features/problem-editor.feature +++ b/cms/djangoapps/contentstore/features/problem-editor.feature @@ -3,65 +3,71 @@ Feature: Problem Editor Scenario: User can view metadata Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then I see five alphabetized settings and their expected values And Edit High Level Source is not visible Scenario: User can modify String values Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then I can modify the display name And my display name change is persisted on save Scenario: User can specify special characters in String values Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then I can specify special characters in the display name And my special characters and persisted on save Scenario: User can revert display name to unset Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then I can revert the display name to unset And my display name is unset on save Scenario: User can select values in a Select Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then I can select Per Student for Randomization And my change to randomization is persisted And I can revert to the default value for randomization Scenario: User can modify float input values Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then I can set the weight to "3.5" And my change to weight is persisted And I can revert to the default value of unset for weight Scenario: User cannot type letters in float number field Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then if I set the weight to "abc", it remains unset Scenario: User cannot type decimal values integer number field Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then if I set the max attempts to "2.34", it displays initially as "234", and is persisted as "234" Scenario: User cannot type out of range values in an integer number field Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then if I set the max attempts to "-3", it displays initially as "-3", and is persisted as "0" Scenario: Settings changes are not saved on Cancel Given I have created a Blank Common Problem - And I edit and select Settings + When I edit and select Settings Then I can set the weight to "3.5" And I can modify the display name Then If I press Cancel my changes are not persisted Scenario: Edit High Level source is available for LaTeX problem Given I have created a LaTeX Problem - And I edit and select Settings + When I edit and select Settings Then Edit High Level Source is visible + + Scenario: High Level source is persisted for LaTeX problem (bug STUD-280) + Given I have created a LaTeX Problem + When I edit and compile the High Level Source + Then my change to the High Level Source is persisted + And when I view the High Level Source I see my changes diff --git a/cms/djangoapps/contentstore/features/problem-editor.py b/cms/djangoapps/contentstore/features/problem-editor.py index 5dfcf55046..8691a6772e 100644 --- a/cms/djangoapps/contentstore/features/problem-editor.py +++ b/cms/djangoapps/contentstore/features/problem-editor.py @@ -3,6 +3,7 @@ from lettuce import world, step from nose.tools import assert_equal +from common import type_in_codemirror DISPLAY_NAME = "Display Name" MAXIMUM_ATTEMPTS = "Maximum Attempts" @@ -41,7 +42,9 @@ def i_see_five_settings_with_values(step): @step('I can modify the display name') def i_can_modify_the_display_name(step): - world.get_setting_entry(DISPLAY_NAME).find_by_css('.setting-input')[0].fill('modified') + # Verifying that the display name can be a string containing a floating point value + # (to confirm that we don't throw an error because it is of the wrong type). + world.get_setting_entry(DISPLAY_NAME).find_by_css('.setting-input')[0].fill('3.4') verify_modified_display_name() @@ -133,12 +136,12 @@ def set_the_max_attempts(step, max_attempts_set, max_attempts_displayed, max_att @step('Edit High Level Source is not visible') def edit_high_level_source_not_visible(step): - verify_high_level_source(step, False) + verify_high_level_source_links(step, False) @step('Edit High Level Source is visible') -def edit_high_level_source_visible(step): - verify_high_level_source(step, True) +def edit_high_level_source_links_visible(step): + verify_high_level_source_links(step, True) @step('If I press Cancel my changes are not persisted') @@ -151,13 +154,33 @@ def cancel_does_not_save_changes(step): @step('I have created a LaTeX Problem') def create_latex_problem(step): world.click_new_component_button(step, '.large-problem-icon') - # Go to advanced tab (waiting for the tab to be visible) - world.css_find('#ui-id-2') + # Go to advanced tab. world.css_click('#ui-id-2') world.click_component_from_menu("i4x://edx/templates/problem/Problem_Written_in_LaTeX", '.xmodule_CapaModule') -def verify_high_level_source(step, visible): +@step('I edit and compile the High Level Source') +def edit_latex_source(step): + open_high_level_source() + type_in_codemirror(1, "hi") + world.css_click('.hls-compile') + + +@step('my change to the High Level Source is persisted') +def high_level_source_persisted(step): + def verify_text(driver): + return world.css_find('.problem').text == 'hi' + + world.wait_for(verify_text) + + +@step('I view the High Level Source I see my changes') +def high_level_source_in_editor(step): + open_high_level_source() + assert_equal('hi', world.css_find('.source-edit-box').value) + + +def verify_high_level_source_links(step, visible): assert_equal(visible, world.is_css_present('.launch-latex-compiler')) world.cancel_component(step) assert_equal(visible, world.is_css_present('.upload-button')) @@ -172,7 +195,7 @@ def verify_modified_randomization(): def verify_modified_display_name(): - world.verify_setting_entry(world.get_setting_entry(DISPLAY_NAME), DISPLAY_NAME, 'modified', True) + world.verify_setting_entry(world.get_setting_entry(DISPLAY_NAME), DISPLAY_NAME, '3.4', True) def verify_modified_display_name_with_special_chars(): @@ -185,3 +208,8 @@ def verify_unset_display_name(): def set_weight(weight): world.get_setting_entry(PROBLEM_WEIGHT).find_by_css('.setting-input')[0].fill(weight) + + +def open_high_level_source(): + world.css_click('a.edit-button') + world.css_click('.launch-latex-compiler > a') diff --git a/cms/djangoapps/contentstore/features/section.py b/cms/djangoapps/contentstore/features/section.py index 4a628ff72b..9d63fa73c8 100644 --- a/cms/djangoapps/contentstore/features/section.py +++ b/cms/djangoapps/contentstore/features/section.py @@ -9,34 +9,34 @@ from nose.tools import assert_equal @step('I click the new section link$') -def i_click_new_section_link(step): +def i_click_new_section_link(_step): link_css = 'a.new-courseware-section-button' world.css_click(link_css) @step('I enter the section name and click save$') -def i_save_section_name(step): +def i_save_section_name(_step): save_section_name('My Section') @step('I enter a section name with a quote and click save$') -def i_save_section_name_with_quote(step): +def i_save_section_name_with_quote(_step): save_section_name('Section with "Quote"') @step('I have added a new section$') -def i_have_added_new_section(step): +def i_have_added_new_section(_step): add_section() @step('I click the Edit link for the release date$') -def i_click_the_edit_link_for_the_release_date(step): +def i_click_the_edit_link_for_the_release_date(_step): button_css = 'div.section-published-date a.edit-button' world.css_click(button_css) @step('I save a new section release date$') -def i_save_a_new_section_release_date(step): +def i_save_a_new_section_release_date(_step): set_date_and_time('input.start-date.date.hasDatepicker', '12/25/2013', 'input.start-time.time.ui-timepicker-input', '00:00') world.browser.click_link_by_text('Save') @@ -46,35 +46,35 @@ def i_save_a_new_section_release_date(step): @step('I see my section on the Courseware page$') -def i_see_my_section_on_the_courseware_page(step): +def i_see_my_section_on_the_courseware_page(_step): see_my_section_on_the_courseware_page('My Section') @step('I see my section name with a quote on the Courseware page$') -def i_see_my_section_name_with_quote_on_the_courseware_page(step): +def i_see_my_section_name_with_quote_on_the_courseware_page(_step): see_my_section_on_the_courseware_page('Section with "Quote"') @step('I click to edit the section name$') -def i_click_to_edit_section_name(step): +def i_click_to_edit_section_name(_step): world.css_click('span.section-name-span') @step('I see the complete section name with a quote in the editor$') -def i_see_complete_section_name_with_quote_in_editor(step): +def i_see_complete_section_name_with_quote_in_editor(_step): css = '.section-name-edit input[type=text]' assert world.is_css_present(css) assert_equal(world.browser.find_by_css(css).value, 'Section with "Quote"') @step('the section does not exist$') -def section_does_not_exist(step): +def section_does_not_exist(_step): css = 'h3[data-name="My Section"]' assert world.is_css_not_present(css) @step('I see a release date for my section$') -def i_see_a_release_date_for_my_section(step): +def i_see_a_release_date_for_my_section(_step): import re css = 'span.published-status' @@ -83,26 +83,32 @@ def i_see_a_release_date_for_my_section(step): # e.g. 11/06/2012 at 16:25 msg = 'Will Release:' - date_regex = '[01][0-9]\/[0-3][0-9]\/[12][0-9][0-9][0-9]' - time_regex = '[0-2][0-9]:[0-5][0-9]' - match_string = '%s %s at %s' % (msg, date_regex, time_regex) + date_regex = r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \d\d?, \d{4}' + if not re.search(date_regex, status_text): + print status_text, date_regex + time_regex = r'[0-2]\d:[0-5]\d( \w{3})?' + if not re.search(time_regex, status_text): + print status_text, time_regex + match_string = r'%s\s+%s at %s' % (msg, date_regex, time_regex) + if not re.match(match_string, status_text): + print status_text, match_string assert re.match(match_string, status_text) @step('I see a link to create a new subsection$') -def i_see_a_link_to_create_a_new_subsection(step): +def i_see_a_link_to_create_a_new_subsection(_step): css = 'a.new-subsection-item' assert world.is_css_present(css) @step('the section release date picker is not visible$') -def the_section_release_date_picker_not_visible(step): +def the_section_release_date_picker_not_visible(_step): css = 'div.edit-subsection-publish-settings' assert not world.css_visible(css) @step('the section release date is updated$') -def the_section_release_date_is_updated(step): +def the_section_release_date_is_updated(_step): css = 'span.published-status' status_text = world.css_text(css) assert_equal(status_text, 'Will Release: 12/25/2013 at 00:00 UTC') diff --git a/cms/djangoapps/contentstore/features/video.feature b/cms/djangoapps/contentstore/features/video.feature index 07771c9d61..0129732d30 100644 --- a/cms/djangoapps/contentstore/features/video.feature +++ b/cms/djangoapps/contentstore/features/video.feature @@ -8,3 +8,8 @@ Feature: Video Component Scenario: Creating a video takes a single click Given I have clicked the new unit button Then creating a video takes a single click + + Scenario: Captions are shown correctly + Given I have created a Video component + And I have hidden captions + Then when I view the video it does not show the captions diff --git a/cms/djangoapps/contentstore/features/video.py b/cms/djangoapps/contentstore/features/video.py index 7cbe8a2258..fd8624999e 100644 --- a/cms/djangoapps/contentstore/features/video.py +++ b/cms/djangoapps/contentstore/features/video.py @@ -16,3 +16,13 @@ def video_takes_a_single_click(step): assert(not world.is_css_present('.xmodule_VideoModule')) world.css_click("a[data-location='i4x://edx/templates/video/default']") assert(world.is_css_present('.xmodule_VideoModule')) + + +@step('I have hidden captions') +def set_show_captions_false(step): + world.css_click('a.hide-subtitles') + + +@step('when I view the video it does not show the captions') +def does_not_show_captions(step): + assert world.css_find('.video')[0].has_class('closed') diff --git a/cms/djangoapps/contentstore/management/commands/empty_asset_trashcan.py b/cms/djangoapps/contentstore/management/commands/empty_asset_trashcan.py new file mode 100644 index 0000000000..9af3277a2b --- /dev/null +++ b/cms/djangoapps/contentstore/management/commands/empty_asset_trashcan.py @@ -0,0 +1,25 @@ +from django.core.management.base import BaseCommand, CommandError +from xmodule.course_module import CourseDescriptor +from xmodule.contentstore.utils import empty_asset_trashcan +from xmodule.modulestore.django import modulestore +from .prompt import query_yes_no + + +class Command(BaseCommand): + help = '''Empty the trashcan. Can pass an optional course_id to limit the damage.''' + + def handle(self, *args, **options): + if len(args) != 1 and len(args) != 0: + raise CommandError("empty_asset_trashcan requires one or no arguments: ||") + + locs = [] + + if len(args) == 1: + locs.append(CourseDescriptor.id_to_location(args[0])) + else: + courses = modulestore('direct').get_courses() + for course in courses: + locs.append(course.location) + + if query_yes_no("Emptying trashcan. Confirm?", default="no"): + empty_asset_trashcan(locs) diff --git a/cms/djangoapps/contentstore/management/commands/restore_asset_from_trashcan.py b/cms/djangoapps/contentstore/management/commands/restore_asset_from_trashcan.py new file mode 100644 index 0000000000..6770bfaf44 --- /dev/null +++ b/cms/djangoapps/contentstore/management/commands/restore_asset_from_trashcan.py @@ -0,0 +1,13 @@ +from django.core.management.base import BaseCommand, CommandError +from xmodule.contentstore.utils import restore_asset_from_trashcan + + +class Command(BaseCommand): + help = '''Restore a deleted asset from the trashcan back to it's original course''' + + def handle(self, *args, **options): + if len(args) != 1 and len(args) != 0: + raise CommandError("restore_asset_from_trashcan requires one argument: ") + + restore_asset_from_trashcan(args[0]) + diff --git a/cms/djangoapps/contentstore/tests/test_checklists.py b/cms/djangoapps/contentstore/tests/test_checklists.py index f0889b0861..54bc726092 100644 --- a/cms/djangoapps/contentstore/tests/test_checklists.py +++ b/cms/djangoapps/contentstore/tests/test_checklists.py @@ -19,6 +19,24 @@ class ChecklistTestCase(CourseTestCase): modulestore = get_modulestore(self.course.location) return modulestore.get_item(self.course.location).checklists + + def compare_checklists(self, persisted, request): + """ + Handles url expansion as possible difference and descends into guts + :param persisted: + :param request: + """ + self.assertEqual(persisted['short_description'], request['short_description']) + compare_urls = (persisted.get('action_urls_expanded') == request.get('action_urls_expanded')) + for pers, req in zip(persisted['items'], request['items']): + self.assertEqual(pers['short_description'], req['short_description']) + self.assertEqual(pers['long_description'], req['long_description']) + self.assertEqual(pers['is_checked'], req['is_checked']) + if compare_urls: + self.assertEqual(pers['action_url'], req['action_url']) + self.assertEqual(pers['action_text'], req['action_text']) + self.assertEqual(pers['action_external'], req['action_external']) + def test_get_checklists(self): """ Tests the get checklists method. """ checklists_url = get_url_reverse('Checklists', self.course) @@ -31,9 +49,9 @@ class ChecklistTestCase(CourseTestCase): self.course.checklists = None modulestore = get_modulestore(self.course.location) modulestore.update_metadata(self.course.location, own_metadata(self.course)) - self.assertEquals(self.get_persisted_checklists(), None) + self.assertEqual(self.get_persisted_checklists(), None) response = self.client.get(checklists_url) - self.assertEquals(payload, response.content) + self.assertEqual(payload, response.content) def test_update_checklists_no_index(self): """ No checklist index, should return all of them. """ @@ -43,7 +61,8 @@ class ChecklistTestCase(CourseTestCase): 'name': self.course.location.name}) returned_checklists = json.loads(self.client.get(update_url).content) - self.assertListEqual(self.get_persisted_checklists(), returned_checklists) + for pay, resp in zip(self.get_persisted_checklists(), returned_checklists): + self.compare_checklists(pay, resp) def test_update_checklists_index_ignored_on_get(self): """ Checklist index ignored on get. """ @@ -53,7 +72,8 @@ class ChecklistTestCase(CourseTestCase): 'checklist_index': 1}) returned_checklists = json.loads(self.client.get(update_url).content) - self.assertListEqual(self.get_persisted_checklists(), returned_checklists) + for pay, resp in zip(self.get_persisted_checklists(), returned_checklists): + self.compare_checklists(pay, resp) def test_update_checklists_post_no_index(self): """ No checklist index, will error on post. """ @@ -78,13 +98,18 @@ class ChecklistTestCase(CourseTestCase): 'course': self.course.location.course, 'name': self.course.location.name, 'checklist_index': 2}) + + def get_first_item(checklist): + return checklist['items'][0] + payload = self.course.checklists[2] - self.assertFalse(payload.get('is_checked')) - payload['is_checked'] = True + self.assertFalse(get_first_item(payload).get('is_checked')) + get_first_item(payload)['is_checked'] = True returned_checklist = json.loads(self.client.post(update_url, json.dumps(payload), "application/json").content) - self.assertTrue(returned_checklist.get('is_checked')) - self.assertEqual(self.get_persisted_checklists()[2], returned_checklist) + self.assertTrue(get_first_item(returned_checklist).get('is_checked')) + pers = self.get_persisted_checklists() + self.compare_checklists(pers[2], returned_checklist) def test_update_checklists_delete_unsupported(self): """ Delete operation is not supported. """ @@ -93,4 +118,4 @@ class ChecklistTestCase(CourseTestCase): 'name': self.course.location.name, 'checklist_index': 100}) response = self.client.delete(update_url) - self.assertContains(response, 'Unsupported request', status_code=400) \ No newline at end of file + self.assertContains(response, 'Unsupported request', status_code=400) diff --git a/cms/djangoapps/contentstore/tests/test_contentstore.py b/cms/djangoapps/contentstore/tests/test_contentstore.py index 232b68ecc8..9346d2189d 100644 --- a/cms/djangoapps/contentstore/tests/test_contentstore.py +++ b/cms/djangoapps/contentstore/tests/test_contentstore.py @@ -28,6 +28,8 @@ from xmodule.templates import update_templates from xmodule.modulestore.xml_exporter import export_to_xml from xmodule.modulestore.xml_importer import import_from_xml, perform_xlint from xmodule.modulestore.inheritance import own_metadata +from xmodule.contentstore.content import StaticContent +from xmodule.contentstore.utils import restore_asset_from_trashcan, empty_asset_trashcan from xmodule.capa_module import CapaDescriptor from xmodule.course_module import CourseDescriptor @@ -35,6 +37,7 @@ from xmodule.seq_module import SequenceDescriptor from xmodule.modulestore.exceptions import ItemNotFoundError from contentstore.views.component import ADVANCED_COMPONENT_TYPES +from xmodule.exceptions import NotFoundError from django_comment_common.utils import are_permissions_roles_seeded from xmodule.exceptions import InvalidVersionError @@ -271,7 +274,7 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase): ) self.assertTrue(getattr(draft_problem, 'is_draft', False)) - #now requery with depth + # now requery with depth course = modulestore('draft').get_item( Location(['i4x', 'edX', 'simple', 'course', '2012_Fall', None]), depth=None @@ -382,6 +385,159 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase): course = module_store.get_item(source_location) self.assertFalse(course.hide_progress_tab) + def test_asset_import(self): + ''' + This test validates that an image asset is imported and a thumbnail was generated for a .gif + ''' + content_store = contentstore() + + module_store = modulestore('direct') + import_from_xml(module_store, 'common/test/data/', ['full'], static_content_store=content_store) + + course_location = CourseDescriptor.id_to_location('edX/full/6.002_Spring_2012') + course = module_store.get_item(course_location) + + self.assertIsNotNone(course) + + # make sure we have some assets in our contentstore + all_assets = content_store.get_all_content_for_course(course_location) + self.assertGreater(len(all_assets), 0) + + # make sure we have some thumbnails in our contentstore + all_thumbnails = content_store.get_all_content_thumbnails_for_course(course_location) + + # + # cdodge: temporarily comment out assertion on thumbnails because many environments + # will not have the jpeg converter installed and this test will fail + # + # + # self.assertGreater(len(all_thumbnails), 0) + + content = None + try: + location = StaticContent.get_location_from_path('/c4x/edX/full/asset/circuits_duality.gif') + content = content_store.find(location) + except NotFoundError: + pass + + self.assertIsNotNone(content) + + # + # cdodge: temporarily comment out assertion on thumbnails because many environments + # will not have the jpeg converter installed and this test will fail + # + # self.assertIsNotNone(content.thumbnail_location) + # + # thumbnail = None + # try: + # thumbnail = content_store.find(content.thumbnail_location) + # except: + # pass + # + # self.assertIsNotNone(thumbnail) + + def test_asset_delete_and_restore(self): + ''' + This test will exercise the soft delete/restore functionality of the assets + ''' + content_store = contentstore() + trash_store = contentstore('trashcan') + module_store = modulestore('direct') + + import_from_xml(module_store, 'common/test/data/', ['full'], static_content_store=content_store) + + # look up original (and thumbnail) in content store, should be there after import + location = StaticContent.get_location_from_path('/c4x/edX/full/asset/circuits_duality.gif') + content = content_store.find(location, throw_on_not_found=False) + thumbnail_location = content.thumbnail_location + self.assertIsNotNone(content) + + # + # cdodge: temporarily comment out assertion on thumbnails because many environments + # will not have the jpeg converter installed and this test will fail + # + # self.assertIsNotNone(thumbnail_location) + + # go through the website to do the delete, since the soft-delete logic is in the view + + url = reverse('remove_asset', kwargs={'org': 'edX', 'course': 'full', 'name': '6.002_Spring_2012'}) + resp = self.client.post(url, {'location': '/c4x/edX/full/asset/circuits_duality.gif'}) + self.assertEqual(resp.status_code, 200) + + asset_location = StaticContent.get_location_from_path('/c4x/edX/full/asset/circuits_duality.gif') + + # now try to find it in store, but they should not be there any longer + content = content_store.find(asset_location, throw_on_not_found=False) + self.assertIsNone(content) + + if thumbnail_location: + thumbnail = content_store.find(thumbnail_location, throw_on_not_found=False) + self.assertIsNone(thumbnail) + + # now try to find it and the thumbnail in trashcan - should be in there + content = trash_store.find(asset_location, throw_on_not_found=False) + self.assertIsNotNone(content) + + if thumbnail_location: + thumbnail = trash_store.find(thumbnail_location, throw_on_not_found=False) + self.assertIsNotNone(thumbnail) + + # let's restore the asset + restore_asset_from_trashcan('/c4x/edX/full/asset/circuits_duality.gif') + + # now try to find it in courseware store, and they should be back after restore + content = content_store.find(asset_location, throw_on_not_found=False) + self.assertIsNotNone(content) + + if thumbnail_location: + thumbnail = content_store.find(thumbnail_location, throw_on_not_found=False) + self.assertIsNotNone(thumbnail) + + def test_empty_trashcan(self): + ''' + This test will exercise the empting of the asset trashcan + ''' + content_store = contentstore() + trash_store = contentstore('trashcan') + module_store = modulestore('direct') + + import_from_xml(module_store, 'common/test/data/', ['full'], static_content_store=content_store) + + course_location = CourseDescriptor.id_to_location('edX/full/6.002_Spring_2012') + + location = StaticContent.get_location_from_path('/c4x/edX/full/asset/circuits_duality.gif') + content = content_store.find(location, throw_on_not_found=False) + self.assertIsNotNone(content) + + # go through the website to do the delete, since the soft-delete logic is in the view + + url = reverse('remove_asset', kwargs={'org': 'edX', 'course': 'full', 'name': '6.002_Spring_2012'}) + resp = self.client.post(url, {'location': '/c4x/edX/full/asset/circuits_duality.gif'}) + self.assertEqual(resp.status_code, 200) + + # make sure there's something in the trashcan + all_assets = trash_store.get_all_content_for_course(course_location) + self.assertGreater(len(all_assets), 0) + + # make sure we have some thumbnails in our trashcan + all_thumbnails = trash_store.get_all_content_thumbnails_for_course(course_location) + # + # cdodge: temporarily comment out assertion on thumbnails because many environments + # will not have the jpeg converter installed and this test will fail + # + # self.assertGreater(len(all_thumbnails), 0) + + # empty the trashcan + empty_asset_trashcan([course_location]) + + # make sure trashcan is empty + all_assets = trash_store.get_all_content_for_course(course_location) + self.assertEqual(len(all_assets), 0) + + + all_thumbnails = trash_store.get_all_content_thumbnails_for_course(course_location) + self.assertEqual(len(all_thumbnails), 0) + def test_clone_course(self): course_data = { @@ -539,7 +695,7 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase): on_disk = loads(grading_policy.read()) self.assertEqual(on_disk, course.grading_policy) - #check for policy.json + # check for policy.json self.assertTrue(filesystem.exists('policy.json')) # compare what's on disk to what we have in the course module diff --git a/cms/djangoapps/contentstore/tests/test_course_settings.py b/cms/djangoapps/contentstore/tests/test_course_settings.py index 2a4ff46038..8c15b1ae95 100644 --- a/cms/djangoapps/contentstore/tests/test_course_settings.py +++ b/cms/djangoapps/contentstore/tests/test_course_settings.py @@ -54,6 +54,7 @@ class CourseDetailsTestCase(CourseTestCase): def test_virgin_fetch(self): details = CourseDetails.fetch(self.course_location) self.assertEqual(details.course_location, self.course_location, "Location not copied into") + self.assertIsNotNone(details.start_date.tzinfo) self.assertIsNone(details.end_date, "end date somehow initialized " + str(details.end_date)) self.assertIsNone(details.enrollment_start, "enrollment_start date somehow initialized " + str(details.enrollment_start)) self.assertIsNone(details.enrollment_end, "enrollment_end date somehow initialized " + str(details.enrollment_end)) @@ -67,7 +68,6 @@ class CourseDetailsTestCase(CourseTestCase): jsondetails = json.dumps(details, cls=CourseSettingsEncoder) jsondetails = json.loads(jsondetails) self.assertTupleEqual(Location(jsondetails['course_location']), self.course_location, "Location !=") - # Note, start_date is being initialized someplace. I'm not sure why b/c the default will make no sense. self.assertIsNone(jsondetails['end_date'], "end date somehow initialized ") self.assertIsNone(jsondetails['enrollment_start'], "enrollment_start date somehow initialized ") self.assertIsNone(jsondetails['enrollment_end'], "enrollment_end date somehow initialized ") @@ -76,6 +76,23 @@ class CourseDetailsTestCase(CourseTestCase): self.assertIsNone(jsondetails['intro_video'], "intro_video somehow initialized") self.assertIsNone(jsondetails['effort'], "effort somehow initialized") + def test_ooc_encoder(self): + """ + Test the encoder out of its original constrained purpose to see if it functions for general use + """ + details = {'location': Location(['tag', 'org', 'course', 'category', 'name']), + 'number': 1, + 'string': 'string', + 'datetime': datetime.datetime.now(UTC())} + jsondetails = json.dumps(details, cls=CourseSettingsEncoder) + jsondetails = json.loads(jsondetails) + + self.assertIn('location', jsondetails) + self.assertIn('org', jsondetails['location']) + self.assertEquals('org', jsondetails['location'][1]) + self.assertEquals(1, jsondetails['number']) + self.assertEqual(jsondetails['string'], 'string') + def test_update_and_fetch(self): # # NOTE: I couldn't figure out how to validly test time setting w/ all the conversions jsondetails = CourseDetails.fetch(self.course_location) @@ -116,11 +133,8 @@ class CourseDetailsViewTest(CourseTestCase): self.compare_details_with_encoding(json.loads(resp.content), details.__dict__, field + str(val)) @staticmethod - def convert_datetime_to_iso(datetime): - if datetime is not None: - return datetime.isoformat("T") - else: - return None + def convert_datetime_to_iso(dt): + return Date().to_json(dt) def test_update_and_fetch(self): details = CourseDetails.fetch(self.course_location) @@ -151,22 +165,12 @@ class CourseDetailsViewTest(CourseTestCase): self.assertEqual(details['intro_video'], encoded.get('intro_video', None), context + " intro_video not ==") self.assertEqual(details['effort'], encoded['effort'], context + " efforts not ==") - @staticmethod - def struct_to_datetime(struct_time): - return datetime.datetime(*struct_time[:6], tzinfo=UTC()) - def compare_date_fields(self, details, encoded, context, field): if details[field] is not None: date = Date() if field in encoded and encoded[field] is not None: - encoded_encoded = date.from_json(encoded[field]) - dt1 = CourseDetailsViewTest.struct_to_datetime(encoded_encoded) - - if isinstance(details[field], datetime.datetime): - dt2 = details[field] - else: - details_encoded = date.from_json(details[field]) - dt2 = CourseDetailsViewTest.struct_to_datetime(details_encoded) + dt1 = date.from_json(encoded[field]) + dt2 = details[field] expected_delta = datetime.timedelta(0) self.assertEqual(dt1 - dt2, expected_delta, str(dt1) + "!=" + str(dt2) + " at " + context) diff --git a/cms/djangoapps/contentstore/tests/tests.py b/cms/djangoapps/contentstore/tests/tests.py index f769652493..f7f330f91e 100644 --- a/cms/djangoapps/contentstore/tests/tests.py +++ b/cms/djangoapps/contentstore/tests/tests.py @@ -3,6 +3,10 @@ from django.core.urlresolvers import reverse from .utils import parse_json, user, registration from xmodule.modulestore.tests.django_utils import ModuleStoreTestCase +from contentstore.tests.test_course_settings import CourseTestCase +from xmodule.modulestore.tests.factories import CourseFactory +import datetime +from pytz import UTC class ContentStoreTestCase(ModuleStoreTestCase): @@ -162,3 +166,21 @@ class AuthTestCase(ContentStoreTestCase): self.assertEqual(resp.status_code, 302) # Logged in should work. + + +class ForumTestCase(CourseTestCase): + def setUp(self): + """ Creates the test course. """ + super(ForumTestCase, self).setUp() + self.course = CourseFactory.create(org='testX', number='727', display_name='Forum Course') + + def test_blackouts(self): + now = datetime.datetime.now(UTC) + self.course.discussion_blackouts = [(t.isoformat(), t2.isoformat()) for t, t2 in + [(now - datetime.timedelta(days=14), now - datetime.timedelta(days=11)), + (now + datetime.timedelta(days=24), now + datetime.timedelta(days=30))]] + self.assertTrue(self.course.forum_posts_allowed) + self.course.discussion_blackouts = [(t.isoformat(), t2.isoformat()) for t, t2 in + [(now - datetime.timedelta(days=14), now + datetime.timedelta(days=2)), + (now + datetime.timedelta(days=24), now + datetime.timedelta(days=30))]] + self.assertFalse(self.course.forum_posts_allowed) diff --git a/cms/djangoapps/contentstore/views/assets.py b/cms/djangoapps/contentstore/views/assets.py index b5041d3e9f..400013b59b 100644 --- a/cms/djangoapps/contentstore/views/assets.py +++ b/cms/djangoapps/contentstore/views/assets.py @@ -25,6 +25,8 @@ from xmodule.modulestore.django import modulestore from xmodule.modulestore import Location from xmodule.contentstore.content import StaticContent from xmodule.util.date_utils import get_default_time_display +from xmodule.modulestore import InvalidLocationError +from xmodule.exceptions import NotFoundError from ..utils import get_url_reverse from .access import get_location_and_verify_access @@ -62,7 +64,7 @@ def asset_index(request, org, course, name): asset_id = asset['_id'] display_info = {} display_info['displayname'] = asset['displayname'] - display_info['uploadDate'] = get_default_time_display(asset['uploadDate'].timetuple()) + display_info['uploadDate'] = get_default_time_display(asset['uploadDate']) asset_location = StaticContent.compute_location(asset_id['org'], asset_id['course'], asset_id['name']) display_info['url'] = StaticContent.get_url_path_from_location(asset_location) @@ -78,10 +80,17 @@ def asset_index(request, org, course, name): 'active_tab': 'assets', 'context_course': course_module, 'assets': asset_display, - 'upload_asset_callback_url': upload_asset_callback_url + 'upload_asset_callback_url': upload_asset_callback_url, + 'remove_asset_callback_url': reverse('remove_asset', kwargs={ + 'org': org, + 'course': course, + 'name': name + }) }) +@login_required +@ensure_csrf_cookie def upload_asset(request, org, course, coursename): ''' cdodge: this method allows for POST uploading of files into the course asset library, which will @@ -103,6 +112,9 @@ def upload_asset(request, org, course, coursename): logging.error('Could not find course' + location) return HttpResponseBadRequest() + if 'file' not in request.FILES: + return HttpResponseBadRequest() + # compute a 'filename' which is similar to the location formatting, we're using the 'filename' # nomenclature since we're using a FileSystem paradigm here. We're just imposing # the Location string formatting expectations to keep things a bit more consistent @@ -131,7 +143,7 @@ def upload_asset(request, org, course, coursename): readback = contentstore().find(content.location) response_payload = {'displayname': content.name, - 'uploadDate': get_default_time_display(readback.last_modified_at.timetuple()), + 'uploadDate': get_default_time_display(readback.last_modified_at), 'url': StaticContent.get_url_path_from_location(content.location), 'thumb_url': StaticContent.get_url_path_from_location(thumbnail_location) if thumbnail_content is not None else None, 'msg': 'Upload completed' @@ -142,6 +154,57 @@ def upload_asset(request, org, course, coursename): return response +@ensure_csrf_cookie +@login_required +def remove_asset(request, org, course, name): + ''' + This method will perform a 'soft-delete' of an asset, which is basically to copy the asset from + the main GridFS collection and into a Trashcan + ''' + get_location_and_verify_access(request, org, course, name) + + location = request.POST['location'] + + # make sure the location is valid + try: + loc = StaticContent.get_location_from_path(location) + except InvalidLocationError: + # return a 'Bad Request' to browser as we have a malformed Location + response = HttpResponse() + response.status_code = 400 + return response + + # also make sure the item to delete actually exists + try: + content = contentstore().find(loc) + except NotFoundError: + response = HttpResponse() + response.status_code = 404 + return response + + # ok, save the content into the trashcan + contentstore('trashcan').save(content) + + # see if there is a thumbnail as well, if so move that as well + if content.thumbnail_location is not None: + try: + thumbnail_content = contentstore().find(content.thumbnail_location) + contentstore('trashcan').save(thumbnail_content) + # hard delete thumbnail from origin + contentstore().delete(thumbnail_content.get_id()) + # remove from any caching + del_cached_content(thumbnail_content.location) + except: + pass # OK if this is left dangling + + # delete the original + contentstore().delete(content.get_id()) + # remove from cache + del_cached_content(content.location) + + return HttpResponse() + + @ensure_csrf_cookie @login_required def import_course(request, org, course, name): @@ -227,11 +290,9 @@ def generate_export_course(request, org, course, name): root_dir = path(mkdtemp()) # export out to a tempdir - logging.debug('root = {0}'.format(root_dir)) export_to_xml(modulestore('direct'), contentstore(), loc, root_dir, name, modulestore()) - #filename = root_dir / name + '.tar.gz' logging.debug('tar file being generated at {0}'.format(export_file.name)) tar_file = tarfile.open(name=export_file.name, mode='w:gz') diff --git a/cms/djangoapps/contentstore/views/course.py b/cms/djangoapps/contentstore/views/course.py index 07f6b9669c..8762eb3a2a 100644 --- a/cms/djangoapps/contentstore/views/course.py +++ b/cms/djangoapps/contentstore/views/course.py @@ -2,7 +2,6 @@ Views related to operations on course objects """ import json -import time from django.contrib.auth.decorators import login_required from django_future.csrf import ensure_csrf_cookie @@ -32,6 +31,8 @@ from .component import OPEN_ENDED_COMPONENT_TYPES, \ NOTE_COMPONENT_TYPES, ADVANCED_COMPONENT_POLICY_KEY from django_comment_common.utils import seed_permissions_roles +import datetime +from django.utils.timezone import UTC # TODO: should explicitly enumerate exports with __all__ @@ -130,7 +131,7 @@ def create_new_course(request): new_course.display_name = display_name # set a default start date to now - new_course.start = time.gmtime() + new_course.start = datetime.datetime.now(UTC()) initialize_course_tabs(new_course) @@ -357,52 +358,55 @@ def course_advanced_updates(request, org, course, name): # Whether or not to filter the tabs key out of the settings metadata filter_tabs = True - #Check to see if the user instantiated any advanced components. This is a hack - #that does the following : - # 1) adds/removes the open ended panel tab to a course automatically if the user + # Check to see if the user instantiated any advanced components. This is a hack + # that does the following : + # 1) adds/removes the open ended panel tab to a course automatically if the user # has indicated that they want to edit the combinedopendended or peergrading module # 2) adds/removes the notes panel tab to a course automatically if the user has # indicated that they want the notes module enabled in their course # TODO refactor the above into distinct advanced policy settings if ADVANCED_COMPONENT_POLICY_KEY in request_body: - #Get the course so that we can scrape current tabs + # Get the course so that we can scrape current tabs course_module = modulestore().get_item(location) - #Maps tab types to components + # Maps tab types to components tab_component_map = { - 'open_ended': OPEN_ENDED_COMPONENT_TYPES, + 'open_ended': OPEN_ENDED_COMPONENT_TYPES, 'notes': NOTE_COMPONENT_TYPES, } - #Check to see if the user instantiated any notes or open ended components + # Check to see if the user instantiated any notes or open ended components for tab_type in tab_component_map.keys(): component_types = tab_component_map.get(tab_type) found_ac_type = False for ac_type in component_types: if ac_type in request_body[ADVANCED_COMPONENT_POLICY_KEY]: - #Add tab to the course if needed + # Add tab to the course if needed changed, new_tabs = add_extra_panel_tab(tab_type, course_module) - #If a tab has been added to the course, then send the metadata along to CourseMetadata.update_from_json + # If a tab has been added to the course, then send the metadata along to CourseMetadata.update_from_json if changed: course_module.tabs = new_tabs request_body.update({'tabs': new_tabs}) - #Indicate that tabs should not be filtered out of the metadata + # Indicate that tabs should not be filtered out of the metadata filter_tabs = False - #Set this flag to avoid the tab removal code below. + # Set this flag to avoid the tab removal code below. found_ac_type = True break - #If we did not find a module type in the advanced settings, + # If we did not find a module type in the advanced settings, # we may need to remove the tab from the course. if not found_ac_type: - #Remove tab from the course if needed + # Remove tab from the course if needed changed, new_tabs = remove_extra_panel_tab(tab_type, course_module) if changed: course_module.tabs = new_tabs request_body.update({'tabs': new_tabs}) - #Indicate that tabs should *not* be filtered out of the metadata + # Indicate that tabs should *not* be filtered out of the metadata filter_tabs = False - - response_json = json.dumps(CourseMetadata.update_from_json(location, + try: + response_json = json.dumps(CourseMetadata.update_from_json(location, request_body, filter_tabs=filter_tabs)) + except (TypeError, ValueError), e: + return HttpResponseBadRequest("Incorrect setting format. " + str(e), content_type="text/plain") + return HttpResponse(response_json, mimetype="application/json") diff --git a/cms/djangoapps/models/settings/course_details.py b/cms/djangoapps/models/settings/course_details.py index 0dbb47b31b..07eb4bc309 100644 --- a/cms/djangoapps/models/settings/course_details.py +++ b/cms/djangoapps/models/settings/course_details.py @@ -3,26 +3,26 @@ from xmodule.modulestore.exceptions import ItemNotFoundError from xmodule.modulestore.inheritance import own_metadata import json from json.encoder import JSONEncoder -import time from contentstore.utils import get_modulestore from models.settings import course_grading from contentstore.utils import update_item from xmodule.fields import Date import re import logging +import datetime class CourseDetails(object): def __init__(self, location): - self.course_location = location # a Location obj + self.course_location = location # a Location obj self.start_date = None # 'start' - self.end_date = None # 'end' + self.end_date = None # 'end' self.enrollment_start = None self.enrollment_end = None - self.syllabus = None # a pdf file asset - self.overview = "" # html to render as the overview - self.intro_video = None # a video pointer - self.effort = None # int hours/week + self.syllabus = None # a pdf file asset + self.overview = "" # html to render as the overview + self.intro_video = None # a video pointer + self.effort = None # int hours/week @classmethod def fetch(cls, course_location): @@ -73,9 +73,9 @@ class CourseDetails(object): """ Decode the json into CourseDetails and save any changed attrs to the db """ - ## TODO make it an error for this to be undefined & for it to not be retrievable from modulestore + # TODO make it an error for this to be undefined & for it to not be retrievable from modulestore course_location = jsondict['course_location'] - ## Will probably want to cache the inflight courses because every blur generates an update + # Will probably want to cache the inflight courses because every blur generates an update descriptor = get_modulestore(course_location).get_item(course_location) dirty = False @@ -181,7 +181,7 @@ class CourseSettingsEncoder(json.JSONEncoder): return obj.__dict__ elif isinstance(obj, Location): return obj.dict() - elif isinstance(obj, time.struct_time): + elif isinstance(obj, datetime.datetime): return Date().to_json(obj) else: return JSONEncoder.default(self, obj) diff --git a/cms/envs/acceptance.py b/cms/envs/acceptance.py index 36616ab257..6293219f43 100644 --- a/cms/envs/acceptance.py +++ b/cms/envs/acceptance.py @@ -23,7 +23,7 @@ MODULESTORE_OPTIONS = { 'db': 'test_xmodule', 'collection': 'acceptance_modulestore', 'fs_root': TEST_ROOT / "data", - 'render_template': 'mitxmako.shortcuts.render_to_string', + 'render_template': 'mitxmako.shortcuts.render_to_string' } MODULESTORE = { diff --git a/cms/envs/aws.py b/cms/envs/aws.py index 35b15fe6ba..c6a383211f 100644 --- a/cms/envs/aws.py +++ b/cms/envs/aws.py @@ -112,9 +112,6 @@ TIME_ZONE = ENV_TOKENS.get('TIME_ZONE', TIME_ZONE) for feature, value in ENV_TOKENS.get('MITX_FEATURES', {}).items(): MITX_FEATURES[feature] = value -# load segment.io key, provide a dummy if it does not exist -SEGMENT_IO_KEY = ENV_TOKENS.get('SEGMENT_IO_KEY', '***REMOVED***') - LOGGING = get_logger_config(LOG_DIR, logging_env=ENV_TOKENS['LOGGING_ENV'], syslog_addr=(ENV_TOKENS['SYSLOG_SERVER'], 514), @@ -126,6 +123,13 @@ LOGGING = get_logger_config(LOG_DIR, with open(ENV_ROOT / CONFIG_PREFIX + "auth.json") as auth_file: AUTH_TOKENS = json.load(auth_file) +# If Segment.io key specified, load it and turn on Segment.io if the feature flag is set +# Note that this is the Studio key. There is a separate key for the LMS. +SEGMENT_IO_KEY = AUTH_TOKENS.get('SEGMENT_IO_KEY') +if SEGMENT_IO_KEY: + MITX_FEATURES['SEGMENT_IO'] = ENV_TOKENS.get('SEGMENT_IO', False) + + AWS_ACCESS_KEY_ID = AUTH_TOKENS["AWS_ACCESS_KEY_ID"] AWS_SECRET_ACCESS_KEY = AUTH_TOKENS["AWS_SECRET_ACCESS_KEY"] DATABASES = AUTH_TOKENS['DATABASES'] diff --git a/cms/envs/common.py b/cms/envs/common.py index 22e69fa08a..8551a56c41 100644 --- a/cms/envs/common.py +++ b/cms/envs/common.py @@ -25,19 +25,30 @@ Longer TODO: import sys import lms.envs.common +from lms.envs.common import USE_TZ from path import path ############################ FEATURE CONFIGURATION ############################# MITX_FEATURES = { 'USE_DJANGO_PIPELINE': True, + 'GITHUB_PUSH': False, + 'ENABLE_DISCUSSION_SERVICE': False, + 'AUTH_USE_MIT_CERTIFICATES': False, - 'STUB_VIDEO_FOR_TESTING': False, # do not display video when running automated acceptance tests - 'STAFF_EMAIL': '', # email address for staff (eg to request course creation) + + # do not display video when running automated acceptance tests + 'STUB_VIDEO_FOR_TESTING': False, + + # email address for staff (eg to request course creation) + 'STAFF_EMAIL': '', + 'STUDIO_NPS_SURVEY': True, - 'SEGMENT_IO': True, + + # Segment.io - must explicitly turn it on for production + 'SEGMENT_IO': False, # Enable URL that shows information about the status of various services 'ENABLE_SERVICE_STATUS': False, @@ -183,7 +194,7 @@ STATICFILES_DIRS = [ # Locale/Internationalization TIME_ZONE = 'America/New_York' # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name -LANGUAGE_CODE = 'en' # http://www.i18nguy.com/unicode/language-identifiers.html +LANGUAGE_CODE = 'en' # http://www.i18nguy.com/unicode/language-identifiers.html USE_I18N = True USE_L10N = True @@ -227,7 +238,8 @@ PIPELINE_JS = { ) + ['js/hesitate.js', 'js/base.js', 'js/models/feedback.js', 'js/views/feedback.js', 'js/models/section.js', 'js/views/section.js', - 'js/models/metadata_model.js', 'js/views/metadata_editor_view.js'], + 'js/models/metadata_model.js', 'js/views/metadata_editor_view.js', + 'js/views/assets.js'], 'output_filename': 'js/cms-application.js', 'test_order': 0 }, diff --git a/cms/envs/dev.py b/cms/envs/dev.py index eea236f0e2..07630bdf31 100644 --- a/cms/envs/dev.py +++ b/cms/envs/dev.py @@ -22,7 +22,7 @@ modulestore_options = { 'db': 'xmodule', 'collection': 'modulestore', 'fs_root': GITHUB_REPO_ROOT, - 'render_template': 'mitxmako.shortcuts.render_to_string', + 'render_template': 'mitxmako.shortcuts.render_to_string' } MODULESTORE = { @@ -43,10 +43,15 @@ CONTENTSTORE = { 'OPTIONS': { 'host': 'localhost', 'db': 'xcontent', + }, + # allow for additional options that can be keyed on a name, e.g. 'trashcan' + 'ADDITIONAL_OPTIONS': { + 'trashcan': { + 'bucket': 'trash_fs' + } } } - DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', @@ -64,7 +69,7 @@ REPOS = { }, 'content-mit-6002x': { 'branch': 'master', - #'origin': 'git@github.com:MITx/6002x-fall-2012.git', + # 'origin': 'git@github.com:MITx/6002x-fall-2012.git', 'origin': 'git@github.com:MITx/content-mit-6002x.git', }, '6.00x': { @@ -163,8 +168,14 @@ MITX_FEATURES['STUDIO_NPS_SURVEY'] = False # Enable URL that shows information about the status of variuous services MITX_FEATURES['ENABLE_SERVICE_STATUS'] = True -# segment-io key for dev -SEGMENT_IO_KEY = 'mty8edrrsg' +############################# SEGMENT-IO ################################## + +# If there's an environment variable set, grab it and turn on Segment.io +# Note that this is the Studio key. There is a separate key for the LMS. +import os +SEGMENT_IO_KEY = os.environ.get('SEGMENT_IO_KEY') +if SEGMENT_IO_KEY: + MITX_FEATURES['SEGMENT_IO'] = True ##################################################################### diff --git a/cms/envs/test.py b/cms/envs/test.py index 8a3f9ba158..954a553e10 100644 --- a/cms/envs/test.py +++ b/cms/envs/test.py @@ -48,7 +48,7 @@ MODULESTORE_OPTIONS = { 'db': 'test_xmodule', 'collection': 'test_modulestore', 'fs_root': TEST_ROOT / "data", - 'render_template': 'mitxmako.shortcuts.render_to_string', + 'render_template': 'mitxmako.shortcuts.render_to_string' } MODULESTORE = { @@ -70,7 +70,13 @@ CONTENTSTORE = { 'ENGINE': 'xmodule.contentstore.mongo.MongoContentStore', 'OPTIONS': { 'host': 'localhost', - 'db': 'xcontent', + 'db': 'test_xmodule', + }, + # allow for additional options that can be keyed on a name, e.g. 'trashcan' + 'ADDITIONAL_OPTIONS': { + 'trashcan': { + 'bucket': 'trash_fs' + } } } @@ -121,7 +127,7 @@ CELERY_RESULT_BACKEND = 'cache' BROKER_TRANSPORT = 'memory' ################### Make tests faster -#http://slacy.com/blog/2012/04/make-your-tests-faster-in-django-1-4/ +# http://slacy.com/blog/2012/04/make-your-tests-faster-in-django-1-4/ PASSWORD_HASHERS = ( 'django.contrib.auth.hashers.SHA1PasswordHasher', 'django.contrib.auth.hashers.MD5PasswordHasher', diff --git a/cms/pydev_manage.py b/cms/pydev_manage.py new file mode 100644 index 0000000000..22c38d89eb --- /dev/null +++ b/cms/pydev_manage.py @@ -0,0 +1,11 @@ +''' +Used for pydev eclipse. Should be innocuous for everyone else. +Created on May 8, 2013 + +@author: dmitchell +''' +#!/home//mitx_all/python/bin/python +from django.core import management + +if __name__ == '__main__': + management.execute_from_command_line() diff --git a/cms/static/coffee/src/views/module_edit.coffee b/cms/static/coffee/src/views/module_edit.coffee index d0a76a6c15..5154591d6f 100644 --- a/cms/static/coffee/src/views/module_edit.coffee +++ b/cms/static/coffee/src/views/module_edit.coffee @@ -44,8 +44,17 @@ class CMS.Views.ModuleEdit extends Backbone.View [@metadataEditor.getDisplayName()]) @$el.find('.component-name').html(title) + customMetadata: -> + # Hack to support metadata fields that aren't part of the metadata editor (ie, LaTeX high level source). + # Walk through the set of elements which have the 'data-metadata_name' attribute and + # build up an object to pass back to the server on the subsequent POST. + # Note that these values will always be sent back on POST, even if they did not actually change. + _metadata = {} + _metadata[$(el).data("metadata-name")] = el.value for el in $('[data-metadata-name]', @$component_editor()) + return _metadata + changedMetadata: -> - return @metadataEditor.getModifiedMetadataValues() + return _.extend(@metadataEditor.getModifiedMetadataValues(), @customMetadata()) cloneTemplate: (parent, template) -> $.post("/clone_item", { diff --git a/cms/static/js/base.js b/cms/static/js/base.js index fe60d80239..92a16b8417 100644 --- a/cms/static/js/base.js +++ b/cms/static/js/base.js @@ -32,8 +32,6 @@ $(document).ready(function() { $modal.bind('click', hideModal); $modalCover.bind('click', hideModal); - $('.uploads .upload-button').bind('click', showUploadModal); - $('.upload-modal .close-button').bind('click', hideModal); $body.on('click', '.embeddable-xml-input', function() { $(this).select(); @@ -145,8 +143,6 @@ $(document).ready(function() { $('.edit-section-start-cancel').bind('click', cancelSetSectionScheduleDate); $('.edit-section-start-save').bind('click', saveSetSectionScheduleDate); - $('.upload-modal .choose-file-button').bind('click', showFileSelectionMenu); - $body.on('click', '.section-published-date .edit-button', editSectionPublishDate); $body.on('click', '.section-published-date .schedule-button', editSectionPublishDate); $body.on('click', '.edit-subsection-publish-settings .save-button', saveSetSectionScheduleDate); @@ -398,69 +394,6 @@ function _deleteItem($el) { }); } -function showUploadModal(e) { - e.preventDefault(); - $modal = $('.upload-modal').show(); - $('.file-input').bind('change', startUpload); - $modalCover.show(); -} - -function showFileSelectionMenu(e) { - e.preventDefault(); - $('.file-input').click(); -} - -function startUpload(e) { - $('.upload-modal h1').html(gettext('Uploading…')); - $('.upload-modal .file-name').html($('.file-input').val().replace('C:\\fakepath\\', '')); - $('.upload-modal .file-chooser').ajaxSubmit({ - beforeSend: resetUploadBar, - uploadProgress: showUploadFeedback, - complete: displayFinishedUpload - }); - $('.upload-modal .choose-file-button').hide(); - $('.upload-modal .progress-bar').removeClass('loaded').show(); -} - -function resetUploadBar() { - var percentVal = '0%'; - $('.upload-modal .progress-fill').width(percentVal); - $('.upload-modal .progress-fill').html(percentVal); -} - -function showUploadFeedback(event, position, total, percentComplete) { - var percentVal = percentComplete + '%'; - $('.upload-modal .progress-fill').width(percentVal); - $('.upload-modal .progress-fill').html(percentVal); -} - -function displayFinishedUpload(xhr) { - if (xhr.status = 200) { - markAsLoaded(); - } - - var resp = JSON.parse(xhr.responseText); - $('.upload-modal .embeddable-xml-input').val(xhr.getResponseHeader('asset_url')); - $('.upload-modal .embeddable').show(); - $('.upload-modal .file-name').hide(); - $('.upload-modal .progress-fill').html(resp.msg); - $('.upload-modal .choose-file-button').html(gettext('Load Another File')).show(); - $('.upload-modal .progress-fill').width('100%'); - - // see if this id already exists, if so, then user must have updated an existing piece of content - $("tr[data-id='" + resp.url + "']").remove(); - - var template = $('#new-asset-element').html(); - var html = Mustache.to_html(template, resp); - $('table > tbody').prepend(html); - - analytics.track('Uploaded a File', { - 'course': course_location_analytics, - 'asset_url': resp.url - }); - -} - function markAsLoaded() { $('.upload-modal .copy-button').css('display', 'inline-block'); $('.upload-modal .progress-bar').addClass('loaded'); diff --git a/cms/static/js/models/feedback.js b/cms/static/js/models/feedback.js index 1f1ee57000..d57cffa779 100644 --- a/cms/static/js/models/feedback.js +++ b/cms/static/js/models/feedback.js @@ -42,6 +42,12 @@ CMS.Models.ErrorMessage = CMS.Models.SystemFeedback.extend({ }) }); +CMS.Models.ConfirmAssetDeleteMessage = CMS.Models.SystemFeedback.extend({ + defaults: $.extend({}, CMS.Models.SystemFeedback.prototype.defaults, { + "intent": "warning" + }) +}); + CMS.Models.ConfirmationMessage = CMS.Models.SystemFeedback.extend({ defaults: $.extend({}, CMS.Models.SystemFeedback.prototype.defaults, { "intent": "confirmation" diff --git a/cms/static/js/views/assets.js b/cms/static/js/views/assets.js new file mode 100644 index 0000000000..9eb521dcb6 --- /dev/null +++ b/cms/static/js/views/assets.js @@ -0,0 +1,128 @@ +$(document).ready(function() { + $('.uploads .upload-button').bind('click', showUploadModal); + $('.upload-modal .close-button').bind('click', hideModal); + $('.upload-modal .choose-file-button').bind('click', showFileSelectionMenu); + $('.remove-asset-button').bind('click', removeAsset); +}); + +function removeAsset(e){ + e.preventDefault(); + + var that = this; + var msg = new CMS.Models.ConfirmAssetDeleteMessage({ + title: gettext("Delete File Confirmation"), + message: gettext("Are you sure you wish to delete this item. It cannot be reversed!\n\nAlso any content that links/refers to this item will no longer work (e.g. broken images and/or links)"), + actions: { + primary: { + text: gettext("OK"), + click: function(view) { + // call the back-end to actually remove the asset + $.post(view.model.get('remove_asset_url'), + { 'location': view.model.get('asset_location') }, + function() { + // show the post-commit confirmation + $(".wrapper-alert-confirmation").addClass("is-shown").attr('aria-hidden','false'); + view.model.get('row_to_remove').remove(); + analytics.track('Deleted Asset', { + 'course': course_location_analytics, + 'id': view.model.get('asset_location') + }); + } + ); + view.hide(); + } + }, + secondary: [{ + text: gettext("Cancel"), + click: function(view) { + view.hide(); + } + }] + }, + remove_asset_url: $('.asset-library').data('remove-asset-callback-url'), + asset_location: $(this).closest('tr').data('id'), + row_to_remove: $(this).closest('tr') + }); + + // workaround for now. We can't spawn multiple instances of the Prompt View + // so for now, a bit of hackery to just make sure we have a single instance + // note: confirm_delete_prompt is in asset_index.html + if (confirm_delete_prompt === null) + confirm_delete_prompt = new CMS.Views.Prompt({model: msg}); + else + { + confirm_delete_prompt.model = msg; + confirm_delete_prompt.show(); + } + + return; +} + +function showUploadModal(e) { + e.preventDefault(); + $modal = $('.upload-modal').show(); + $('.file-input').bind('change', startUpload); + $modalCover.show(); +} + +function showFileSelectionMenu(e) { + e.preventDefault(); + $('.file-input').click(); +} + +function startUpload(e) { + var files = $('.file-input').get(0).files; + if (files.length === 0) + return; + + $('.upload-modal h1').html(gettext('Uploading…')); + $('.upload-modal .file-name').html(files[0].name); + $('.upload-modal .file-chooser').ajaxSubmit({ + beforeSend: resetUploadBar, + uploadProgress: showUploadFeedback, + complete: displayFinishedUpload + }); + $('.upload-modal .choose-file-button').hide(); + $('.upload-modal .progress-bar').removeClass('loaded').show(); +} + +function resetUploadBar() { + var percentVal = '0%'; + $('.upload-modal .progress-fill').width(percentVal); + $('.upload-modal .progress-fill').html(percentVal); +} + +function showUploadFeedback(event, position, total, percentComplete) { + var percentVal = percentComplete + '%'; + $('.upload-modal .progress-fill').width(percentVal); + $('.upload-modal .progress-fill').html(percentVal); +} + +function displayFinishedUpload(xhr) { + if (xhr.status == 200) { + markAsLoaded(); + } + + var resp = JSON.parse(xhr.responseText); + $('.upload-modal .embeddable-xml-input').val(xhr.getResponseHeader('asset_url')); + $('.upload-modal .embeddable').show(); + $('.upload-modal .file-name').hide(); + $('.upload-modal .progress-fill').html(resp.msg); + $('.upload-modal .choose-file-button').html(gettext('Load Another File')).show(); + $('.upload-modal .progress-fill').width('100%'); + + // see if this id already exists, if so, then user must have updated an existing piece of content + $("tr[data-id='" + resp.url + "']").remove(); + + var template = $('#new-asset-element').html(); + var html = Mustache.to_html(template, resp); + $('table > tbody').prepend(html); + + // re-bind the listeners to delete it + $('.remove-asset-button').bind('click', removeAsset); + + analytics.track('Uploaded a File', { + 'course': course_location_analytics, + 'asset_url': resp.url + }); +} \ No newline at end of file diff --git a/cms/static/sass/_base.scss b/cms/static/sass/_base.scss index e0e7543b8e..3713f83ae3 100644 --- a/cms/static/sass/_base.scss +++ b/cms/static/sass/_base.scss @@ -14,7 +14,7 @@ body { color: $gray-d2; } -body, input { +body, input, button { font-family: 'Open Sans', sans-serif; } diff --git a/cms/static/sass/views/_assets.scss b/cms/static/sass/views/_assets.scss index d01dd988ef..d4cff42ee9 100644 --- a/cms/static/sass/views/_assets.scss +++ b/cms/static/sass/views/_assets.scss @@ -76,6 +76,10 @@ body.course.uploads { width: 250px; } + .delete-col { + width: 20px; + } + .embeddable-xml-input { @include box-shadow(none); width: 100%; diff --git a/cms/templates/asset_index.html b/cms/templates/asset_index.html index f03a9012f8..e8dc523ba7 100644 --- a/cms/templates/asset_index.html +++ b/cms/templates/asset_index.html @@ -1,5 +1,6 @@ <%inherit file="base.html" /> <%! from django.core.urlresolvers import reverse %> +<%! from django.utils.translation import ugettext as _ %> <%block name="bodyclass">is-signedin course uploads <%block name="title">Files & Uploads @@ -7,6 +8,12 @@ <%block name="jsextra"> + + + <%block name="content"> @@ -30,6 +37,9 @@ + + + @@ -56,7 +66,7 @@
-
+
@@ -64,6 +74,7 @@ + @@ -86,6 +97,9 @@ + % endfor @@ -129,3 +143,21 @@ + +<%block name="view_alerts"> + +
+
+ + +
+

${_('Your file has been deleted.')}

+
+ + + + ${_('close alert')} + +
+
+ diff --git a/cms/templates/edit_subsection.html b/cms/templates/edit_subsection.html index 9bb9b3a506..cbce91ab44 100644 --- a/cms/templates/edit_subsection.html +++ b/cms/templates/edit_subsection.html @@ -1,7 +1,7 @@ <%inherit file="base.html" /> <%! import logging - from xmodule.util.date_utils import get_time_struct_display + from xmodule.util.date_utils import get_default_time_display %> <%! from django.core.urlresolvers import reverse %> @@ -36,11 +36,15 @@
- +
- +
% if subsection.lms.start != parent_item.lms.start and subsection.lms.start: @@ -48,7 +52,7 @@

The date above differs from the release date of ${parent_item.display_name_with_default}, which is unset. % else:

The date above differs from the release date of ${parent_item.display_name_with_default} – - ${get_time_struct_display(parent_item.lms.start, '%m/%d/%Y at %H:%M UTC')}. + ${get_default_time_display(parent_item.lms.start)}. % endif

% endif @@ -65,11 +69,15 @@
- +
- +
Remove due date
diff --git a/cms/templates/overview.html b/cms/templates/overview.html index d327c8b324..43d0afc263 100644 --- a/cms/templates/overview.html +++ b/cms/templates/overview.html @@ -1,7 +1,7 @@ <%inherit file="base.html" /> <%! import logging - from xmodule.util.date_utils import get_time_struct_display + from xmodule.util import date_utils %> <%! from django.core.urlresolvers import reverse %> <%block name="title">Course Outline @@ -154,14 +154,19 @@

<% - start_date_str = get_time_struct_display(section.lms.start, '%m/%d/%Y') - start_time_str = get_time_struct_display(section.lms.start, '%H:%M') + if section.lms.start is not None: + start_date_str = section.lms.start.strftime('%m/%d/%Y') + start_time_str = section.lms.start.strftime('%H:%M') + else: + start_date_str = '' + start_time_str = '' %> %if section.lms.start is None: This section has not been released. Schedule %else: - Will Release: ${get_time_struct_display(section.lms.start, '%m/%d/%Y at %H:%M UTC')} + Will Release: + ${date_utils.get_default_time_display(section.lms.start)} Edit %endif
diff --git a/cms/urls.py b/cms/urls.py index e7444de4e9..a9a7f0a68a 100644 --- a/cms/urls.py +++ b/cms/urls.py @@ -35,6 +35,8 @@ urlpatterns = ('', # nopep8 'contentstore.views.preview_dispatch', name='preview_dispatch'), url(r'^(?P[^/]+)/(?P[^/]+)/course/(?P[^/]+)/upload_asset$', 'contentstore.views.upload_asset', name='upload_asset'), + + url(r'^manage_users/(?P.*?)$', 'contentstore.views.manage_users', name='manage_users'), url(r'^add_user/(?P.*?)$', 'contentstore.views.add_user', name='add_user'), @@ -71,8 +73,11 @@ urlpatterns = ('', # nopep8 'contentstore.views.edit_static', name='edit_static'), url(r'^edit_tabs/(?P[^/]+)/(?P[^/]+)/course/(?P[^/]+)$', 'contentstore.views.edit_tabs', name='edit_tabs'), + url(r'^(?P[^/]+)/(?P[^/]+)/assets/(?P[^/]+)$', 'contentstore.views.asset_index', name='asset_index'), + url(r'^(?P[^/]+)/(?P[^/]+)/assets/(?P[^/]+)/remove$', + 'contentstore.views.assets.remove_asset', name='remove_asset'), # this is a generic method to return the data/metadata associated with a xmodule url(r'^module_info/(?P.*)$', diff --git a/cms/xmodule_namespace.py b/cms/xmodule_namespace.py index 4857fe68ca..eef4b41f37 100644 --- a/cms/xmodule_namespace.py +++ b/cms/xmodule_namespace.py @@ -5,7 +5,6 @@ Namespace defining common fields used by Studio for all blocks import datetime from xblock.core import Namespace, Scope, ModelType, String -from xmodule.fields import StringyBoolean class DateTuple(ModelType): @@ -28,4 +27,3 @@ class CmsNamespace(Namespace): """ published_date = DateTuple(help="Date when the module was published", scope=Scope.settings) published_by = String(help="Id of the user who published this module", scope=Scope.settings) - diff --git a/common/djangoapps/contentserver/middleware.py b/common/djangoapps/contentserver/middleware.py index 8e9e70046d..7deb0901aa 100644 --- a/common/djangoapps/contentserver/middleware.py +++ b/common/djangoapps/contentserver/middleware.py @@ -1,7 +1,4 @@ -import logging -import time - -from django.http import HttpResponse, Http404, HttpResponseNotModified +from django.http import HttpResponse, HttpResponseNotModified from xmodule.contentstore.django import contentstore from xmodule.contentstore.content import StaticContent, XASSET_LOCATION_TAG @@ -20,7 +17,7 @@ class StaticContentServer(object): # return a 'Bad Request' to browser as we have a malformed Location response = HttpResponse() response.status_code = 400 - return response + return response # first look in our cache so we don't have to round-trip to the DB content = get_cached_content(loc) diff --git a/common/djangoapps/mitxmako/tests.py b/common/djangoapps/mitxmako/tests.py index f419daa681..e7e56a9472 100644 --- a/common/djangoapps/mitxmako/tests.py +++ b/common/djangoapps/mitxmako/tests.py @@ -1,18 +1,15 @@ from django.test import TestCase from django.test.utils import override_settings from django.core.urlresolvers import reverse -from django.conf import settings from mitxmako.shortcuts import marketing_link from mock import patch -from nose.plugins.skip import SkipTest +from util.testing import UrlResetMixin -class ShortcutsTests(TestCase): + +class ShortcutsTests(UrlResetMixin, TestCase): """ Test the mitxmako shortcuts file """ - # TODO: fix this test. It is causing intermittent test failures on - # subsequent tests due to the way urls are loaded - raise SkipTest() @override_settings(MKTG_URLS={'ROOT': 'dummy-root', 'ABOUT': '/about-us'}) @override_settings(MKTG_URL_LINK_MAP={'ABOUT': 'login'}) def test_marketing_link(self): diff --git a/common/djangoapps/student/management/commands/assigngroups.py b/common/djangoapps/student/management/commands/assigngroups.py index fb7bfc85cd..5269c8690e 100644 --- a/common/djangoapps/student/management/commands/assigngroups.py +++ b/common/djangoapps/student/management/commands/assigngroups.py @@ -14,6 +14,7 @@ import sys import datetime import json +from pytz import UTC middleware.MakoMiddleware() @@ -32,7 +33,7 @@ def group_from_value(groups, v): class Command(BaseCommand): - help = \ + help = \ ''' Assign users to test groups. Takes a list of groups: a:0.3,b:0.4,c:0.3 file.txt "Testing something" @@ -75,7 +76,7 @@ Will log what happened to file.txt. utg = UserTestGroup() utg.name = group utg.description = json.dumps({"description": args[2]}, - {"time": datetime.datetime.utcnow().isoformat()}) + {"time": datetime.datetime.now(UTC).isoformat()}) group_objects[group] = utg group_objects[group].save() diff --git a/common/djangoapps/student/management/commands/pearson_export_cdd.py b/common/djangoapps/student/management/commands/pearson_export_cdd.py index bad98b9d25..efb4a55387 100644 --- a/common/djangoapps/student/management/commands/pearson_export_cdd.py +++ b/common/djangoapps/student/management/commands/pearson_export_cdd.py @@ -8,6 +8,7 @@ from django.conf import settings from django.core.management.base import BaseCommand, CommandError from student.models import TestCenterUser +from pytz import UTC class Command(BaseCommand): @@ -58,7 +59,7 @@ class Command(BaseCommand): def handle(self, **options): # update time should use UTC in order to be comparable to the user_updated_at # field - uploaded_at = datetime.utcnow() + uploaded_at = datetime.now(UTC) # if specified destination is an existing directory, then # create a filename for it automatically. If it doesn't exist, @@ -100,7 +101,7 @@ class Command(BaseCommand): extrasaction='ignore') writer.writeheader() for tcu in TestCenterUser.objects.order_by('id'): - if tcu.needs_uploading: # or dump_all + if tcu.needs_uploading: # or dump_all record = dict((csv_field, ensure_encoding(getattr(tcu, model_field))) for csv_field, model_field in Command.CSV_TO_MODEL_FIELDS.items()) diff --git a/common/djangoapps/student/management/commands/pearson_export_ead.py b/common/djangoapps/student/management/commands/pearson_export_ead.py index 03dbce0024..ec10ab1599 100644 --- a/common/djangoapps/student/management/commands/pearson_export_ead.py +++ b/common/djangoapps/student/management/commands/pearson_export_ead.py @@ -8,6 +8,7 @@ from django.conf import settings from django.core.management.base import BaseCommand, CommandError from student.models import TestCenterRegistration, ACCOMMODATION_REJECTED_CODE +from pytz import UTC class Command(BaseCommand): @@ -51,7 +52,7 @@ class Command(BaseCommand): def handle(self, **options): # update time should use UTC in order to be comparable to the user_updated_at # field - uploaded_at = datetime.utcnow() + uploaded_at = datetime.now(UTC) # if specified destination is an existing directory, then # create a filename for it automatically. If it doesn't exist, diff --git a/common/djangoapps/student/management/commands/pearson_import_conf_zip.py b/common/djangoapps/student/management/commands/pearson_import_conf_zip.py index d0b2938df0..2339383719 100644 --- a/common/djangoapps/student/management/commands/pearson_import_conf_zip.py +++ b/common/djangoapps/student/management/commands/pearson_import_conf_zip.py @@ -13,6 +13,7 @@ from django.core.management.base import BaseCommand, CommandError from django.conf import settings from student.models import TestCenterUser, TestCenterRegistration +from pytz import UTC class Command(BaseCommand): @@ -68,7 +69,7 @@ class Command(BaseCommand): Command.datadog_error("Found authorization record for user {}".format(registration.testcenter_user.user.username), eacfile.name) # now update the record: registration.upload_status = row['Status'] - registration.upload_error_message = row['Message'] + registration.upload_error_message = row['Message'] try: registration.processed_at = strftime('%Y-%m-%d %H:%M:%S', strptime(row['Date'], '%Y/%m/%d %H:%M:%S')) except ValueError as ve: @@ -80,7 +81,7 @@ class Command(BaseCommand): except ValueError as ve: Command.datadog_error("Bad AuthorizationID value found for {}: message {}".format(client_authorization_id, ve), eacfile.name) - registration.confirmed_at = datetime.utcnow() + registration.confirmed_at = datetime.now(UTC) registration.save() except TestCenterRegistration.DoesNotExist: Command.datadog_error("Failed to find record for client_auth_id {}".format(client_authorization_id), eacfile.name) diff --git a/common/djangoapps/student/management/commands/pearson_make_tc_registration.py b/common/djangoapps/student/management/commands/pearson_make_tc_registration.py index b10cf143a0..50e56bb4be 100644 --- a/common/djangoapps/student/management/commands/pearson_make_tc_registration.py +++ b/common/djangoapps/student/management/commands/pearson_make_tc_registration.py @@ -1,5 +1,4 @@ from optparse import make_option -from time import strftime from django.contrib.auth.models import User from django.core.management.base import BaseCommand, CommandError @@ -128,8 +127,8 @@ class Command(BaseCommand): exam = CourseDescriptor.TestCenterExam(course_id, exam_name, exam_info) # update option values for date_first and date_last to use YYYY-MM-DD format # instead of YYYY-MM-DDTHH:MM - our_options['eligibility_appointment_date_first'] = strftime("%Y-%m-%d", exam.first_eligible_appointment_date) - our_options['eligibility_appointment_date_last'] = strftime("%Y-%m-%d", exam.last_eligible_appointment_date) + our_options['eligibility_appointment_date_first'] = exam.first_eligible_appointment_date.strftime("%Y-%m-%d") + our_options['eligibility_appointment_date_last'] = exam.last_eligible_appointment_date.strftime("%Y-%m-%d") if exam is None: raise CommandError("Exam for course_id {} does not exist".format(course_id)) diff --git a/common/djangoapps/student/models.py b/common/djangoapps/student/models.py index ab68b05f4b..af93c34317 100644 --- a/common/djangoapps/student/models.py +++ b/common/djangoapps/student/models.py @@ -16,7 +16,6 @@ import json import logging import uuid from random import randint -from time import strftime from django.conf import settings @@ -27,6 +26,7 @@ from django.dispatch import receiver from django.forms import ModelForm, forms import comment_client as cc +from pytz import UTC log = logging.getLogger(__name__) @@ -54,7 +54,7 @@ class UserProfile(models.Model): class Meta: db_table = "auth_userprofile" - ## CRITICAL TODO/SECURITY + # CRITICAL TODO/SECURITY # Sanitize all fields. # This is not visible to other users, but could introduce holes later user = models.OneToOneField(User, unique=True, db_index=True, related_name='profile') @@ -254,7 +254,7 @@ class TestCenterUserForm(ModelForm): def update_and_save(self): new_user = self.save(commit=False) # create additional values here: - new_user.user_updated_at = datetime.utcnow() + new_user.user_updated_at = datetime.now(UTC) new_user.upload_status = '' new_user.save() log.info("Updated demographic information for user's test center exam registration: username \"{}\" ".format(new_user.user.username)) @@ -429,8 +429,8 @@ class TestCenterRegistration(models.Model): registration.course_id = exam.course_id registration.accommodation_request = accommodation_request.strip() registration.exam_series_code = exam.exam_series_code - registration.eligibility_appointment_date_first = strftime("%Y-%m-%d", exam.first_eligible_appointment_date) - registration.eligibility_appointment_date_last = strftime("%Y-%m-%d", exam.last_eligible_appointment_date) + registration.eligibility_appointment_date_first = exam.first_eligible_appointment_date.strftime("%Y-%m-%d") + registration.eligibility_appointment_date_last = exam.last_eligible_appointment_date.strftime("%Y-%m-%d") registration.client_authorization_id = cls._create_client_authorization_id() # accommodation_code remains blank for now, along with Pearson confirmation information return registration @@ -556,7 +556,7 @@ class TestCenterRegistrationForm(ModelForm): def update_and_save(self): registration = self.save(commit=False) # create additional values here: - registration.user_updated_at = datetime.utcnow() + registration.user_updated_at = datetime.now(UTC) registration.upload_status = '' registration.save() log.info("Updated registration information for user's test center exam registration: username \"{}\" course \"{}\", examcode \"{}\"".format(registration.testcenter_user.user.username, registration.course_id, registration.exam_series_code)) @@ -598,7 +598,7 @@ def unique_id_for_user(user): return h.hexdigest() -## TODO: Should be renamed to generic UserGroup, and possibly +# TODO: Should be renamed to generic UserGroup, and possibly # Given an optional field for type of group class UserTestGroup(models.Model): users = models.ManyToManyField(User, db_index=True) @@ -626,7 +626,6 @@ class Registration(models.Model): def activate(self): self.user.is_active = True self.user.save() - #self.delete() class PendingNameChange(models.Model): @@ -648,7 +647,7 @@ class CourseEnrollment(models.Model): created = models.DateTimeField(auto_now_add=True, null=True, db_index=True) class Meta: - unique_together = (('user', 'course_id'), ) + unique_together = (('user', 'course_id'),) def __unicode__(self): return "[CourseEnrollment] %s: %s (%s)" % (self.user, self.course_id, self.created) @@ -667,12 +666,12 @@ class CourseEnrollmentAllowed(models.Model): created = models.DateTimeField(auto_now_add=True, null=True, db_index=True) class Meta: - unique_together = (('email', 'course_id'), ) + unique_together = (('email', 'course_id'),) def __unicode__(self): return "[CourseEnrollmentAllowed] %s: %s (%s)" % (self.email, self.course_id, self.created) -#cache_relation(User.profile) +# cache_relation(User.profile) #### Helper methods for use from python manage.py shell and other classes. diff --git a/common/djangoapps/student/tests/factories.py b/common/djangoapps/student/tests/factories.py index d73bb6f01d..49864fcbd4 100644 --- a/common/djangoapps/student/tests/factories.py +++ b/common/djangoapps/student/tests/factories.py @@ -5,6 +5,7 @@ from django.contrib.auth.models import Group from datetime import datetime from factory import DjangoModelFactory, SubFactory, PostGenerationMethodCall, post_generation, Sequence from uuid import uuid4 +from pytz import UTC # Factories don't have __init__ methods, and are self documenting # pylint: disable=W0232 @@ -46,8 +47,8 @@ class UserFactory(DjangoModelFactory): is_staff = False is_active = True is_superuser = False - last_login = datetime(2012, 1, 1) - date_joined = datetime(2011, 1, 1) + last_login = datetime(2012, 1, 1, tzinfo=UTC) + date_joined = datetime(2011, 1, 1, tzinfo=UTC) @post_generation def profile(obj, create, extracted, **kwargs): diff --git a/common/djangoapps/student/views.py b/common/djangoapps/student/views.py index 87e9f8c804..f129f1b4b1 100644 --- a/common/djangoapps/student/views.py +++ b/common/djangoapps/student/views.py @@ -49,6 +49,7 @@ from courseware.views import get_module_for_descriptor, jump_to from courseware.model_data import ModelDataCache from statsd import statsd +from pytz import UTC log = logging.getLogger("mitx.student") Article = namedtuple('Article', 'title url author image deck publication publish_date') @@ -77,7 +78,7 @@ def index(request, extra_context={}, user=None): ''' # The course selection work is done in courseware.courses. - domain = settings.MITX_FEATURES.get('FORCE_UNIVERSITY_DOMAIN') # normally False + domain = settings.MITX_FEATURES.get('FORCE_UNIVERSITY_DOMAIN') # normally False # do explicit check, because domain=None is valid if domain == False: domain = request.META.get('HTTP_HOST') @@ -630,7 +631,7 @@ def create_account(request, post_override=None): # Ok, looks like everything is legit. Create the account. ret = _do_create_account(post_vars) - if isinstance(ret, HttpResponse): # if there was an error then return that + if isinstance(ret, HttpResponse): # if there was an error then return that return ret (user, profile, registration) = ret @@ -668,7 +669,7 @@ def create_account(request, post_override=None): if DoExternalAuth: eamap.user = login_user - eamap.dtsignup = datetime.datetime.now() + eamap.dtsignup = datetime.datetime.now(UTC) eamap.save() log.debug('Updated ExternalAuthMap for %s to be %s' % (post_vars['username'], eamap)) diff --git a/common/djangoapps/terrain/ui_helpers.py b/common/djangoapps/terrain/ui_helpers.py index ecd43eb719..b1c5f30467 100644 --- a/common/djangoapps/terrain/ui_helpers.py +++ b/common/djangoapps/terrain/ui_helpers.py @@ -3,6 +3,7 @@ from lettuce import world import time +import platform from urllib import quote_plus from selenium.common.exceptions import WebDriverException, StaleElementReferenceException from selenium.webdriver.support import expected_conditions as EC @@ -57,20 +58,28 @@ def css_find(css, wait_time=5): @world.absorb -def css_click(css_selector): +def css_click(css_selector, index=0, attempts=5): """ Perform a click on a CSS selector, retrying if it initially fails + This function will return if the click worked (since it is try/excepting all errors) """ assert is_css_present(css_selector) - try: - world.browser.find_by_css(css_selector).click() - - except WebDriverException: - # Occassionally, MathJax or other JavaScript can cover up - # an element temporarily. - # If this happens, wait a second, then try again - world.wait(1) - world.browser.find_by_css(css_selector).click() + attempt = 0 + result = False + while attempt < attempts: + try: + world.css_find(css_selector)[index].click() + result = True + break + except WebDriverException: + # Occasionally, MathJax or other JavaScript can cover up + # an element temporarily. + # If this happens, wait a second, then try again + world.wait(1) + attempt += 1 + except: + attempt += 1 + return result @world.absorb @@ -158,3 +167,8 @@ def click_tools(): tools_css = 'li.nav-course-tools' if world.browser.is_element_present_by_css(tools_css): world.css_click(tools_css) + + +@world.absorb +def is_mac(): + return platform.mac_ver()[0] is not '' diff --git a/common/djangoapps/tests.py b/common/djangoapps/tests.py new file mode 100644 index 0000000000..8e78ee7f37 --- /dev/null +++ b/common/djangoapps/tests.py @@ -0,0 +1,49 @@ +''' +Created on Jun 6, 2013 + +@author: dmitchell +''' +from xmodule.modulestore.tests.factories import CourseFactory, ItemFactory +from student.tests.factories import AdminFactory +from xmodule.modulestore.tests.django_utils import ModuleStoreTestCase +import xmodule_modifiers +import datetime +from pytz import UTC +from xmodule.modulestore.tests import factories + +class TestXmoduleModfiers(ModuleStoreTestCase): + + # FIXME disabled b/c start date inheritance is not occuring and render_... in get_html is failing due + # to middleware.lookup['main'] not being defined + def _test_add_histogram(self): + instructor = AdminFactory.create() + self.client.login(username=instructor.username, password='test') + + course = CourseFactory.create(org='test', + number='313', display_name='histogram test') + section = ItemFactory.create( + parent_location=course.location, display_name='chapter hist', + template='i4x://edx/templates/chapter/Empty') + problem = ItemFactory.create( + parent_location=section.location, display_name='problem hist 1', + template='i4x://edx/templates/problem/Blank_Common_Problem') + problem.has_score = False # don't trip trying to retrieve db data + + late_problem = ItemFactory.create( + parent_location=section.location, display_name='problem hist 2', + template='i4x://edx/templates/problem/Blank_Common_Problem') + late_problem.lms.start = datetime.datetime.now(UTC) + datetime.timedelta(days=32) + late_problem.has_score = False + + + problem_module = factories.get_test_xmodule_for_descriptor(problem) + problem_module.get_html = xmodule_modifiers.add_histogram(lambda:'', problem_module, instructor) + + self.assertRegexpMatches( + problem_module.get_html(), r'.*Not yet.*') + + problem_module = factories.get_test_xmodule_for_descriptor(late_problem) + problem_module.get_html = xmodule_modifiers.add_histogram(lambda: '', problem_module, instructor) + + self.assertRegexpMatches( + problem_module.get_html(), r'.*Yes!.*') diff --git a/common/djangoapps/track/views.py b/common/djangoapps/track/views.py index ae3a1dcb3e..b2935a6a89 100644 --- a/common/djangoapps/track/views.py +++ b/common/djangoapps/track/views.py @@ -14,6 +14,7 @@ from mitxmako.shortcuts import render_to_response from django_future.csrf import ensure_csrf_cookie from track.models import TrackingLog +from pytz import UTC log = logging.getLogger("tracking") @@ -59,7 +60,7 @@ def user_track(request): "event": request.GET['event'], "agent": agent, "page": request.GET['page'], - "time": datetime.datetime.utcnow().isoformat(), + "time": datetime.datetime.now(UTC).isoformat(), "host": request.META['SERVER_NAME'], } log_event(event) @@ -85,11 +86,11 @@ def server_track(request, event_type, event, page=None): "event": event, "agent": agent, "page": page, - "time": datetime.datetime.utcnow().isoformat(), + "time": datetime.datetime.now(UTC).isoformat(), "host": request.META['SERVER_NAME'], } - if event_type.startswith("/event_logs") and request.user.is_staff: # don't log + if event_type.startswith("/event_logs") and request.user.is_staff: # don't log return log_event(event) diff --git a/common/djangoapps/util/testing.py b/common/djangoapps/util/testing.py new file mode 100644 index 0000000000..d33f1c8f8b --- /dev/null +++ b/common/djangoapps/util/testing.py @@ -0,0 +1,34 @@ +import sys + +from django.conf import settings +from django.core.urlresolvers import clear_url_caches + + +class UrlResetMixin(object): + """Mixin to reset urls.py before and after a test + + Django memoizes the function that reads the urls module (whatever module + urlconf names). The module itself is also stored by python in sys.modules. + To fully reload it, we need to reload the python module, and also clear django's + cache of the parsed urls. + + However, the order in which we do this doesn't matter, because neither one will + get reloaded until the next request + + Doing this is expensive, so it should only be added to tests that modify settings + that affect the contents of urls.py + """ + + def _reset_urls(self, urlconf=None): + if urlconf is None: + urlconf = settings.ROOT_URLCONF + + if urlconf in sys.modules: + reload(sys.modules[urlconf]) + clear_url_caches() + + def setUp(self): + """Reset django default urlconf before tests and after tests""" + super(UrlResetMixin, self).setUp() + self._reset_urls() + self.addCleanup(self._reset_urls) diff --git a/common/djangoapps/xmodule_modifiers.py b/common/djangoapps/xmodule_modifiers.py index 45691cd854..570b38c942 100644 --- a/common/djangoapps/xmodule_modifiers.py +++ b/common/djangoapps/xmodule_modifiers.py @@ -1,7 +1,6 @@ import re import json import logging -import time import static_replace from django.conf import settings @@ -9,6 +8,8 @@ from functools import wraps from mitxmako.shortcuts import render_to_string from xmodule.seq_module import SequenceModule from xmodule.vertical_module import VerticalModule +import datetime +from django.utils.timezone import UTC log = logging.getLogger("mitx.xmodule_modifiers") @@ -83,7 +84,7 @@ def grade_histogram(module_id): cursor.execute(q, [module_id]) grades = list(cursor.fetchall()) - grades.sort(key=lambda x: x[0]) # Add ORDER BY to sql query? + grades.sort(key=lambda x: x[0]) # Add ORDER BY to sql query? if len(grades) >= 1 and grades[0][0] is None: return [] return grades @@ -101,7 +102,7 @@ def add_histogram(get_html, module, user): @wraps(get_html) def _get_html(): - if type(module) in [SequenceModule, VerticalModule]: # TODO: make this more general, eg use an XModule attribute instead + if type(module) in [SequenceModule, VerticalModule]: # TODO: make this more general, eg use an XModule attribute instead return get_html() module_id = module.id @@ -132,7 +133,7 @@ def add_histogram(get_html, module, user): # useful to indicate to staff if problem has been released or not # TODO (ichuang): use _has_access_descriptor.can_load in lms.courseware.access, instead of now>mstart comparison here - now = time.gmtime() + now = datetime.datetime.now(UTC()) is_released = "unknown" mstart = module.descriptor.lms.start diff --git a/common/lib/calc/calc.py b/common/lib/calc/calc.py index 2ee82e2fb4..f0934a9ed5 100644 --- a/common/lib/calc/calc.py +++ b/common/lib/calc/calc.py @@ -1,34 +1,63 @@ +""" +Parser and evaluator for FormulaResponse and NumericalResponse + +Uses pyparsing to parse. Main function as of now is evaluator(). +""" + import copy -import logging import math import operator import re import numpy -import numbers import scipy.constants +import calcfunctions -from pyparsing import Word, alphas, nums, oneOf, Literal -from pyparsing import ZeroOrMore, OneOrMore, StringStart -from pyparsing import StringEnd, Optional, Forward -from pyparsing import CaselessLiteral, Group, StringEnd -from pyparsing import NoMatch, stringEnd, alphanums +# have numpy raise errors on functions outside its domain +# See http://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html +numpy.seterr(all='ignore') # Also: 'ignore', 'warn' (default), 'raise' -default_functions = {'sin': numpy.sin, +from pyparsing import (Word, nums, Literal, + ZeroOrMore, MatchFirst, + Optional, Forward, + CaselessLiteral, + stringEnd, Suppress, Combine) + +DEFAULT_FUNCTIONS = {'sin': numpy.sin, 'cos': numpy.cos, 'tan': numpy.tan, + 'sec': calcfunctions.sec, + 'csc': calcfunctions.csc, + 'cot': calcfunctions.cot, 'sqrt': numpy.sqrt, 'log10': numpy.log10, 'log2': numpy.log2, 'ln': numpy.log, + 'exp': numpy.exp, 'arccos': numpy.arccos, 'arcsin': numpy.arcsin, 'arctan': numpy.arctan, + 'arcsec': calcfunctions.arcsec, + 'arccsc': calcfunctions.arccsc, + 'arccot': calcfunctions.arccot, 'abs': numpy.abs, 'fact': math.factorial, - 'factorial': math.factorial + 'factorial': math.factorial, + 'sinh': numpy.sinh, + 'cosh': numpy.cosh, + 'tanh': numpy.tanh, + 'sech': calcfunctions.sech, + 'csch': calcfunctions.csch, + 'coth': calcfunctions.coth, + 'arcsinh': numpy.arcsinh, + 'arccosh': numpy.arccosh, + 'arctanh': numpy.arctanh, + 'arcsech': calcfunctions.arcsech, + 'arccsch': calcfunctions.arccsch, + 'arccoth': calcfunctions.arccoth } -default_variables = {'j': numpy.complex(0, 1), +DEFAULT_VARIABLES = {'i': numpy.complex(0, 1), + 'j': numpy.complex(0, 1), 'e': numpy.e, 'pi': numpy.pi, 'k': scipy.constants.k, @@ -37,65 +66,166 @@ default_variables = {'j': numpy.complex(0, 1), 'q': scipy.constants.e } -log = logging.getLogger("mitx.courseware.capa") +# We eliminated the following extreme suffixes: +# P (1e15), E (1e18), Z (1e21), Y (1e24), +# f (1e-15), a (1e-18), z (1e-21), y (1e-24) +# since they're rarely used, and potentially +# confusing. They may also conflict with variables if we ever allow e.g. +# 5R instead of 5*R +SUFFIXES = {'%': 0.01, 'k': 1e3, 'M': 1e6, 'G': 1e9, 'T': 1e12, + 'c': 1e-2, 'm': 1e-3, 'u': 1e-6, 'n': 1e-9, 'p': 1e-12} class UndefinedVariable(Exception): - def raiseself(self): - ''' Helper so we can use inside of a lambda ''' - raise self - - -general_whitespace = re.compile('[^\w]+') + """ + Used to indicate the student input of a variable, which was unused by the + instructor. + """ + pass def check_variables(string, variables): - '''Confirm the only variables in string are defined. + """ + Confirm the only variables in string are defined. - Pyparsing uses a left-to-right parser, which makes the more + Otherwise, raise an UndefinedVariable containing all bad variables. + + Pyparsing uses a left-to-right parser, which makes a more elegant approach pretty hopeless. - - achar = reduce(lambda a,b:a|b ,map(Literal,alphas)) # Any alphabetic character - undefined_variable = achar + Word(alphanums) - undefined_variable.setParseAction(lambda x:UndefinedVariable("".join(x)).raiseself()) - varnames = varnames | undefined_variable - ''' - possible_variables = re.split(general_whitespace, string) # List of all alnums in string - bad_variables = list() - for v in possible_variables: - if len(v) == 0: + """ + general_whitespace = re.compile('[^\\w]+') + # List of all alnums in string + possible_variables = re.split(general_whitespace, string) + bad_variables = [] + for var in possible_variables: + if len(var) == 0: continue - if v[0] <= '9' and '0' <= 'v': # Skip things that begin with numbers + if var[0].isdigit(): # Skip things that begin with numbers continue - if v not in variables: - bad_variables.append(v) + if var not in variables: + bad_variables.append(var) if len(bad_variables) > 0: raise UndefinedVariable(' '.join(bad_variables)) +def lower_dict(input_dict): + """ + takes each key in the dict and makes it lowercase, still mapping to the + same value. + + keep in mind that it is possible (but not useful?) to define different + variables that have the same lowercase representation. It would be hard to + tell which is used in the final dict and which isn't. + """ + return {k.lower(): v for k, v in input_dict.iteritems()} + + +# The following few functions define parse actions, which are run on lists of +# results from each parse component. They convert the strings and (previously +# calculated) numbers into the number that component represents. + +def super_float(text): + """ + Like float, but with si extensions. 1k goes to 1000 + """ + if text[-1] in SUFFIXES: + return float(text[:-1]) * SUFFIXES[text[-1]] + else: + return float(text) + + +def number_parse_action(parse_result): + """ + Create a float out of its string parts + + e.g. [ '7', '.', '13' ] -> [ 7.13 ] + Calls super_float above + """ + return super_float("".join(parse_result)) + + +def exp_parse_action(parse_result): + """ + Take a list of numbers and exponentiate them, right to left + + e.g. [ 3, 2, 3 ] (which is 3^2^3 = 3^(2^3)) -> 6561 + """ + # pyparsing.ParseResults doesn't play well with reverse() + parse_result = reversed(parse_result) + # the result of an exponentiation is called a power + power = reduce(lambda a, b: b ** a, parse_result) + return power + + +def parallel(parse_result): + """ + Compute numbers according to the parallel resistors operator + + BTW it is commutative. Its formula is given by + out = 1 / (1/in1 + 1/in2 + ...) + e.g. [ 1, 2 ] => 2/3 + + Return NaN if there is a zero among the inputs + """ + # convert from pyparsing.ParseResults, which doesn't support '0 in parse_result' + parse_result = parse_result.asList() + if len(parse_result) == 1: + return parse_result[0] + if 0 in parse_result: + return float('nan') + reciprocals = [1. / e for e in parse_result] + return 1. / sum(reciprocals) + + +def sum_parse_action(parse_result): + """ + Add the inputs + + [ 1, '+', 2, '-', 3 ] -> 0 + + Allow a leading + or - + """ + total = 0.0 + current_op = operator.add + for token in parse_result: + if token is '+': + current_op = operator.add + elif token is '-': + current_op = operator.sub + else: + total = current_op(total, token) + return total + + +def prod_parse_action(parse_result): + """ + Multiply the inputs + + [ 1, '*', 2, '/', 3 ] => 0.66 + """ + prod = 1.0 + current_op = operator.mul + for token in parse_result: + if token is '*': + current_op = operator.mul + elif token is '/': + current_op = operator.truediv + else: + prod = current_op(prod, token) + return prod + + def evaluator(variables, functions, string, cs=False): - ''' + """ Evaluate an expression. Variables are passed as a dictionary from string to value. Unary functions are passed as a dictionary from string to function. Variables must be floats. cs: Case sensitive - TODO: Fix it so we can pass integers and complex numbers in variables dict - ''' - # log.debug("variables: {0}".format(variables)) - # log.debug("functions: {0}".format(functions)) - # log.debug("string: {0}".format(string)) - - def lower_dict(d): - return dict([(k.lower(), d[k]) for k in d]) - - all_variables = copy.copy(default_variables) - all_functions = copy.copy(default_functions) - - if not cs: - all_variables = lower_dict(all_variables) - all_functions = lower_dict(all_functions) + """ + all_variables = copy.copy(DEFAULT_VARIABLES) + all_functions = copy.copy(DEFAULT_FUNCTIONS) all_variables.update(variables) all_functions.update(functions) @@ -113,122 +243,59 @@ def evaluator(variables, functions, string, cs=False): if string.strip() == "": return float('nan') - ops = {"^": operator.pow, - "*": operator.mul, - "/": operator.truediv, - "+": operator.add, - "-": operator.sub, - } - # We eliminated extreme ones, since they're rarely used, and potentially - # confusing. They may also conflict with variables if we ever allow e.g. - # 5R instead of 5*R - suffixes = {'%': 0.01, 'k': 1e3, 'M': 1e6, 'G': 1e9, - 'T': 1e12, # 'P':1e15,'E':1e18,'Z':1e21,'Y':1e24, - 'c': 1e-2, 'm': 1e-3, 'u': 1e-6, - 'n': 1e-9, 'p': 1e-12} # ,'f':1e-15,'a':1e-18,'z':1e-21,'y':1e-24} - - def super_float(text): - ''' Like float, but with si extensions. 1k goes to 1000''' - if text[-1] in suffixes: - return float(text[:-1]) * suffixes[text[-1]] - else: - return float(text) - - def number_parse_action(x): # [ '7' ] -> [ 7 ] - return [super_float("".join(x))] - - def exp_parse_action(x): # [ 2 ^ 3 ^ 2 ] -> 512 - x = [e for e in x if isinstance(e, numbers.Number)] # Ignore ^ - x.reverse() - x = reduce(lambda a, b: b ** a, x) - return x - - def parallel(x): # Parallel resistors [ 1 2 ] => 2/3 - # convert from pyparsing.ParseResults, which doesn't support '0 in x' - x = list(x) - if len(x) == 1: - return x[0] - if 0 in x: - return float('nan') - x = [1. / e for e in x if isinstance(e, numbers.Number)] # Ignore || - return 1. / sum(x) - - def sum_parse_action(x): # [ 1 + 2 - 3 ] -> 0 - total = 0.0 - op = ops['+'] - for e in x: - if e in set('+-'): - op = ops[e] - else: - total = op(total, e) - return total - - def prod_parse_action(x): # [ 1 * 2 / 3 ] => 0.66 - prod = 1.0 - op = ops['*'] - for e in x: - if e in set('*/'): - op = ops[e] - else: - prod = op(prod, e) - return prod - - def func_parse_action(x): - return [all_functions[x[0]](x[1])] - # SI suffixes and percent - number_suffix = reduce(lambda a, b: a | b, map(Literal, suffixes.keys()), NoMatch()) - (dot, minus, plus, times, div, lpar, rpar, exp) = map(Literal, ".-+*/()^") + number_suffix = MatchFirst([Literal(k) for k in SUFFIXES.keys()]) + plus_minus = Literal('+') | Literal('-') + times_div = Literal('*') | Literal('/') number_part = Word(nums) # 0.33 or 7 or .34 or 16. inner_number = (number_part + Optional("." + Optional(number_part))) | ("." + number_part) + # by default pyparsing allows spaces between tokens--Combine prevents that + inner_number = Combine(inner_number) # 0.33k or -17 - number = (Optional(minus | plus) + inner_number - + Optional(CaselessLiteral("E") + Optional((plus | minus)) + number_part) + number = (inner_number + + Optional(CaselessLiteral("E") + Optional(plus_minus) + number_part) + Optional(number_suffix)) - number = number.setParseAction(number_parse_action) # Convert to number + number.setParseAction(number_parse_action) # Convert to number # Predefine recursive variables expr = Forward() - factor = Forward() - def sreduce(f, l): - ''' Same as reduce, but handle len 1 and len 0 lists sensibly ''' - if len(l) == 0: - return NoMatch() - if len(l) == 1: - return l[0] - return reduce(f, l) + # Handle variables passed in. + # E.g. if we have {'R':0.5}, we make the substitution. + # We sort the list so that var names (like "e2") match before + # mathematical constants (like "e"). This is kind of a hack. + all_variables_keys = sorted(all_variables.keys(), key=len, reverse=True) + varnames = MatchFirst([CasedLiteral(k) for k in all_variables_keys]) + varnames.setParseAction( + lambda x: [all_variables[k] for k in x] + ) - # Handle variables passed in. E.g. if we have {'R':0.5}, we make the substitution. - # Special case for no variables because of how we understand PyParsing is put together - if len(all_variables) > 0: - # We sort the list so that var names (like "e2") match before - # mathematical constants (like "e"). This is kind of a hack. - all_variables_keys = sorted(all_variables.keys(), key=len, reverse=True) - varnames = sreduce(lambda x, y: x | y, map(lambda x: CasedLiteral(x), all_variables_keys)) - varnames.setParseAction(lambda x: map(lambda y: all_variables[y], x)) - else: - varnames = NoMatch() + # if all_variables were empty, then pyparsing wants + # varnames = NoMatch() + # this is not the case, as all_variables contains the defaults # Same thing for functions. - if len(all_functions) > 0: - funcnames = sreduce(lambda x, y: x | y, - map(lambda x: CasedLiteral(x), all_functions.keys())) - function = funcnames + lpar.suppress() + expr + rpar.suppress() - function.setParseAction(func_parse_action) - else: - function = NoMatch() + all_functions_keys = sorted(all_functions.keys(), key=len, reverse=True) + funcnames = MatchFirst([CasedLiteral(k) for k in all_functions_keys]) + function = funcnames + Suppress("(") + expr + Suppress(")") + function.setParseAction( + lambda x: [all_functions[x[0]](x[1])] + ) - atom = number | function | varnames | lpar + expr + rpar - factor << (atom + ZeroOrMore(exp + atom)).setParseAction(exp_parse_action) # 7^6 - paritem = factor + ZeroOrMore(Literal('||') + factor) # 5k || 4k - paritem = paritem.setParseAction(parallel) - term = paritem + ZeroOrMore((times | div) + paritem) # 7 * 5 / 4 - 3 - term = term.setParseAction(prod_parse_action) - expr << Optional((plus | minus)) + term + ZeroOrMore((plus | minus) + term) # -5 + 4 - 3 - expr = expr.setParseAction(sum_parse_action) + atom = number | function | varnames | Suppress("(") + expr + Suppress(")") + + # Do the following in the correct order to preserve order of operation + pow_term = atom + ZeroOrMore(Suppress("^") + atom) + pow_term.setParseAction(exp_parse_action) # 7^6 + par_term = pow_term + ZeroOrMore(Suppress('||') + pow_term) # 5k || 4k + par_term.setParseAction(parallel) + prod_term = par_term + ZeroOrMore(times_div + par_term) # 7 * 5 / 4 - 3 + prod_term.setParseAction(prod_parse_action) + sum_term = Optional(plus_minus) + prod_term + ZeroOrMore(plus_minus + prod_term) # -5 + 4 - 3 + sum_term.setParseAction(sum_parse_action) + expr << sum_term # finish the recursion return (expr + stringEnd).parseString(string)[0] diff --git a/common/lib/calc/calcfunctions.py b/common/lib/calc/calcfunctions.py new file mode 100644 index 0000000000..d0ac4f7a3d --- /dev/null +++ b/common/lib/calc/calcfunctions.py @@ -0,0 +1,99 @@ +""" +Provide the mathematical functions that numpy doesn't. + +Specifically, the secant/cosecant/cotangents and their inverses and +hyperbolic counterparts +""" +import numpy + + +# Normal Trig +def sec(arg): + """ + Secant + """ + return 1 / numpy.cos(arg) + + +def csc(arg): + """ + Cosecant + """ + return 1 / numpy.sin(arg) + + +def cot(arg): + """ + Cotangent + """ + return 1 / numpy.tan(arg) + + +# Inverse Trig +# http://en.wikipedia.org/wiki/Inverse_trigonometric_functions#Relationships_among_the_inverse_trigonometric_functions +def arcsec(val): + """ + Inverse secant + """ + return numpy.arccos(1. / val) + + +def arccsc(val): + """ + Inverse cosecant + """ + return numpy.arcsin(1. / val) + + +def arccot(val): + """ + Inverse cotangent + """ + if numpy.real(val) < 0: + return -numpy.pi / 2 - numpy.arctan(val) + else: + return numpy.pi / 2 - numpy.arctan(val) + + +# Hyperbolic Trig +def sech(arg): + """ + Hyperbolic secant + """ + return 1 / numpy.cosh(arg) + + +def csch(arg): + """ + Hyperbolic cosecant + """ + return 1 / numpy.sinh(arg) + + +def coth(arg): + """ + Hyperbolic cotangent + """ + return 1 / numpy.tanh(arg) + + +# And their inverses +def arcsech(val): + """ + Inverse hyperbolic secant + """ + return numpy.arccosh(1. / val) + + +def arccsch(val): + """ + Inverse hyperbolic cosecant + """ + return numpy.arcsinh(1. / val) + + +def arccoth(val): + """ + Inverse hyperbolic cotangent + """ + return numpy.arctanh(1. / val) diff --git a/common/lib/calc/tests/test_calc.py b/common/lib/calc/tests/test_calc.py index cfa1b7525d..13cd9e9471 100644 --- a/common/lib/calc/tests/test_calc.py +++ b/common/lib/calc/tests/test_calc.py @@ -194,6 +194,105 @@ class EvaluatorTest(unittest.TestCase): arctan_angles = arcsin_angles self.assert_function_values('arctan', arctan_inputs, arctan_angles) + def test_reciprocal_trig_functions(self): + """ + Test the reciprocal trig functions provided in calc.py + + which are: sec, csc, cot, arcsec, arccsc, arccot + """ + angles = ['-pi/4', 'pi/6', 'pi/5', '5*pi/4', '9*pi/4', '1 + j'] + sec_values = [1.414, 1.155, 1.236, -1.414, 1.414, 0.498 + 0.591j] + csc_values = [-1.414, 2, 1.701, -1.414, 1.414, 0.622 - 0.304j] + cot_values = [-1, 1.732, 1.376, 1, 1, 0.218 - 0.868j] + + self.assert_function_values('sec', angles, sec_values) + self.assert_function_values('csc', angles, csc_values) + self.assert_function_values('cot', angles, cot_values) + + arcsec_inputs = ['1.1547', '1.2361', '2', '-2', '-1.4142', '0.4983+0.5911*j'] + arcsec_angles = [0.524, 0.628, 1.047, 2.094, 2.356, 1 + 1j] + self.assert_function_values('arcsec', arcsec_inputs, arcsec_angles) + + arccsc_inputs = ['-1.1547', '-1.4142', '2', '1.7013', '1.1547', '0.6215-0.3039*j'] + arccsc_angles = [-1.047, -0.785, 0.524, 0.628, 1.047, 1 + 1j] + self.assert_function_values('arccsc', arccsc_inputs, arccsc_angles) + + # Has the same range as arccsc + arccot_inputs = ['-0.5774', '-1', '1.7321', '1.3764', '0.5774', '(0.2176-0.868*j)'] + arccot_angles = arccsc_angles + self.assert_function_values('arccot', arccot_inputs, arccot_angles) + + def test_hyperbolic_functions(self): + """ + Test the hyperbolic functions + + which are: sinh, cosh, tanh, sech, csch, coth + """ + inputs = ['0', '0.5', '1', '2', '1+j'] + neg_inputs = ['0', '-0.5', '-1', '-2', '-1-j'] + negate = lambda x: [-k for k in x] + + # sinh is odd + sinh_vals = [0, 0.521, 1.175, 3.627, 0.635 + 1.298j] + self.assert_function_values('sinh', inputs, sinh_vals) + self.assert_function_values('sinh', neg_inputs, negate(sinh_vals)) + + # cosh is even - do not negate + cosh_vals = [1, 1.128, 1.543, 3.762, 0.834 + 0.989j] + self.assert_function_values('cosh', inputs, cosh_vals) + self.assert_function_values('cosh', neg_inputs, cosh_vals) + + # tanh is odd + tanh_vals = [0, 0.462, 0.762, 0.964, 1.084 + 0.272j] + self.assert_function_values('tanh', inputs, tanh_vals) + self.assert_function_values('tanh', neg_inputs, negate(tanh_vals)) + + # sech is even - do not negate + sech_vals = [1, 0.887, 0.648, 0.266, 0.498 - 0.591j] + self.assert_function_values('sech', inputs, sech_vals) + self.assert_function_values('sech', neg_inputs, sech_vals) + + # the following functions do not have 0 in their domain + inputs = inputs[1:] + neg_inputs = neg_inputs[1:] + + # csch is odd + csch_vals = [1.919, 0.851, 0.276, 0.304 - 0.622j] + self.assert_function_values('csch', inputs, csch_vals) + self.assert_function_values('csch', neg_inputs, negate(csch_vals)) + + # coth is odd + coth_vals = [2.164, 1.313, 1.037, 0.868 - 0.218j] + self.assert_function_values('coth', inputs, coth_vals) + self.assert_function_values('coth', neg_inputs, negate(coth_vals)) + + def test_hyperbolic_inverses(self): + """ + Test the inverse hyperbolic functions + + which are of the form arc[X]h + """ + results = [0, 0.5, 1, 2, 1 + 1j] + + sinh_vals = ['0', '0.5211', '1.1752', '3.6269', '0.635+1.2985*j'] + self.assert_function_values('arcsinh', sinh_vals, results) + + cosh_vals = ['1', '1.1276', '1.5431', '3.7622', '0.8337+0.9889*j'] + self.assert_function_values('arccosh', cosh_vals, results) + + tanh_vals = ['0', '0.4621', '0.7616', '0.964', '1.0839+0.2718*j'] + self.assert_function_values('arctanh', tanh_vals, results) + + sech_vals = ['1.0', '0.8868', '0.6481', '0.2658', '0.4983-0.5911*j'] + self.assert_function_values('arcsech', sech_vals, results) + + results = results[1:] + csch_vals = ['1.919', '0.8509', '0.2757', '0.3039-0.6215*j'] + self.assert_function_values('arccsch', csch_vals, results) + + coth_vals = ['2.164', '1.313', '1.0373', '0.868-0.2176*j'] + self.assert_function_values('arccoth', coth_vals, results) + def test_other_functions(self): """ Test the non-trig functions provided in calc.py diff --git a/common/lib/capa/capa/capa_problem.py b/common/lib/capa/capa/capa_problem.py index 150b3b3c9b..7dcd7b925e 100644 --- a/common/lib/capa/capa/capa_problem.py +++ b/common/lib/capa/capa/capa_problem.py @@ -470,6 +470,7 @@ class LoncapaProblem(object): python_path=python_path, cache=self.system.cache, slug=self.problem_id, + unsafely=self.system.can_execute_unsafe_code(), ) except Exception as err: log.exception("Error while execing script code: " + all_code) diff --git a/common/lib/capa/capa/inputtypes.py b/common/lib/capa/capa/inputtypes.py index 65280d6d29..446b832dd7 100644 --- a/common/lib/capa/capa/inputtypes.py +++ b/common/lib/capa/capa/inputtypes.py @@ -144,11 +144,11 @@ class InputTypeBase(object): self.tag = xml.tag self.system = system - ## NOTE: ID should only come from one place. If it comes from multiple, - ## we use state first, XML second (in case the xml changed, but we have - ## existing state with an old id). Since we don't make this guarantee, - ## we can swap this around in the future if there's a more logical - ## order. + # NOTE: ID should only come from one place. If it comes from multiple, + # we use state first, XML second (in case the xml changed, but we have + # existing state with an old id). Since we don't make this guarantee, + # we can swap this around in the future if there's a more logical + # order. self.input_id = state.get('id', xml.get('id')) if self.input_id is None: @@ -769,7 +769,7 @@ class MatlabInput(CodeInput): # construct xqueue headers qinterface = self.system.xqueue['interface'] - qtime = datetime.strftime(datetime.utcnow(), xqueue_interface.dateformat) + qtime = datetime.utcnow().strftime(xqueue_interface.dateformat) callback_url = self.system.xqueue['construct_callback']('ungraded_response') anonymous_student_id = self.system.anonymous_student_id queuekey = xqueue_interface.make_hashkey(str(self.system.seed) + qtime + diff --git a/common/lib/capa/capa/responsetypes.py b/common/lib/capa/capa/responsetypes.py index 0fa50079de..80227490da 100644 --- a/common/lib/capa/capa/responsetypes.py +++ b/common/lib/capa/capa/responsetypes.py @@ -288,7 +288,14 @@ class LoncapaResponse(object): } try: - safe_exec.safe_exec(code, globals_dict, python_path=self.context['python_path'], slug=self.id) + safe_exec.safe_exec( + code, + globals_dict, + python_path=self.context['python_path'], + slug=self.id, + random_seed=self.context['seed'], + unsafely=self.system.can_execute_unsafe_code(), + ) except Exception as err: msg = 'Error %s in evaluating hint function %s' % (err, hintfn) msg += "\nSee XML source line %s" % getattr( @@ -973,7 +980,14 @@ class CustomResponse(LoncapaResponse): 'ans': ans, } globals_dict.update(kwargs) - safe_exec.safe_exec(code, globals_dict, python_path=self.context['python_path'], slug=self.id) + safe_exec.safe_exec( + code, + globals_dict, + python_path=self.context['python_path'], + slug=self.id, + random_seed=self.context['seed'], + unsafely=self.system.can_execute_unsafe_code(), + ) return globals_dict['cfn_return'] return check_function @@ -1090,7 +1104,14 @@ class CustomResponse(LoncapaResponse): # exec the check function if isinstance(self.code, basestring): try: - safe_exec.safe_exec(self.code, self.context, cache=self.system.cache, slug=self.id) + safe_exec.safe_exec( + self.code, + self.context, + cache=self.system.cache, + slug=self.id, + random_seed=self.context['seed'], + unsafely=self.system.can_execute_unsafe_code(), + ) except Exception as err: self._handle_exec_exception(err) @@ -1717,6 +1738,7 @@ class FormulaResponse(LoncapaResponse): student_variables = dict() # ranges give numerical ranges for testing for var in ranges: + # TODO: allow specified ranges (i.e. integers and complex numbers) for random variables value = random.uniform(*ranges[var]) instructor_variables[str(var)] = value student_variables[str(var)] = value @@ -1814,7 +1836,14 @@ class SchematicResponse(LoncapaResponse): ] self.context.update({'submission': submission}) try: - safe_exec.safe_exec(self.code, self.context, cache=self.system.cache, slug=self.id) + safe_exec.safe_exec( + self.code, + self.context, + cache=self.system.cache, + slug=self.id, + random_seed=self.context['seed'], + unsafely=self.system.can_execute_unsafe_code(), + ) except Exception as err: msg = 'Error %s in evaluating SchematicResponse' % err raise ResponseError(msg) diff --git a/common/lib/capa/capa/safe_exec/safe_exec.py b/common/lib/capa/capa/safe_exec/safe_exec.py index 67e93be46f..3ab8f0bf9e 100644 --- a/common/lib/capa/capa/safe_exec/safe_exec.py +++ b/common/lib/capa/capa/safe_exec/safe_exec.py @@ -1,6 +1,7 @@ """Capa's specialized use of codejail.safe_exec.""" from codejail.safe_exec import safe_exec as codejail_safe_exec +from codejail.safe_exec import not_safe_exec as codejail_not_safe_exec from codejail.safe_exec import json_safe, SafeExecException from . import lazymod from statsd import statsd @@ -71,7 +72,7 @@ def update_hash(hasher, obj): @statsd.timed('capa.safe_exec.time') -def safe_exec(code, globals_dict, random_seed=None, python_path=None, cache=None, slug=None): +def safe_exec(code, globals_dict, random_seed=None, python_path=None, cache=None, slug=None, unsafely=False): """ Execute python code safely. @@ -90,6 +91,8 @@ def safe_exec(code, globals_dict, random_seed=None, python_path=None, cache=None `slug` is an arbitrary string, a description that's meaningful to the caller, that will be used in log messages. + If `unsafely` is true, then the code will actually be executed without sandboxing. + """ # Check the cache for a previous result. if cache: @@ -111,9 +114,15 @@ def safe_exec(code, globals_dict, random_seed=None, python_path=None, cache=None # Create the complete code we'll run. code_prolog = CODE_PROLOG % random_seed + # Decide which code executor to use. + if unsafely: + exec_fn = codejail_not_safe_exec + else: + exec_fn = codejail_safe_exec + # Run the code! Results are side effects in globals_dict. try: - codejail_safe_exec( + exec_fn( code_prolog + LAZY_IMPORTS + code, globals_dict, python_path=python_path, slug=slug, ) diff --git a/common/lib/capa/capa/safe_exec/tests/test_safe_exec.py b/common/lib/capa/capa/safe_exec/tests/test_safe_exec.py index 4592af8305..f8a8a32297 100644 --- a/common/lib/capa/capa/safe_exec/tests/test_safe_exec.py +++ b/common/lib/capa/capa/safe_exec/tests/test_safe_exec.py @@ -1,13 +1,17 @@ """Test safe_exec.py""" import hashlib +import os import os.path import random import textwrap import unittest +from nose.plugins.skip import SkipTest + from capa.safe_exec import safe_exec, update_hash from codejail.safe_exec import SafeExecException +from codejail.jail_code import is_configured class TestSafeExec(unittest.TestCase): @@ -68,6 +72,24 @@ class TestSafeExec(unittest.TestCase): self.assertIn("ZeroDivisionError", cm.exception.message) +class TestSafeOrNot(unittest.TestCase): + def test_cant_do_something_forbidden(self): + # Can't test for forbiddenness if CodeJail isn't configured for python. + if not is_configured("python"): + raise SkipTest + + g = {} + with self.assertRaises(SafeExecException) as cm: + safe_exec("import os; files = os.listdir('/')", g) + self.assertIn("OSError", cm.exception.message) + self.assertIn("Permission denied", cm.exception.message) + + def test_can_do_something_forbidden_if_run_unsafely(self): + g = {} + safe_exec("import os; files = os.listdir('/')", g, unsafely=True) + self.assertEqual(g['files'], os.listdir('/')) + + class DictCache(object): """A cache implementation over a simple dict, for testing.""" diff --git a/common/lib/capa/capa/templates/annotationinput.html b/common/lib/capa/capa/templates/annotationinput.html index e0172bb13b..145a7c2cad 100644 --- a/common/lib/capa/capa/templates/annotationinput.html +++ b/common/lib/capa/capa/templates/annotationinput.html @@ -14,7 +14,7 @@
${comment}
${comment_prompt}
- +
${tag_prompt}
    @@ -22,11 +22,11 @@
  • % if has_options_value: % if all([c == 'correct' for c in option['choice'], status]): - + Status: Correct % elif all([c == 'partially-correct' for c in option['choice'], status]): - + Status: Partially Correct % elif all([c == 'incorrect' for c in option['choice'], status]): - + Status: Incorrect % endif % endif @@ -53,11 +53,11 @@ % endif % if status == 'unsubmitted': - + Status: Unanswered % elif status == 'incomplete': - + Status: Incorrect % elif status == 'incorrect' and not has_options_value: - + Status: Incorrect % endif

    diff --git a/common/lib/capa/capa/templates/chemicalequationinput.html b/common/lib/capa/capa/templates/chemicalequationinput.html index 17c84114e5..34709c3e5e 100644 --- a/common/lib/capa/capa/templates/chemicalequationinput.html +++ b/common/lib/capa/capa/templates/chemicalequationinput.html @@ -11,13 +11,13 @@
    % endif - -

    +

    % if status == 'unsubmitted': unanswered % elif status == 'correct': diff --git a/common/lib/capa/capa/templates/choicegroup.html b/common/lib/capa/capa/templates/choicegroup.html index c9cc3fd28d..17f7efcec4 100644 --- a/common/lib/capa/capa/templates/choicegroup.html +++ b/common/lib/capa/capa/templates/choicegroup.html @@ -3,12 +3,12 @@ % if input_type == 'checkbox' or not value: % if status == 'unsubmitted' or show_correctness == 'never': - % elif status == 'correct': - + % elif status == 'correct': + Status: correct % elif status == 'incorrect': - + Status: incorrect % elif status == 'incomplete': - + Status: incomplete % endif % endif

    @@ -18,7 +18,7 @@ % for choice_id, choice_description in choices: + /> ${choice_description} + + % if input_type == 'radio' and ( (isinstance(value, basestring) and (choice_id == value)) or (not isinstance(value, basestring) and choice_id in value) ): + <% + if status == 'correct': + correctness = 'correct' + elif status == 'incorrect': + correctness = 'incorrect' + else: + correctness = None + %> + % if correctness and not show_correctness=='never': + Status: ${correctness} + % endif + % endif + % endfor diff --git a/common/lib/capa/capa/templates/codeinput.html b/common/lib/capa/capa/templates/codeinput.html index eb8cad0d70..08ad4ff062 100644 --- a/common/lib/capa/capa/templates/codeinput.html +++ b/common/lib/capa/capa/templates/codeinput.html @@ -1,5 +1,5 @@
    -
Name Date Added URL
+ +