ORA documentation updates

Updated main ORA documentation based on several bugs; added new
student-facing ORA docs

docs/course_authors/source/index.rst

@@ -30,6 +30,7 @@ Contents
    view_course_content
    export_import_course
    checking_student_progress
+   ora_students
    change_log
 
 

docs/course_authors/source/open_response_assessment.rst

@@ -1,16 +1,14 @@
 .. _Open Response Assessment Problems:
 
-Open Response Assessments
+Open Response Assessment Problems
 ---------------------------------
 
 Introduction to Open Response Assessments
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. note:: 
-
-   Open response assessments are still in beta. We recommend that
-   you test them thoroughly in a practice course and only add them to
-   courses that are **not** already running. Contact your edX Program Manager for more information.
+.. note:: Open response assessments are still in beta. We recommend that
+          you test them thoroughly in a practice course and only add them to
+          courses that are **not** already running.
 
 Open response assessments allow instructors to assess student learning
 through questions that may not have definite answers. Tens of thousands
@@ -140,25 +138,27 @@ Creating an open response assessment is a multi-step process.
 
 Each of these steps is described in detail below.
 
-1. Create the Component
-~~~~~~~~~~~~~~~~~~~~~~~
+Step 1. Create the Component
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 #. Add the advanced component for open response assessments. To do this,
    add the "peergrading","combinedopenended" key value to the **Advanced
    Settings** page. (For more information, see the instructions in
-   Specialized Problems.)
+   :ref:`Specialized Problems`.)
 #. In Studio, open the unit where you want to create the ORA.
 #. Under **Add New Component**, click **Advanced**, and then click
    **Open Response Assessment**.
 #. In the problem component that appears, click **Edit**, and then click
    **OK** in the dialog box that appears.
 #. The component editor opens. The component editor contains a sample
-   question ("prompt"), rubric, assessment type specification, and
+   question ("prompt"), rubric, and the code for the assessment type and
    scoring. You'll replace this sample content with the content for your
    problem. 
  
-2. Add the Question
-~~~~~~~~~~~~~~~~~~~
+.. image:: Images/ORAComponentEditor.gif
+  
+Step 2. Add the Question
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 -  In the component editor, locate the [prompt] tags.
 
@@ -171,8 +171,12 @@ these guidelines to avoid common formatting mistakes.
 -  Leave the **[prompt]** tags in place.
 -  Enclose all text in HTML tags.
 
-3. Add the Rubric
-~~~~~~~~~~~~~~~~~
+Step 3. Add the Rubric
+~~~~~~~~~~~~~~~~~~~~~~
+
+**Note** *After you release your problem to students in your live course, make sure
+you don't make any changes to the rubric that affect scoring, such as adding or removing an option
+in a category. Changing the rubric can cause errors in live courses.*
 
 #. In the component editor, locate the [rubric] tags. (The sample rubric
    is long, so you'll have to scroll down to locate the second tag.)
@@ -205,13 +209,16 @@ these guidelines to avoid common formatting mistakes.
 
     [rubric]
 
-4. Set the Assessment Type and Scoring
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Step 4. Set the Assessment Type and Scoring
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To set the assessment type and scoring for your open response
-assessment, you'll enter code that specifies the type and order of
-assessments to use along with the scoring thresholds for each
-assessment. The code uses the following format.
+**Note** *After you release your problem to students in your live course, make sure
+you don't make any changes to the code for the assessment type and scoring. Changing 
+this code can cause errors in live courses.*
+
+To set the assessment type and scoring for your open response assessment, you'll 
+enter code that specifies the type and order of assessments to use along with 
+the scoring thresholds for each assessment. The code uses the following format.
 
 ::
 
@@ -257,8 +264,17 @@ Set the Type and Scoring
 
 #. Replace the sample code with the code for your problem.
 
-5. Set the Problem Name
-~~~~~~~~~~~~~~~~~~~~~~~
+Step 5. Set the Problem Name
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Note** *After you release your problem to students in your live course, make sure
+you don't change the name of the problem. Changing the display name when the problem
+is live can cause a loss of student data.*
+
+*You can change the display name of a problem while you're still testing the problem.
+However, note that all the test responses and scores associated with the problem 
+will be lost when you change the name. To update the problem name on the 
+instructor dashboard, submit a new test response to the problem.*
 
 The name of the problem appears as a heading above the problem in the
 courseware. It also appears in the list of problems on the **Staff
@@ -273,8 +289,8 @@ To change the name:
 #. In the **Display Name** field, replace **Open Response Assessment**
    with the name of your problem.
 
-6. Set Other Options
-~~~~~~~~~~~~~~~~~~~~
+Step 6. Set Other Options
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you want to change the problem settings, which include the number of
 responses a student has to peer grade and whether students can upload
@@ -304,7 +320,7 @@ Open response assessments include the following settings.
 |                                             | responses that are very short or that have many spelling or        |
 |                                             | grammatical errors) to be peer graded. For example, you may        |
 |                                             | disable the quality filter if you want students to include URLs to |
-|                                             | external content—otherwise Studio sees a URL, which may contain a  |
+|                                             | external content - otherwise Studio sees a URL, which may contain a|
 |                                             | long string of seemingly random characters, as a misspelled word.  |
 |                                             | When the quality filter is enabled (when this value is set to      |
 |                                             | False), Studio does not allow poor-quality submissions to be peer  |
@@ -358,6 +374,10 @@ Open response assessments include the following settings.
 +---------------------------------------------+--------------------------------------------------------------------+
 | **Problem Weight**                          | This setting specifies the number of points the problem is worth.  |
 |                                             | By default, each problem is worth one point.                       |
+|                                             |                                                                    |
+|                                             | **Note** *Every problem must have a problem weight of at least     |
+|                                             | one point. Problems that have a problem weight of zero points      |
+|                                             | don't appear on the instructor dashboard.                          |
 +---------------------------------------------+--------------------------------------------------------------------+
 | **Required Peer Grading**                   | This setting specifies the number of responses that each student   |
 |                                             | who submits a response has to grade before the student receives a  |
@@ -371,8 +391,8 @@ Open response assessments include the following settings.
 |                                             | previously graded responses.)                                      |
 +---------------------------------------------+--------------------------------------------------------------------+
 
-7. Save the Problem
-~~~~~~~~~~~~~~~~~~~
+Step 7. Save the Problem
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 -  After you have created the prompt and the rubric, set the assessment
    type and scoring, changed the name of the problem, and specified any
@@ -383,8 +403,8 @@ Open response assessments include the following settings.
 
 .. image:: Images/ORA_Component.gif
 
-8. Add the Peer Grading Interface (for peer assessments only)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Step 8. Add the Peer Grading Interface (for peer assessments only)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 You can add just one peer grading interface for the whole course, or you
 can add a separate peer grading interface for each individual problem.
@@ -457,8 +477,8 @@ week).
 
 #. Click **Save** to close the component editor.
 
-9. Test the Problem
-~~~~~~~~~~~~~~~~~~~
+Step 9. Test the Problem
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 Test your problem by adding and grading a response.
 
@@ -495,6 +515,16 @@ that you use as an instructor.
    this, you can't make changes to your course without signing out and
    signing back in as an instructor.
 
+When you test your problem, you may want to submit test responses that contain little
+text, random characters, or other content that doesn't resemble the responses that you
+expect from your students. Open response assessments include a quality filter that
+prevents instructors and other students from seeing these "low-quality" responses. 
+This quality filter is enabled by default. If you want to see all of your test 
+responses, including the "low-quality" responses, disable the quality filter. 
+
+To disable the quality filter, open the problem component, click the **Settings** tab,
+and then set the **Disable Quality Filter** setting to **True**.
+
 Grade an Open Response Assessment Problem
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -564,8 +594,7 @@ Grade Responses
    describes the response.
 #. If applicable, add additional feedback.
 
-   -  You can provide comments for the student in the **Written
-      Feedback** field.
+   -  You can provide comments for the student in the **Written Feedback** field.
    -  If you do not feel that you can grade the response (for example,
       if you're a member of course staff but you would rather have the
       instructor grade the response), you can click **Skip** to skip it.
@@ -593,6 +622,26 @@ Click **Back to problem list** to return to the list of problems. You
 can also wait for a few minutes and click **Re-check for submissions**
 to see if any other students have submitted responses.
 
+**Note**
+
+When a response opens for you to grade, it leaves the current "grading pool" 
+that other instructors or students are grading from, which prevents other 
+instructors or students from 
+grading the response while you are working on it. If you do not submit a score 
+for this response within 30 minutes, the response returns to the grading pool 
+(so that it again becomes available for others to grade), even if you still have
+the response open on your screen.
+ 
+If the response returns to the grading pool (because the 30 minutes have passed), 
+but the response is still open on your screen, you can still submit feedback for 
+that response. If another instructor or student grades the response after it returns to the 
+grading pool but before you submit your feedback, the response receives two grades.
+ 
+If you click your browser's **Back** button to return to the problem list before you 
+click **Submit** to submit your feedback for a response, the response stays outside 
+the grading pool until 30 minutes have passed. When the response returns to the 
+grading pool, you can grade it. 
+
 Access Scores and Feedback
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/course_authors/source/ora_students.rst

@@ -0,0 +1,366 @@
+.. _ORA for Students:
+
+Open Response Assessments for Students
+======================================
+
+.. _ORA Introduction:
+
+Introduction to Open Response Assessments
+-----------------------------------------
+
+Open response assessments allow you to submit a short written answer, 
+an essay, or a file such as an image or computer code file. 
+
+When you come to an open response assessment problem, you see the name of the
+problem, the assessment types, the text of the question, the field where you'll
+enter your response, and the **Save** and **Submit** buttons. 
+
+.. image:: /Images/ExampleORA.gif
+
+If an open response assessment asks you to submit a file, you'll also see a button
+that you'll click to upload your file.
+
+.. image:: /Images/ExampleORA_File.gif
+
+The *assessment types* can include *self assessment*, *peer assessment*, and *artificial intelligence (AI) assessment*. The
+assessment types run in the order in which they appear in the problem. 
+
+- In a self assessment, you assess your response according a rubric that the
+  instructor has created. For more information, see :ref:`ORA Self Assessment`.
+  
+- In a peer assessment, you grade
+  responses that your peers have submitted while several of your peers
+  grade your response. For more information, see 
+  :ref:`ORA Peer Assessment`.
+  
+- In an AI assessment, a computer algorithm grades your response. For more information, 
+  see :ref:`ORA AI Assessment`.
+
+An open response assessment problem doesn't have to use all assessment types. For example, one problem
+may use self assessment and AI assessment, while another problem uses self assessment
+and peer assessment, and another problem uses only peer assessment.
+
+You'll answer open response assessment problems in much the same way that you answer other
+problems. For more information about how to submit responses, see :ref:`ORA Submit a Response`.
+ 
+When you submit a response to an open response assessment, the next step 
+depends on the type of assessment that the problem uses. For more information,
+see :ref:`ORA Self Assessment`, :ref:`ORA Peer Assessment`, and :ref:`ORA AI Assessment`.
+
+After you submit your response, your score will be available shortly - sometimes within a few
+minutes. For information about how to access your score after your response has been graded,
+see :ref:`ORA Access Scores`.
+
+If you want to experiment with open response assessments, you can try out the open
+assessment problems in the `EdX Demo <https://courses.edx.org/courses/edX/DemoX/Demo_Course/info>`_
+course. To get started, go 
+to the `Self-Assessed Essay <https://courses.edx.org/courses/edX/DemoX/Demo_Course/courseware/graded_interactions/machine_grading/2>`_ 
+unit, and then enter a response in the **Response** field under the
+question. You can enter your own response, or you can use one of the sample 
+responses in the `Sample Answers <https://courses.edx.org/courses/edX/DemoX/Demo_Course/courseware/graded_interactions/machine_grading/6/>`_
+unit. 
+
+.. _ORA Submit a Response:
+
+Submit a Response
+-----------------
+
+Submitting a response is slightly different if you're submitting a written response
+or uploading a file.
+
+#. Enter the response that you want to submit.
+
+   - If you're submitting a written response, type your response in the 
+     **Response** field.
+     
+   - If you're uploading a file, click **Choose File** under the **Response** 
+     field. In the dialog box that opens, select the file that you want to upload, 
+     and then click **Open**.
+
+#. Click **Submit**, and then click **OK** in the dialog box to continue.
+
+   **Note** If you want to save your response and work on it again later, click **Save**. 
+   An "Answer saved, but not yet submitted" message appears directly under the **Save** and 
+   **Submit** buttons.
+   
+After you submit your response, the assessment types start running in the order in which they
+appear in the problem. For more information,
+see :ref:`ORA Self Assessment`, :ref:`ORA Peer Assessment`, or :ref:`ORA AI Assessment`.
+
+.. _ORA Self Assessment:
+
+Self Assessment
+---------------
+
+In a self assessment, the rubric for the problem appears below your response immediately
+after you submit the response. You then assess your response based on the rubric.
+
+Perform a Self Assessment
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Submit a response to a self-assessed ORA problem.
+
+#. When the rubric appears, compare your response with the rubric, and select the 
+   option that you think is appropriate for each category.
+
+   .. image:: /Images/Rubric1.gif
+
+#. Click **Submit assessment**.
+
+   Your response appears, and you can see the scores that you gave
+   yourself.
+
+.. _ORA Peer Assessment:
+
+Peer Assessment
+---------------
+
+In a peer assessment, several students in the course grade your response while you grade
+other students' responses. You have to grade a number of your peers' responses before 
+you receive your score. (After you grade the minimum number of responses required to 
+receive your score, you can grade as many additional responses as you want.)
+
+After you submit your response for grading, the following 
+message appears under your response.
+
+    **Your response has been submitted. Please check back later for your grade.**
+
+Peer Grading Interface
+~~~~~~~~~~~~~~~~~~~~~~
+
+The area where you'll grade responses is the *peer
+grading interface*. Each course that has peer assessments has at least
+one peer grading interface. There may be just one peer grading interface
+for the whole course, or each individual problem may have its own
+separate peer grading interface.
+
+.. image:: /Images/PGI_FromOEC_2Problems.gif
+
+
+Perform a Peer Assessment
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Performing a peer assessment has several steps. You can find detailed instructions for each step
+below.
+
+#. :ref:`Access Responses`, either in the body of the
+   course or from the **Open Ended Console** page.
+#. :ref:`Learn to Grade` (this process is called
+   *calibration*).
+#. :ref:`Grade Responses` from other students.
+
+.. _Access Responses:
+
+Step 1: Access responses from other students
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**Note** *You can only grade a response if you've submitted a response to the
+question, an instructor has already graded at least 20 responses, and
+there are more essays from other students left to grade. If you haven't submitted 
+a response or no responses are available for grading, you see a yellow message in the
+interface.*
+
+.. image:: /Images/PAStudent_NoSubmissions.gif
+
+There are several ways to access other students' responses, depending on
+the way that the course is set up. 
+
+-  Through the **Open Ended Console** page. This option is always
+   available for every course. To access the **Open Ended Console** page, 
+   click the **Open Ended Panel** tab at the top of any page in the course.
+   When you see the list of problems that have responses available to grade,
+   click the name of the problem that you want to open it. 
+
+   .. image:: /Images/PGI_FromOEC_2Problems.gif
+
+-  Through the courseware, in a specific unit. This option is only available if the
+   instructor has included a peer grading interface for the problem in the body of
+   the course. To access responses in the courseware, go to the unit that contains 
+   the open response assessment problem. Scroll down past the response that you 
+   submitted until you see the peer grading interface that appears below the problem.
+   
+   .. image:: /Images/PGI_InUnitComposite.gif
+   
+-  Through the courseware, in a separate section. This option may not be available
+   for your course. If it is, you'll see the section for peer grading in the 
+   course accordion on the left side of your screen. For example, MIT's 6.00x: 
+   Introduction to Computer Science and Programming course has a separate section 
+   that holds all the course peer grading interfaces. To access peer grading for 
+   a problem, you click the problem name.
+   
+   .. image:: /Images/PGI_Multiple-600x.gif
+
+
+.. _Learn to Grade:
+
+Step 2: Learn to grade
+^^^^^^^^^^^^^^^^^^^^^^
+
+Before you grade your peers' responses, you must learn to grade
+the same way that an instructor would. In this process, called
+*calibration*, you'll grade several responses that an instructor has already 
+graded. If your grading is similar to the instructor's, you can begin grading 
+other students' responses to the question. 
+
+#. Click the name of the problem. When the **Learning to grade** page
+   opens, click **Start learning to grade**.
+
+#. When the problem opens, compare the student's response with the
+   rubric. Select the options that best apply to the response, and then
+   click **Submit**.
+
+#. Review the **How did I do?** message that you receive, and then click
+   **Continue**.
+
+   .. image:: /Images/PG_Calibration_Correct.gif
+
+   .. image:: /Images/PG_Calibration_Incorrect.gif
+
+   When you click **Continue**, the next student response appears for
+   you to grade, and you see a yellow **Calibration essay saved** message in
+   the top left corner of the page.
+
+#. Continue to grade responses. After you grade the required number of
+   responses correctly, you receive a **Ready to grade!** message. You
+   can then start to grade responses for other students.
+
+.. _Grade Responses:
+
+Step 3: Grade responses
+^^^^^^^^^^^^^^^^^^^^^^^
+
+When you grade a peer assessment response, you can not only select
+options in the rubric, but also provide additional feedback for the 
+student who submitted the response.
+
+#. When the response opens, select the options in the rubric that you
+   feel best apply to the response, as you did in the calibration process.
+
+   If you have concerns about the response, you can select other
+   options to flag the response for instructor review. You don't have to fill 
+   out the rubric before you select these options.
+
+   -  If you aren't sure how to grade the response, select the **I am unsure about
+      the scores I have given above** check box. 
+   -  If the response is offensive, or if you suspect that it contains plagiarized 
+      material, select the **This submission has explicit, offensive, or (I suspect)
+      plagiarized content** check box.
+      
+#. Under **Written Feedback**, write a comment about the score that you
+   gave the response.
+
+#. Click **Submit**. You see a **Successfully saved your feedback**
+   message at the top of the screen, and the next response opens.
+
+#. Continue to grade until you've graded the required number of
+   responses (usually 3). When you've graded enough responses, you
+   receive the following message.
+
+   .. image:: /Images/DoneGrading.gif
+
+   When you see this message, you can access the score for your own
+   response. For more information, see :ref:`ORA Access Scores`.
+
+If you want to grade additional responses at any time, you can go back
+to the **Peer Grading** page and click the name of the problem that you want
+to continue grading.
+
+**Note**
+
+When a response opens for you to grade, it leaves the current "grading pool"
+that other instructors or students are grading from, which prevents other 
+instructors or students from 
+grading the response while you are working on it. If you do not submit a score
+for this response within 30 minutes, the response returns to the grading pool 
+(so that it again becomes available for others to grade), even if you still have
+the response open on your screen.
+ 
+If the response returns to the grading pool (because the 30 minutes have passed), 
+but the response is still open on your screen, you can still submit feedback for 
+that response. If another instructor or student grades the response after it returns to the 
+grading pool but before you submit your feedback, the response receives two grades.
+ 
+If you click your browser<65>s **Back** button to return to the problem list before you 
+click **Submit** to submit your feedback for a response, the response stays outside 
+the grading pool until 30 minutes have passed. When the response returns to the 
+grading pool, you can grade it. 
+
+.. _ORA AI Assessment:
+
+Artificial Intelligence (AI) Assessment
+---------------------------------------
+
+In an AI assessment, an instructor grades a sample set of student responses to the 
+open response assessment problem. A machine learning algorithm then creates a model 
+based on the instructor's scores and grades the remaining students' responses.
+
+After you submit your response to an AI assessment, the following message appears under your
+response.
+
+    **Your response has been submitted. Please check back later for your grade.**
+    
+Depending on the time that it takes for the instructor to grade a sample set of
+responses, you may receive your grade within minutes, or you may have to wait
+a few days. You won't receive a notification when your score is ready, so keep
+checking back. 
+
+For more information about accessing your scores, see :ref:`ORA Access Scores`.
+
+.. _ORA Access Scores:
+
+Access Scores and Feedback
+--------------------------
+
+For *self assessments*, the score that you give yourself appears as soon as you submit
+the score. 
+
+For *peer assessments* and *AI assessments*, you'll access your scores through the **Open Ended Console** page.
+
+#. In the EdX Demo course, click the **Open Ended Panel** tab at the top
+   of the page.
+
+#. On the **Open Ended Console** page, click **Problems You Have
+   Submitted**.
+
+#. On the **Open Ended Problems** page, check the **Status** column to
+   see whether your responses have been graded. The status for each problem is
+   either **Waiting to be Graded** or **Finished**.
+
+#. If **Finished** appears in the **Status** column for the problem you want, 
+   click the name of the problem to see your score for that problem. When you
+   click the name of the problem, the problem opens in the courseware.
+
+For both AI and peer assessments, the score appears below your response
+in an abbreviated version of the rubric. 
+
+.. image:: /Images/AIScoredResponse.gif
+
+For peer assessments, you can
+also see the written feedback that your response received from different
+graders.
+
+.. image:: /Images/PeerScoredResponse.gif
+
+If you want to see the full rubric for either an AI or peer assessment,
+click **Toggle Full Rubric**.
+
+**Note** *For a peer assessment, if you haven't yet graded enough
+problems to see your score, you receive a message that lets you know how
+many problems you still need to grade.*
+
+.. image:: /Images/FeedbackNotAvailable.gif
+
+For more information about grading peer assessments, see :ref:`ORA Peer Assessment`.
+
+Resubmitting a Response
+-----------------------
+
+Some open response assessments allow multiple attempts. For these
+problems, a **New Submission** button appears below your original
+response.
+
+If you want to answer the question again, click **New Submission** to
+clear your former response, and click **OK** in the dialog box that
+appears. You can then enter a new response for the problem.
+
+