diff --git a/docs/course_authors/source/Images/AIScoredResponse.gif b/docs/course_authors/source/Images/AIScoredResponse.gif new file mode 100644 index 0000000000..cdffaa4574 Binary files /dev/null and b/docs/course_authors/source/Images/AIScoredResponse.gif differ diff --git a/docs/course_authors/source/Images/DoneGrading.gif b/docs/course_authors/source/Images/DoneGrading.gif new file mode 100644 index 0000000000..8e0d5e20c5 Binary files /dev/null and b/docs/course_authors/source/Images/DoneGrading.gif differ diff --git a/docs/course_authors/source/Images/ExampleORA.gif b/docs/course_authors/source/Images/ExampleORA.gif new file mode 100644 index 0000000000..47b0f75902 Binary files /dev/null and b/docs/course_authors/source/Images/ExampleORA.gif differ diff --git a/docs/course_authors/source/Images/ExampleORA_File.gif b/docs/course_authors/source/Images/ExampleORA_File.gif new file mode 100644 index 0000000000..621fa00872 Binary files /dev/null and b/docs/course_authors/source/Images/ExampleORA_File.gif differ diff --git a/docs/course_authors/source/Images/ORAComponentEditor.gif b/docs/course_authors/source/Images/ORAComponentEditor.gif new file mode 100644 index 0000000000..7954c9b08b Binary files /dev/null and b/docs/course_authors/source/Images/ORAComponentEditor.gif differ diff --git a/docs/course_authors/source/Images/PAStudent_NoSubmissions.gif b/docs/course_authors/source/Images/PAStudent_NoSubmissions.gif new file mode 100644 index 0000000000..b7f63b07c1 Binary files /dev/null and b/docs/course_authors/source/Images/PAStudent_NoSubmissions.gif differ diff --git a/docs/course_authors/source/Images/PGI_FromOEC_2Problems.gif b/docs/course_authors/source/Images/PGI_FromOEC_2Problems.gif new file mode 100644 index 0000000000..47ac20ca13 Binary files /dev/null and b/docs/course_authors/source/Images/PGI_FromOEC_2Problems.gif differ diff --git a/docs/course_authors/source/Images/PGI_InUnitComposite.gif b/docs/course_authors/source/Images/PGI_InUnitComposite.gif new file mode 100644 index 0000000000..ecc3951fa6 Binary files /dev/null and b/docs/course_authors/source/Images/PGI_InUnitComposite.gif differ diff --git a/docs/course_authors/source/Images/PGI_Multiple-600x.gif b/docs/course_authors/source/Images/PGI_Multiple-600x.gif new file mode 100644 index 0000000000..ee5e256769 Binary files /dev/null and b/docs/course_authors/source/Images/PGI_Multiple-600x.gif differ diff --git a/docs/course_authors/source/Images/PG_Calibration_Correct.gif b/docs/course_authors/source/Images/PG_Calibration_Correct.gif new file mode 100644 index 0000000000..9fc5d24476 Binary files /dev/null and b/docs/course_authors/source/Images/PG_Calibration_Correct.gif differ diff --git a/docs/course_authors/source/Images/PG_Calibration_Incorrect.gif b/docs/course_authors/source/Images/PG_Calibration_Incorrect.gif new file mode 100644 index 0000000000..77ea20b8c6 Binary files /dev/null and b/docs/course_authors/source/Images/PG_Calibration_Incorrect.gif differ diff --git a/docs/course_authors/source/Images/PeerScoredResponse.gif b/docs/course_authors/source/Images/PeerScoredResponse.gif new file mode 100644 index 0000000000..9a6402df00 Binary files /dev/null and b/docs/course_authors/source/Images/PeerScoredResponse.gif differ diff --git a/docs/course_authors/source/Images/Rubric1.gif b/docs/course_authors/source/Images/Rubric1.gif new file mode 100644 index 0000000000..37df57b039 Binary files /dev/null and b/docs/course_authors/source/Images/Rubric1.gif differ diff --git a/docs/course_authors/source/index.rst b/docs/course_authors/source/index.rst index fe82ca5024..4f740908ae 100755 --- a/docs/course_authors/source/index.rst +++ b/docs/course_authors/source/index.rst @@ -30,6 +30,7 @@ Contents view_course_content export_import_course checking_student_progress + ora_students change_log diff --git a/docs/course_authors/source/open_response_assessment.rst b/docs/course_authors/source/open_response_assessment.rst index 0eb31d62e5..40fb1bd4f3 100644 --- a/docs/course_authors/source/open_response_assessment.rst +++ b/docs/course_authors/source/open_response_assessment.rst @@ -1,16 +1,14 @@ .. _Open Response Assessment Problems: -Open Response Assessments +Open Response Assessment Problems --------------------------------- Introduction to Open Response Assessments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. note:: - - Open response assessments are still in beta. We recommend that - you test them thoroughly in a practice course and only add them to - courses that are **not** already running. Contact your edX Program Manager for more information. +.. note:: Open response assessments are still in beta. We recommend that + you test them thoroughly in a practice course and only add them to + courses that are **not** already running. Open response assessments allow instructors to assess student learning through questions that may not have definite answers. Tens of thousands @@ -140,25 +138,27 @@ Creating an open response assessment is a multi-step process. Each of these steps is described in detail below. -1. Create the Component -~~~~~~~~~~~~~~~~~~~~~~~ +Step 1. Create the Component +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Add the advanced component for open response assessments. To do this, add the "peergrading","combinedopenended" key value to the **Advanced Settings** page. (For more information, see the instructions in - Specialized Problems.) + :ref:`Specialized Problems`.) #. In Studio, open the unit where you want to create the ORA. #. Under **Add New Component**, click **Advanced**, and then click **Open Response Assessment**. #. In the problem component that appears, click **Edit**, and then click **OK** in the dialog box that appears. #. The component editor opens. The component editor contains a sample - question ("prompt"), rubric, assessment type specification, and + question ("prompt"), rubric, and the code for the assessment type and scoring. You'll replace this sample content with the content for your problem. -2. Add the Question -~~~~~~~~~~~~~~~~~~~ +.. image:: Images/ORAComponentEditor.gif + +Step 2. Add the Question +~~~~~~~~~~~~~~~~~~~~~~~~ - In the component editor, locate the [prompt] tags. @@ -171,8 +171,12 @@ these guidelines to avoid common formatting mistakes. - Leave the **[prompt]** tags in place. - Enclose all text in HTML tags. -3. Add the Rubric -~~~~~~~~~~~~~~~~~ +Step 3. Add the Rubric +~~~~~~~~~~~~~~~~~~~~~~ + +**Note** *After you release your problem to students in your live course, make sure +you don't make any changes to the rubric that affect scoring, such as adding or removing an option +in a category. Changing the rubric can cause errors in live courses.* #. In the component editor, locate the [rubric] tags. (The sample rubric is long, so you'll have to scroll down to locate the second tag.) @@ -205,13 +209,16 @@ these guidelines to avoid common formatting mistakes. [rubric] -4. Set the Assessment Type and Scoring -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Step 4. Set the Assessment Type and Scoring +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To set the assessment type and scoring for your open response -assessment, you'll enter code that specifies the type and order of -assessments to use along with the scoring thresholds for each -assessment. The code uses the following format. +**Note** *After you release your problem to students in your live course, make sure +you don't make any changes to the code for the assessment type and scoring. Changing +this code can cause errors in live courses.* + +To set the assessment type and scoring for your open response assessment, you'll +enter code that specifies the type and order of assessments to use along with +the scoring thresholds for each assessment. The code uses the following format. :: @@ -257,8 +264,17 @@ Set the Type and Scoring #. Replace the sample code with the code for your problem. -5. Set the Problem Name -~~~~~~~~~~~~~~~~~~~~~~~ +Step 5. Set the Problem Name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Note** *After you release your problem to students in your live course, make sure +you don't change the name of the problem. Changing the display name when the problem +is live can cause a loss of student data.* + +*You can change the display name of a problem while you're still testing the problem. +However, note that all the test responses and scores associated with the problem +will be lost when you change the name. To update the problem name on the +instructor dashboard, submit a new test response to the problem.* The name of the problem appears as a heading above the problem in the courseware. It also appears in the list of problems on the **Staff @@ -273,8 +289,8 @@ To change the name: #. In the **Display Name** field, replace **Open Response Assessment** with the name of your problem. -6. Set Other Options -~~~~~~~~~~~~~~~~~~~~ +Step 6. Set Other Options +~~~~~~~~~~~~~~~~~~~~~~~~~ If you want to change the problem settings, which include the number of responses a student has to peer grade and whether students can upload @@ -304,7 +320,7 @@ Open response assessments include the following settings. | | responses that are very short or that have many spelling or | | | grammatical errors) to be peer graded. For example, you may | | | disable the quality filter if you want students to include URLs to | -| | external content—otherwise Studio sees a URL, which may contain a | +| | external content - otherwise Studio sees a URL, which may contain a| | | long string of seemingly random characters, as a misspelled word. | | | When the quality filter is enabled (when this value is set to | | | False), Studio does not allow poor-quality submissions to be peer | @@ -358,6 +374,10 @@ Open response assessments include the following settings. +---------------------------------------------+--------------------------------------------------------------------+ | **Problem Weight** | This setting specifies the number of points the problem is worth. | | | By default, each problem is worth one point. | +| | | +| | **Note** *Every problem must have a problem weight of at least | +| | one point. Problems that have a problem weight of zero points | +| | don't appear on the instructor dashboard. | +---------------------------------------------+--------------------------------------------------------------------+ | **Required Peer Grading** | This setting specifies the number of responses that each student | | | who submits a response has to grade before the student receives a | @@ -371,8 +391,8 @@ Open response assessments include the following settings. | | previously graded responses.) | +---------------------------------------------+--------------------------------------------------------------------+ -7. Save the Problem -~~~~~~~~~~~~~~~~~~~ +Step 7. Save the Problem +~~~~~~~~~~~~~~~~~~~~~~~~ - After you have created the prompt and the rubric, set the assessment type and scoring, changed the name of the problem, and specified any @@ -383,8 +403,8 @@ Open response assessments include the following settings. .. image:: Images/ORA_Component.gif -8. Add the Peer Grading Interface (for peer assessments only) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Step 8. Add the Peer Grading Interface (for peer assessments only) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can add just one peer grading interface for the whole course, or you can add a separate peer grading interface for each individual problem. @@ -457,8 +477,8 @@ week). #. Click **Save** to close the component editor. -9. Test the Problem -~~~~~~~~~~~~~~~~~~~ +Step 9. Test the Problem +~~~~~~~~~~~~~~~~~~~~~~~~ Test your problem by adding and grading a response. @@ -495,6 +515,16 @@ that you use as an instructor. this, you can't make changes to your course without signing out and signing back in as an instructor. +When you test your problem, you may want to submit test responses that contain little +text, random characters, or other content that doesn't resemble the responses that you +expect from your students. Open response assessments include a quality filter that +prevents instructors and other students from seeing these "low-quality" responses. +This quality filter is enabled by default. If you want to see all of your test +responses, including the "low-quality" responses, disable the quality filter. + +To disable the quality filter, open the problem component, click the **Settings** tab, +and then set the **Disable Quality Filter** setting to **True**. + Grade an Open Response Assessment Problem ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -564,8 +594,7 @@ Grade Responses describes the response. #. If applicable, add additional feedback. - - You can provide comments for the student in the **Written - Feedback** field. + - You can provide comments for the student in the **Written Feedback** field. - If you do not feel that you can grade the response (for example, if you're a member of course staff but you would rather have the instructor grade the response), you can click **Skip** to skip it. @@ -593,6 +622,26 @@ Click **Back to problem list** to return to the list of problems. You can also wait for a few minutes and click **Re-check for submissions** to see if any other students have submitted responses. +**Note** + +When a response opens for you to grade, it leaves the current "grading pool" +that other instructors or students are grading from, which prevents other +instructors or students from +grading the response while you are working on it. If you do not submit a score +for this response within 30 minutes, the response returns to the grading pool +(so that it again becomes available for others to grade), even if you still have +the response open on your screen. + +If the response returns to the grading pool (because the 30 minutes have passed), +but the response is still open on your screen, you can still submit feedback for +that response. If another instructor or student grades the response after it returns to the +grading pool but before you submit your feedback, the response receives two grades. + +If you click your browser's **Back** button to return to the problem list before you +click **Submit** to submit your feedback for a response, the response stays outside +the grading pool until 30 minutes have passed. When the response returns to the +grading pool, you can grade it. + Access Scores and Feedback ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/course_authors/source/ora_students.rst b/docs/course_authors/source/ora_students.rst new file mode 100644 index 0000000000..846128f1ec --- /dev/null +++ b/docs/course_authors/source/ora_students.rst @@ -0,0 +1,366 @@ +.. _ORA for Students: + +Open Response Assessments for Students +====================================== + +.. _ORA Introduction: + +Introduction to Open Response Assessments +----------------------------------------- + +Open response assessments allow you to submit a short written answer, +an essay, or a file such as an image or computer code file. + +When you come to an open response assessment problem, you see the name of the +problem, the assessment types, the text of the question, the field where you'll +enter your response, and the **Save** and **Submit** buttons. + +.. image:: /Images/ExampleORA.gif + +If an open response assessment asks you to submit a file, you'll also see a button +that you'll click to upload your file. + +.. image:: /Images/ExampleORA_File.gif + +The *assessment types* can include *self assessment*, *peer assessment*, and *artificial intelligence (AI) assessment*. The +assessment types run in the order in which they appear in the problem. + +- In a self assessment, you assess your response according a rubric that the + instructor has created. For more information, see :ref:`ORA Self Assessment`. + +- In a peer assessment, you grade + responses that your peers have submitted while several of your peers + grade your response. For more information, see + :ref:`ORA Peer Assessment`. + +- In an AI assessment, a computer algorithm grades your response. For more information, + see :ref:`ORA AI Assessment`. + +An open response assessment problem doesn't have to use all assessment types. For example, one problem +may use self assessment and AI assessment, while another problem uses self assessment +and peer assessment, and another problem uses only peer assessment. + +You'll answer open response assessment problems in much the same way that you answer other +problems. For more information about how to submit responses, see :ref:`ORA Submit a Response`. + +When you submit a response to an open response assessment, the next step +depends on the type of assessment that the problem uses. For more information, +see :ref:`ORA Self Assessment`, :ref:`ORA Peer Assessment`, and :ref:`ORA AI Assessment`. + +After you submit your response, your score will be available shortly - sometimes within a few +minutes. For information about how to access your score after your response has been graded, +see :ref:`ORA Access Scores`. + +If you want to experiment with open response assessments, you can try out the open +assessment problems in the `EdX Demo `_ +course. To get started, go +to the `Self-Assessed Essay `_ +unit, and then enter a response in the **Response** field under the +question. You can enter your own response, or you can use one of the sample +responses in the `Sample Answers `_ +unit. + +.. _ORA Submit a Response: + +Submit a Response +----------------- + +Submitting a response is slightly different if you're submitting a written response +or uploading a file. + +#. Enter the response that you want to submit. + + - If you're submitting a written response, type your response in the + **Response** field. + + - If you're uploading a file, click **Choose File** under the **Response** + field. In the dialog box that opens, select the file that you want to upload, + and then click **Open**. + +#. Click **Submit**, and then click **OK** in the dialog box to continue. + + **Note** If you want to save your response and work on it again later, click **Save**. + An "Answer saved, but not yet submitted" message appears directly under the **Save** and + **Submit** buttons. + +After you submit your response, the assessment types start running in the order in which they +appear in the problem. For more information, +see :ref:`ORA Self Assessment`, :ref:`ORA Peer Assessment`, or :ref:`ORA AI Assessment`. + +.. _ORA Self Assessment: + +Self Assessment +--------------- + +In a self assessment, the rubric for the problem appears below your response immediately +after you submit the response. You then assess your response based on the rubric. + +Perform a Self Assessment +~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. Submit a response to a self-assessed ORA problem. + +#. When the rubric appears, compare your response with the rubric, and select the + option that you think is appropriate for each category. + + .. image:: /Images/Rubric1.gif + +#. Click **Submit assessment**. + + Your response appears, and you can see the scores that you gave + yourself. + +.. _ORA Peer Assessment: + +Peer Assessment +--------------- + +In a peer assessment, several students in the course grade your response while you grade +other students' responses. You have to grade a number of your peers' responses before +you receive your score. (After you grade the minimum number of responses required to +receive your score, you can grade as many additional responses as you want.) + +After you submit your response for grading, the following +message appears under your response. + + **Your response has been submitted. Please check back later for your grade.** + +Peer Grading Interface +~~~~~~~~~~~~~~~~~~~~~~ + +The area where you'll grade responses is the *peer +grading interface*. Each course that has peer assessments has at least +one peer grading interface. There may be just one peer grading interface +for the whole course, or each individual problem may have its own +separate peer grading interface. + +.. image:: /Images/PGI_FromOEC_2Problems.gif + + +Perform a Peer Assessment +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Performing a peer assessment has several steps. You can find detailed instructions for each step +below. + +#. :ref:`Access Responses`, either in the body of the + course or from the **Open Ended Console** page. +#. :ref:`Learn to Grade` (this process is called + *calibration*). +#. :ref:`Grade Responses` from other students. + +.. _Access Responses: + +Step 1: Access responses from other students +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +**Note** *You can only grade a response if you've submitted a response to the +question, an instructor has already graded at least 20 responses, and +there are more essays from other students left to grade. If you haven't submitted +a response or no responses are available for grading, you see a yellow message in the +interface.* + +.. image:: /Images/PAStudent_NoSubmissions.gif + +There are several ways to access other students' responses, depending on +the way that the course is set up. + +- Through the **Open Ended Console** page. This option is always + available for every course. To access the **Open Ended Console** page, + click the **Open Ended Panel** tab at the top of any page in the course. + When you see the list of problems that have responses available to grade, + click the name of the problem that you want to open it. + + .. image:: /Images/PGI_FromOEC_2Problems.gif + +- Through the courseware, in a specific unit. This option is only available if the + instructor has included a peer grading interface for the problem in the body of + the course. To access responses in the courseware, go to the unit that contains + the open response assessment problem. Scroll down past the response that you + submitted until you see the peer grading interface that appears below the problem. + + .. image:: /Images/PGI_InUnitComposite.gif + +- Through the courseware, in a separate section. This option may not be available + for your course. If it is, you'll see the section for peer grading in the + course accordion on the left side of your screen. For example, MIT's 6.00x: + Introduction to Computer Science and Programming course has a separate section + that holds all the course peer grading interfaces. To access peer grading for + a problem, you click the problem name. + + .. image:: /Images/PGI_Multiple-600x.gif + + +.. _Learn to Grade: + +Step 2: Learn to grade +^^^^^^^^^^^^^^^^^^^^^^ + +Before you grade your peers' responses, you must learn to grade +the same way that an instructor would. In this process, called +*calibration*, you'll grade several responses that an instructor has already +graded. If your grading is similar to the instructor's, you can begin grading +other students' responses to the question. + +#. Click the name of the problem. When the **Learning to grade** page + opens, click **Start learning to grade**. + +#. When the problem opens, compare the student's response with the + rubric. Select the options that best apply to the response, and then + click **Submit**. + +#. Review the **How did I do?** message that you receive, and then click + **Continue**. + + .. image:: /Images/PG_Calibration_Correct.gif + + .. image:: /Images/PG_Calibration_Incorrect.gif + + When you click **Continue**, the next student response appears for + you to grade, and you see a yellow **Calibration essay saved** message in + the top left corner of the page. + +#. Continue to grade responses. After you grade the required number of + responses correctly, you receive a **Ready to grade!** message. You + can then start to grade responses for other students. + +.. _Grade Responses: + +Step 3: Grade responses +^^^^^^^^^^^^^^^^^^^^^^^ + +When you grade a peer assessment response, you can not only select +options in the rubric, but also provide additional feedback for the +student who submitted the response. + +#. When the response opens, select the options in the rubric that you + feel best apply to the response, as you did in the calibration process. + + If you have concerns about the response, you can select other + options to flag the response for instructor review. You don't have to fill + out the rubric before you select these options. + + - If you aren't sure how to grade the response, select the **I am unsure about + the scores I have given above** check box. + - If the response is offensive, or if you suspect that it contains plagiarized + material, select the **This submission has explicit, offensive, or (I suspect) + plagiarized content** check box. + +#. Under **Written Feedback**, write a comment about the score that you + gave the response. + +#. Click **Submit**. You see a **Successfully saved your feedback** + message at the top of the screen, and the next response opens. + +#. Continue to grade until you've graded the required number of + responses (usually 3). When you've graded enough responses, you + receive the following message. + + .. image:: /Images/DoneGrading.gif + + When you see this message, you can access the score for your own + response. For more information, see :ref:`ORA Access Scores`. + +If you want to grade additional responses at any time, you can go back +to the **Peer Grading** page and click the name of the problem that you want +to continue grading. + +**Note** + +When a response opens for you to grade, it leaves the current "grading pool" +that other instructors or students are grading from, which prevents other +instructors or students from +grading the response while you are working on it. If you do not submit a score +for this response within 30 minutes, the response returns to the grading pool +(so that it again becomes available for others to grade), even if you still have +the response open on your screen. + +If the response returns to the grading pool (because the 30 minutes have passed), +but the response is still open on your screen, you can still submit feedback for +that response. If another instructor or student grades the response after it returns to the +grading pool but before you submit your feedback, the response receives two grades. + +If you click your browser’s **Back** button to return to the problem list before you +click **Submit** to submit your feedback for a response, the response stays outside +the grading pool until 30 minutes have passed. When the response returns to the +grading pool, you can grade it. + +.. _ORA AI Assessment: + +Artificial Intelligence (AI) Assessment +--------------------------------------- + +In an AI assessment, an instructor grades a sample set of student responses to the +open response assessment problem. A machine learning algorithm then creates a model +based on the instructor's scores and grades the remaining students' responses. + +After you submit your response to an AI assessment, the following message appears under your +response. + + **Your response has been submitted. Please check back later for your grade.** + +Depending on the time that it takes for the instructor to grade a sample set of +responses, you may receive your grade within minutes, or you may have to wait +a few days. You won't receive a notification when your score is ready, so keep +checking back. + +For more information about accessing your scores, see :ref:`ORA Access Scores`. + +.. _ORA Access Scores: + +Access Scores and Feedback +-------------------------- + +For *self assessments*, the score that you give yourself appears as soon as you submit +the score. + +For *peer assessments* and *AI assessments*, you'll access your scores through the **Open Ended Console** page. + +#. In the EdX Demo course, click the **Open Ended Panel** tab at the top + of the page. + +#. On the **Open Ended Console** page, click **Problems You Have + Submitted**. + +#. On the **Open Ended Problems** page, check the **Status** column to + see whether your responses have been graded. The status for each problem is + either **Waiting to be Graded** or **Finished**. + +#. If **Finished** appears in the **Status** column for the problem you want, + click the name of the problem to see your score for that problem. When you + click the name of the problem, the problem opens in the courseware. + +For both AI and peer assessments, the score appears below your response +in an abbreviated version of the rubric. + +.. image:: /Images/AIScoredResponse.gif + +For peer assessments, you can +also see the written feedback that your response received from different +graders. + +.. image:: /Images/PeerScoredResponse.gif + +If you want to see the full rubric for either an AI or peer assessment, +click **Toggle Full Rubric**. + +**Note** *For a peer assessment, if you haven't yet graded enough +problems to see your score, you receive a message that lets you know how +many problems you still need to grade.* + +.. image:: /Images/FeedbackNotAvailable.gif + +For more information about grading peer assessments, see :ref:`ORA Peer Assessment`. + +Resubmitting a Response +----------------------- + +Some open response assessments allow multiple attempts. For these +problems, a **New Submission** button appears below your original +response. + +If you want to answer the question again, click **New Submission** to +clear your former response, and click **OK** in the dialog box that +appears. You can then enter a new response for the problem. + + \ No newline at end of file