Commit Graph

8070 Commits

Author SHA1 Message Date
SaadYousaf
782ee338c9 fix: remove pre tag processing for course import 2022-02-21 16:29:01 +05:00
SaadYousaf
65f7800083 refactor: cleanup investigation logs 2022-02-21 13:28:39 +05:00
Carlos Muniz
37bf73ca50 refactor: Replace videosequence & problemset (#29905)
`videosequence` and `problemset` have been replaced with `sequential`.

`problemset` and `videosequence` are old-but-not-entirely-unused aliases to the `sequential` block type (in Studio-speak, "Subsection").
Since [these block types have been removed from the 6 courses that used them](https://openedx.atlassian.net/browse/DEPR-151?focusedCommentId=588197), this ticket removes the support for the `problemset` and `videosequence` block-types.

For more information, see ticket: [DEPR-151](https://openedx.atlassian.net/browse/DEPR-151)
2022-02-16 10:21:31 -05:00
Bianca Severino
af81b3a609 chore: replace integrity signature flag with django setting 2022-02-15 09:57:00 -05:00
Michael Terry
e81c4fe49d Merge pull request #29911 from openedx/mikix/more-old-mongo-test-cleanup
test: more Old Mongo removal from tests
2022-02-11 15:10:45 -05:00
Michael Terry
b905de757b test: more Old Mongo removal from tests
Convert more tests from MONGO_AMNESTY to SPLIT modulestores.

This is in preparation for just wholesale denying access to Old
Mongo, so I either converted tests to split or just deleted some
test variants that were Old Mongo specific. (e.g. ddt lines)
2022-02-11 14:50:30 -05:00
Pooja Kulkarni
b7d420a29e * feat: Add completion progress bar to dashboard
[SE-4473] Add completion progress bar to dashboard (#29271)
2022-02-09 09:28:48 -05:00
Michael Terry
cb1bb7fa64 test: switch default test store to the split store
It's long past time that the default test modulestore was Split,
instead of Old Mongo. This commit switches the default store and
fixes some tests that now fail:
- Tests that didn't expect MFE to be enabled (because we don't
  enable MFE for Old Mongo) - opt out of MFE for those
- Tests that hardcoded old key string formats
- Lots of other random little differences

In many places, I didn't spend much time trying to figure out how to
properly fix the test, and instead just set the modulestore to Old
Mongo.

For those tests that I didn't spend time investigating, I've set
the modulestore to TEST_DATA_MONGO_AMNESTY_MODULESTORE - search for
that string to find further work.
2022-02-04 14:32:50 -05:00
SaadYousaf
87b029f293 fix: add investigation logs for course import. 2022-02-04 15:45:57 +05:00
SaadYousaf
314aa86f1e fix: remove investigation logs 2022-02-04 12:25:56 +05:00
SaadYousaf
52db919b6c fix: add further additional logs for course import 2022-02-03 12:25:29 +05:00
SaadYousaf
7a4ed05279 fix: add additional logs to investigate import issue 2022-02-02 15:07:38 +05:00
Peter Pinch
4f58ed4f25 Merge pull request #29260 from open-craft/jill/BD-13-sandbox
[BD-13] refactor: deprecates ModuleSystem properties related to code sandboxing
2022-01-26 09:57:41 -05:00
SaadYousaf
739083e301 fix: remove check for default values for video component when exporting. 2022-01-26 18:14:25 +05:00
Jillian Vogel
57b390f8e2 refactor: removes calls to deprecated properties from source and tests. 2022-01-25 14:49:21 +10:30
Jillian Vogel
2173a98ef8 refactor: deprecates ModuleSystem properties for code sandboxing and cache
* Deprecates ModuleSystem can_execute_unsafe_code, get_python_lib_zip and cache properties
* Adds a new CacheService and SandboxService to provide the deprecated property
* Adds tests for the added CacheService and SandboxService
* Updates the ModuleSystemShim tests in Lms and Studio
2022-01-25 14:42:39 +10:30
Simon Chen
895cc10c1b fix: do not validate proctoring_provider if proctoring is disabled (#29803)
Co-authored-by: Simon Chen <schen@edx-c02fw0guml85.lan>
2022-01-21 12:45:43 -05:00
SaadYousaf
2c6989feab fix: correct studio sequence navigation bar with correct Next and Previous button actions 2022-01-12 16:20:05 +05:00
Jillian Vogel
31827bc3b1 refactor: removes deprecated calls to runtime.xqueue from code and tests
* Uses the XQueueService object from the ModuleSystem in the LoncapaSystem
  instead of the dict of xqueue-related values.
* Adds a StubXQueueService for testing.
2021-12-20 15:41:42 +01:00
Jillian Vogel
1974bacadd refactor: deprecates runtime.xqueue in favor of the XQueueService
* Deprecates ModuleSystem.xqueue property
* Adds new XQueueService to provide the deprecated property values to the LMS runtime
  (Studio does not need the XQueueService.)
* Adds tests for new service and updates the ModuleSystemShim tests in LMS and Studio
* Fixes existing tests.
2021-12-20 15:41:42 +01:00
Jillian Vogel
f828d89feb refactor: use UserService to remove deprecation warnings
from code and tests.
2021-12-20 15:41:25 +01:00
Jillian Vogel
e378e42bfa refactor: deprecate user-related ModuleSystem attributes
Deprecates these ModuleSystem attributes in favor of the user service:

* user_location
* get_real_user
* get_user_role

Related changes:

* Stores the user location into DjangoXBlockUserService's optional attribute as request_country_code
* Uses the student model's user_by_anonymous_it to fetch the (cached) real user
* Updates affected tests
2021-12-20 15:41:25 +01:00
Jillian Vogel
ad5ad72273 [BD-13] Deprecate ModuleSystem.render_template (fixed) (#29354)
* refactor: deprecates ModuleSystem.render_template

in favor of the added MakoSystem render_template method.

Related changes:
* Adds the MakoService to the StudioEditModuleRuntime,
  PreviewModuleSystem, LmsModuleSystem, and XBlockRuntime
* MakoService constructor takes a `namespace_prefix` string, so that the
  CMS PreviewModuleSystem can render to LMS templates, without needing
  the special render_from_lms helper method.
* ModuleSystem.render_template becomes a read-only property, so the
  constructor calls and test module systems are updated accordingly.
* Adds tests for the MakoService and module system shims.

(cherry picked from commit 457f959356)

* refactor: use MakoService.render_template to remove deprecation warnings

from block code.

(cherry picked from commit 8d62d337f5)

* refactor: use MakoService.render_template to remove deprecation warnings

from test code.

(cherry picked from commit 26b43465a4)

* test: Adds a test to verify the bug introduced by the previous changes

The AuthoringMixin is automatically added to all XBlocks (see
settings.XBLOCK_MIXINS), and AuthoringMixin.visibility_view expects the
"mako" service.

This test verifies the bug by testing the PureXBlock, which does not
require the "mako" service, and so fails when the visibility_view is
rendered.

* fix: AuthoringMixin needs mako service

which fixes the visibility_view for XBlocks which don't explicitly
require the mako service.

Also removes the unneeded class property _services_requested from
AuthoringMixin and StudioEditableBlock. This property is better provided
by the XBlockMixin class.
2021-11-29 14:42:52 -05:00
Kshitij Sobti
e8e8f4acbe feat!: Change the way tabs are ordered [BD-38] [TNL-9174] [BB-5076] (#29262)
* feat!: Change the way tabs are ordered
The change imposes a new ordering for tabs based on their new priority. When reordering tabs, this ordering will be maintained.

* fix: Apply suggestions from code review

Co-authored-by: Farhaan Bukhsh <farhaan@opencraft.com>

* fix: review feedback

Co-authored-by: Farhaan Bukhsh <farhaan@opencraft.com>
2021-11-22 11:17:30 +05:00
Justin Lapierre
f6f3a79a15 Revert "[BD-13] Deprecate ModuleSystem.render_template" 2021-11-16 15:16:08 -05:00
Piotr Surowiec
4bf829dcbd fix: do not remove comments from XML during course import (#28557)
This is a follow-up to edx#1087, which reverted this change.
According to the PR comments, parsing strings with XML comments inside them was
causing errors. This does not seem to be the case anymore - these strings are
just hidden when the block is rendered, but they are not breaking XBlocks.
This also handles (ignores) the comments that could be added directly to the
LibraryContentBlock in the XML export by users.
2021-11-16 12:42:08 -05:00
Ned Batchelder
d9dd10dc97 style: code cleanups from Steven Burch (#29292)
* chore: update deprecated import from collections

* chore: remove outdated imports from markdown library

as it hasn't been supported since 2.0.3 and we're on 3.x.
This was deprecated at least as early as 2012!

* docs: add docstring and remove lint-amnesty to markdown plugin

* chore: remove deprecated etree import

* style: remove unnecessary-comprehension for sets

* style: resolve a number of amnestied pylint complaints

Co-authored-by: stvn <stvn@mit.edu>
2021-11-10 07:11:57 -08:00
David Ormsbee
2d60224125 Merge pull request #28571 from open-craft/jill/bd-13-render_template
feat!: [BD-13] Deprecate ModuleSystem.render_template
2021-11-09 12:20:59 -05:00
David Ormsbee
0ec2a3b9b5 docs: ADR for removing MongoDB usage. 2021-11-09 10:55:28 -05:00
Jillian Vogel
26b43465a4 refactor: use MakoService.render_template to remove deprecation warnings
from test code.
2021-11-09 14:03:54 +10:30
Jillian Vogel
8d62d337f5 refactor: use MakoService.render_template to remove deprecation warnings
from block code.
2021-11-09 14:03:54 +10:30
Jillian Vogel
457f959356 refactor: deprecates ModuleSystem.render_template
in favor of the added MakoSystem render_template method.

Related changes:
* Adds the MakoService to the StudioEditModuleRuntime,
  PreviewModuleSystem, LmsModuleSystem, and XBlockRuntime
* MakoService constructor takes a `namespace_prefix` string, so that the
  CMS PreviewModuleSystem can render to LMS templates, without needing
  the special render_from_lms helper method.
* ModuleSystem.render_template becomes a read-only property, so the
  constructor calls and test module systems are updated accordingly.
* Adds tests for the MakoService and module system shims.
2021-11-09 14:03:50 +10:30
Braden MacDonald
480e8997ec fix: A test (test_save_one_and_confirm) was flaky 2021-11-08 10:46:33 -08:00
David Ormsbee
15eca3ba59 Merge pull request #29190 from open-craft/jill/BD-13-user-service-fix
[FAL-2457] Let ModuleSystem use UserService
2021-11-08 09:18:10 -05:00
Farhaan Bukhsh
cfe75e95b9 feat: Add discussion_enabled for Unit (#28864)
For each unit discussion_enabled flag is added
so that each unit can be made discussable when needed.

Signed-off-by: Farhaan Bukhsh <farhaan@opencraft.com>
2021-11-08 16:17:07 +05:00
Jansen Kantor
b6c92513c2 style: add blank line (#29199) 2021-11-02 16:05:17 -04:00
Jillian Vogel
f8c96c1df7 chore: pylint 2021-11-02 20:04:33 +10:30
Jillian Vogel
b1b3c646ad refactor: fixes tests
* UserStubService now takes user, user_is_staff, and anonymous_user_id
* get_test_system() creates a UserStubService with an anonymous_user_id of 'student'
* Removes references to deprecated ModuleSystem attributes from test code
* Fixes and simplifies the ConditionalBlock tests, using get_module provided by TestModuleSystem instead of trying to mock out all the pieces.

(cherry picked from commit 927016a8df)
2021-11-02 18:25:52 +10:30
Jillian Vogel
bafc0cd91f refactor: removes calls to deprecated ModuleSystem attributes
Removes references to these deprecated attributes from the platform code:
* runtime.anonymous_student_id
* runtime.seed
* runtime.user_id
* runtime.user_is_staff

Related changes:

* Ensure that all platform XBlocks which use these attributes "need" the user service.
* ProblemBlock: Removes check for existence of runtime.seed attribute in preparation for removal of this attribute from ModuleSystem.
* edxnotes: Catches NoSuchServiceError just in case some XBlocks using notes don't have the user service.
* UserTagsService refactor: pass user and course_id on creation

(cherry picked from commit 753839276f)
2021-11-02 18:24:14 +10:30
Jillian Vogel
cf1064616c refactor: deprecate ModuleSystem user attributes in favor of user service
The following ModuleSystem attributes are deprecated by this change, and should be pulled directly from the user service instead:

* anonymous_student_id
* seed
* user_id
* user_is_staff

Related changes:

* Removes the `user` and `anonymous_student_id` parameters from the ModuleService constructor.
* Stores anonymous_user_id in XBlockDjangoUserService's opt_attr
* Pulls out constants used by DjangoXBlockUserService opt_attr so they can be used in the platform code.
* LmsModuleSystem uses the user service created in wrapper function for runtime.publish to avoid requiring the user
  service to be "needed" by all XBlocks.
* LmsModuleSystem no longer checks for instances of XModuleDescriptor when deciding what kind of anonymous_user_id to
  provide:  all XModules are XBlocks, so this check is unnecessary.
* XBlockRuntime returns a user service when requested
* Adds tests for deprecated ModuleSystem attributes and changes to XBlockDjangoUserService.

(cherry picked from commit c41e7fb93a)
2021-11-02 18:24:06 +10:30
David Ormsbee
3754bd219a fix: force reset of modulestore data for test cases
In #8bdf59b, the SplitModuleStore was given a Django ORM backed store
for active version data (i.e. "which version is currently published?").
This started to cause sporadic test failures depending on the test
ordering, such as this module:

  openedx/features/course_experience/tests/views/test_course_home.py

The root cause was that the database table holding these active versions
was not being properly cleared after tests, probably because of the odd
ordering we do MongoDB vs. Django ORM data initialization in the
ModuleStoreTestCase and SharedModuleStoreTestCase classes. This is an
overly broad hammer fix for this, because:

1. The obvious thing to add it into the ModuleStoreIsolationMixin didn't
   seem to work.
2. While overly broad, it's a small bit of code and should be safe.
3. It's more urgent to fix this flakiness in the build (affecting maybe
   1/4 test runs?) ASAP, rather than tracking this down.
2021-11-01 17:41:08 -04:00
Kshitij Sobti
ff61434893 feat: Add a new way to enable/disable teams [BD-38] [TNL-9175] [BB-5066] (#29099)
* feat: Add a new way to enable/disable teams

Adds a new mechanism for enabling/disabling the team feature in a course using an 'enabled' field to the teams config.
If this field is set to true, teams is enabled (team sets/groups) still need to be defined. If this is set to false then teams is disabled whether or not team sets are defined.

* fix: review feedback
2021-11-01 17:12:06 +05:00
Awais Jibran
5f97bc60cf test: adds discussion API permissions test cases (#29135) 2021-10-28 14:54:32 +05:00
Kshitij Sobti
79cd0b1ef8 feat: Adds discussions settings for new discusions experience [BD-38] [TNL-8621] [BB-4854] (#29131)
* feat: Adds discussions settings for new discusions experience
This commit adds new discussions settings for the new discussions experience. These are stored in the course so they can be a part of course import/export flow.
These are also added to the discussions configuraiton API to allow MFEs to update the settings.
The discussions API is currently available via LMS, however that means it cannot save changes to the modulestore. This also adds the API to the studio config so it can now also be accessed from studio and be used to save course settings.

* fix: tests
2021-10-28 11:56:17 +05:00
Ken Clary
bfd09236a2 fix: default for CertificatesDisplayBehaviors is enum string value, not enum.
This should correct a large number of errors encountered when trying
to dump courses to neo4j.

TNL-8386
2021-10-27 14:59:36 -04:00
connorhaugh
b01e773d2b feat: cache course index queries (#29107)
* feat: store split modulestore's course indexes in Django/MySQL

Course outline generation is calling active_versions hundreds or even thousands of times on course_publish. In practice, the handling of a block requires a fetch of the course index. That requires a call to the active_versions table for each touching of a block. In larger courses, that can mean 1000s of calls to the db to retrieve them. A quick solution to this problem is to use a request cache, so we don't have to query the slow db every time and memoize the relevant data
[An example run can be seen here.](https://one.newrelic.com/launcher/nr1-core.explorer?platform[filters]=IihuYW1lIGxpa2UgJ3Byb2QtZWR4LWVkeGFwcCcgb3IgaWQgPSAncHJvZC1lZHgtZWR4YXBwJyBvciBkb21haW5JZCA9ICdwcm9kLWVkeC1lZHhhcHAnKSBhbmQgKG5hbWUgbGlrZSAnbG1zJyBvciBpZCA9ICdsbXMnIG9yIGRvbWFpbklkID0gJ2xtcycpIg==&platform[accountId]=88178&platform[timeRange][begin_time]=1626715880701&platform[timeRange][end_time]=1627320680701&pane=eyJuZXJkbGV0SWQiOiJhcG0tbmVyZGxldHMudHJhbnNhY3Rpb25zIiwiZW50aXR5R3VpZCI6Ik9EZ3hOemg4UVZCTmZFRlFVRXhKUTBGVVNVOU9mRFk1TVRNM05EUTROQSIsInNlbGVjdGVkU2VyaWVzIjoiZjAzYjNmNzY5OTQ0MjlmOTFhYWQ4MDBkNTEwZTU5MDM5OWNjMzNhMSIsImRyaWxsZG93biI6eyJ0cmFuc2FjdGlvbk5hbWUiOiJPdGhlclRyYW5zYWN0aW9uL0NlbGVyeS9jbXMuZGphbmdvYXBwcy5jb250ZW50c3RvcmUudGFza3MudXBkYXRlX291dGxpbmVfZnJvbV9tb2R1bGVzdG9yZV90YXNrIn19&cards[0]=eyJuZXJkbGV0SWQiOiJhcG0tbmVyZGxldHMudHJhbnNhY3Rpb24tdHJhY2UiLCJ0cmFjZUlkIjoiY2I2OGNkMDktZWI5Yi0xMWViLWJkY2QtMDI0MmFjMTEwMDBlXzI1MjU0MV8zMTgzMTMiLCJlbnRpdHlHdWlkIjoiT0RneE56aDhRVkJOZkVGUVVFeEpRMEZVU1U5T2ZEWTVNVE0zTkRRNE5BIn0=&sidebars[0]=eyJuZXJkbGV0SWQiOiJucjEtY29yZS5hY3Rpb25zIiwic2VsZWN0ZWROZXJkbGV0Ijp7Im5lcmRsZXRJZCI6ImFwbS1uZXJkbGV0cy50cmFuc2FjdGlvbnMifSwiZW50aXR5R3VpZCI6Ik9EZ3hOemg4UVZCTmZFRlFVRXhKUTBGVVNVOU9mRFk1TVRNM05EUTROQSJ9&state=d9946155-ea53-cb11-c1c0-cc873d6c7d39)

Useful information to include:
In theory, this should provide a minor performance boost to authors and learners, and be noticeable in the above function trace, once live on prod. Note, that in several places the cache must be invalidated, as to prevent a stale cache.


Co-authored-by: Braden MacDonald <braden@opencraft.com>
2021-10-27 13:53:10 -04:00
David Ormsbee
5d0132305d Revert "feat: Adds discussions settings for new discusions experience (#28749)"
This reverts commit bb0c03123c.
2021-10-27 10:40:12 -04:00
Kshitij Sobti
bb0c03123c feat: Adds discussions settings for new discusions experience (#28749)
This commit adds new discussions settings for the new discussions experience. These are stored in the course so they can be a part of course import/export flow.
These are also added to the discussions configuraiton API to allow MFEs to update the settings.
The discussions API is currently available via LMS, however that means it cannot save changes to the modulestore. This also adds the API to the studio config so it can now also be accessed from studio and be used to save course settings.
2021-10-27 15:41:27 +05:00
Braden MacDonald
6c85668099 feat: write split modulestore's course indexes to Django/MySQL
Split modulestore persists data in three MongoDB "collections": course_index (list of courses and the current version of each), structure (outline of the courses, and some XBlock fields), and definition (other XBlock fields). While "structure" and "definition" data can get very large, which is one of the reasons MongoDB was chosen for modulestore, the course index data is very small.

This commit starts writing course indexes (active_versions) to both MySQL and Mongo, but continues to read from MongoDB only.

By moving course index data to MySQL / a django model, we get these advantages:
* Full history of changes to the course index data is now preserved
* Includes a django admin view to inspect the list of courses and libraries
* It's much easier to "reset" a corrupted course to a known working state, by using the simple-history revert tools from the django admin.
* The remaining MongoDB collections (structure and definition) are essentially just used as key-value stores of large JSON data structures. This paves the way for future changes that allow migrating courses one at a time from MongoDB to S3, and thus eliminating any use of MongoDB by split modulestore, simplifying the stack.
2021-10-26 10:06:52 -07:00
M. Zulqarnain
ba75bb6569 feat: django codemods changes for common folder (#28775) 2021-10-21 13:52:19 +05:00