Commit Graph

150 Commits

Author SHA1 Message Date
David Ormsbee
f5b74fcf31 feat: Backfill and Django Admin for Learning Sequence Outline
* Adds the backfill_course_outlines management command to contentstore
* Adds a read-only Django admin interface to learning_sequences for the
  support team and debugging.
* Adds two new functions to the learning_sequences public API:
  key_supports_outlines and get_course_keys_with_outlines

The learning_sequences app isn't supposed to know about contentstore or
modulestore, as it's intended to be extracted out of edx-platform in the
long term. Therefore, the backfill_course_outlines command is in
contentstore, and not learning_sequences.

This work was tracked in TNL-7983, but it also fixes a bug where we were
trying to generate course outlines for libraries (TNL-7981).

All Open edX instances upgrading to Lilac should run the
backfill_course_outlines command as part of their upgrade process.
2021-03-09 09:05:57 -05:00
M. Zulqarnain
da970bf215 BOM-2369 (B): Pyupgrade on contentstore/management (#26758)
* pyupgrade on contentstore/management
2021-03-04 14:31:10 +05:00
usamasadiq
7bbde8f0f5 Applied pylint-amnesty 2021-02-08 13:00:22 +05:00
Kyle McCormick
599d663779 Add option to backfill org data as inactive
Add an `--inactive` option to the
`bulk_add_orgs_and_org_courses` management
command, which causes the backfill to set
`active=False` on all rows created in the
Organization and OrganizationCourse tables.

This will lower the potential data
integrity risk to production systems
such as courses.edx.org.

Upgrade edx-organizations to 6.6.0

TNL-7774
2021-01-06 16:00:41 -05:00
Kyle McCormick
70f910935c Improve backfill_orgs_and_org_courses output formatting (#25880)
The output was previously surfaced via `log.info` calls.
Given that the command is reporting on thousands of records,
that ended up being very difficult to parse.
Instead, print one organization or org-course linkage
per line.

Add example run of command with output to command
class docstring.

TNL-7774
2021-01-06 13:23:19 -05:00
Kyle McCormick
f5134e1201 Fix typo in backfill_orgs_and_org_courses dry-run call (#25859)
Also, add test to confirm fix of typo.

TNL-7774
2020-12-11 11:13:35 -05:00
Kyle McCormick
739e8a1e71 Stringify keys in backfill_orgs_and_org_courses (#25802)
The command was failing when it encountered both Old Mongo
and Split Mongo course keys, as it tried to `sort` the keys,
but Opaque Keys are only comparable if they are of the same
type. The solution is to convert them to strings before sorting.
The edx-organizations code that instruments the backfill knows
to parse them back into CourseKeys.
2020-12-08 09:36:46 -05:00
Kyle McCormick
4dda73d797 [BD-14] Migrate all environments to use database-backed organizations (#25153)
* Install `organizations` app into LMS and Studio non-optionally.
* Add toggle `ORGANIZATIONS_AUTOCREATE` to Studio.
* Remove the `FEATURES["ORGANIZATIONS_APP"]` toggle.
* Use the new `organizations.api.ensure_organization` function to
  either validate or get-or-create organizations, depending
  on the value of `ORGANIZATIONS_AUTOCREATE`,
  when creating course runs and V2 content libraries.
  We'll soon use it for V1 content libraries as well.
* Remove the `util.organizations_helpers` wrapper layer
  that had to exist because `organizations` was an optional app.
* Add `.get_library_keys()` method to the Split modulestore.
* Add Studio management command for backfilling organizations tables
  (`backfill_orgs_and_org_courses`).

For full details, see
https://github.com/edx/edx-organizations/blob/master/docs/decisions/0001-phase-in-db-backed-organizations-to-all.rst

TNL-7646
2020-12-02 13:58:40 -05:00
Kyle McCormick
9f239ffe8f Add reset_course_content Studio management command (#25653)
Given a course key and a split-mongo version GUID,
it resets the course run's draft branch to a specified
version and publishes.

The purpose of this is to allow us to restore overwritten
course content without having to write to Mongo via a
dbshell.

Adds `reset_course_to_version` method to modulestore API.

TNL-7705
2020-11-23 15:14:10 -05:00
Diana Huang
65d0c11591 Catch individual errors instead of failing the entire job for course
indexing.
2020-11-17 13:23:13 -05:00
Kyle McCormick
151bd13666 Use full names for common.djangoapps imports; warn when using old style (#25477)
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
2020-11-10 07:02:01 -05:00
Kyle McCormick
a3deb6e317 Sort imports in cms/
Command: `isort --recursive cms/`
2020-10-19 09:34:31 -04:00
Kyle McCormick
538dbf696b Import CMS code using fully qualified module names
or, using proper the proper Python 3 relative imports
(e.g., 'from .views import x') which are standard and
unambiguous.
2020-10-19 09:34:31 -04:00
Kyle McCormick
7538691484 Patch CMS app code in tests using fully qualified names 2020-10-19 09:34:31 -04:00
mariajgrimaldi
003ccf4f1f changed insert to insert_one (#23908)
I wanted to make a byte-sized contribution but there were no Jira tickets so we decided, thanks to a conversation with @jmbowman through the Open Edx Community #incr (Slack) channel, to collaborate in the elimination of warnings listed in the Warnings Report at https://build.testeng.edx.org/job/edx-platform-python-pipeline-master/warning_5freport_5fall_2ehtml/

This PR contributes to the elimination of deprecation warnings, specifically the one mentioned above and reported in the Warnings Report.

Changed insert to insert_one in the following file:

* cms/djangoapps/contentstore/management/commands/tests/test_cleanup_assets.py

This warning occurs due to deprecation: https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert
2020-05-08 12:04:29 -04:00
Aarif
6ee2089077 fixed warnings for wrong-import-order 2020-04-08 23:43:06 +05:00
Tim McCormack
11fc118402 Remove django-celery from transitive deps; remove unused tasks (#23318)
The video thumbnail and transcript tasks were the only things using chord_task from
edx-celeryutils, which in turn was blocking django-celery removal. But, they're no longer used.

Co-authored-by: Diana Huang <diana.k.huang@gmail.com>
2020-03-05 21:57:12 +00:00
Feanil Patel
046feb0cf0 Merge pull request #22649 from edx/feanil/fix_pep8
Fix all E303 pep8 errors.
2019-12-30 13:32:26 -05:00
Feanil Patel
6e3fe00fff Fix all E303 pep8 errors. 2019-12-30 12:25:38 -05:00
Feanil Patel
2df8b8226b Merge pull request #22643 from edx/feanil/2to3_asserts
Run `2to3 -f asserts . -w` on edx-platform.
2019-12-30 12:13:42 -05:00
Feanil Patel
39d3a7ca17 Fix pep8 issues caused by the fixer. 2019-12-30 11:42:04 -05:00
Feanil Patel
9cf2f9f298 Run 2to3 -f future . -w
This will remove imports from __future__ that are no longer needed.

https://docs.python.org/3.5/library/2to3.html#2to3fixer-future
2019-12-30 10:35:30 -05:00
Feanil Patel
70294b0ad2 Run 2to3 -f asserts . -w on edx-platform.
https://docs.python.org/3.5/library/2to3.html#2to3fixer-asserts
2019-12-30 10:22:19 -05:00
Feanil Patel
2d766e7b5d Skip tests that can't succeed in Python3 + Django 1.11
There is a bug in the underlying management/base code that tries to make
all manageent command output be unicode.  This management command
outputs the binary tar file data and so breaks in python3.  In python2
the code is happy to pass bytes back and forth and in later versions of
django this is fixed.  Howevere it's not possible to get this test to
pass in Python3 and django 1.11
2019-10-08 15:26:52 -04:00
Awais Qureshi
9b39f91599 BOM-529
Python3 fix.
2019-09-30 12:02:54 +05:00
Jeremy Bowman
fc57bf9763 Use bytes with GridFS when not specifying an encoding - BOM-157 2019-09-19 11:17:01 -04:00
Ayub khan
d7d1f9ddaf BOM-455
python3 compatibility
2019-09-11 17:01:25 +05:00
Ayub khan
52f9544b6a BOM-368
python3 compatibility
2019-09-05 12:40:39 +05:00
Ayub khan
f5d97260a5 Fixed all py3 tests with AssertionError: "Error: too few arguments" 2019-08-26 14:19:57 +05:00
Ayub khan
8a95a8e520 BOM-95
assertItemsEqual with six.assertCountEqual
2019-08-21 17:01:40 +05:00
aarif
61800c619f replaced StringIO imports with import from six
replaced cStringIO and StringIO imports with import from six

fixed StringIO imports

fixed StringIO imports

fixed XSS python-wrap warning
2019-08-08 00:09:16 +05:00
Usman Khalid
8dfc8e5fda Convert AboutModule, CourseInfoModule, HtmlModule and StaticTabModule to XBlocks. 2019-07-24 20:19:44 +05:00
Ayub
56170466f1 Merge pull request #20988 from edx/INCR-335
INCR-335 python3 compatibility
2019-07-15 12:29:55 +05:00
Ayub khan
b06fcd27d2 INCR-334 python3 compatibility 2019-07-09 15:43:24 +05:00
Ayub khan
d7daac92e7 INCR-335 python3 compatibility 2019-07-09 14:43:10 +05:00
Zachary Hancock
b6b04a4551 sync courses command (#20946) 2019-07-08 10:45:36 -04:00
Usman Khalid
0b27c62ca6 Convert VideoModule to VideoBlock.
Some deprecated functionality has been removed:

- Reading data field and transforms being applied in the init() method.
- The source field.
- The source_visible attribute.
2019-06-20 22:12:39 +05:00
Nimisha Asthagiri
dfa3728edf Move django_comment_common from common to openedx 2019-05-03 12:10:18 -04:00
Nimisha Asthagiri
ef0e06cc0a Revert "Discussions consolidation" 2019-05-02 14:59:56 -04:00
Nimisha Asthagiri
df962a31b7 Move django_comment_common to openedx/core/djangoapps/discussion_common 2019-05-01 19:15:02 -04:00
Jeremy Bowman
d866fed2f1 Fix import order issues in contentstore 2019-02-14 14:20:47 -05:00
Matthew Piatetsky
7df30938c4 fix unicode strings in cms/ 2019-02-04 13:01:51 -05:00
Qubad786
318937552a Update VAL version and thumbnails scrapper job in accordance with val api 2018-09-25 20:12:10 +05:00
irfanuddinahmad
527edc5c45 Add video thumbnails management command to scrape youtube thumbnails and update them in edxval 2018-09-04 19:07:03 +05:00
Michael Youngstrom
52a8c9703b Maintain ignore file tests without storing themn on platform 2018-08-17 16:05:28 -04:00
Jeremy Bowman
68902a2f3f TE-2689 Remove useless pylint suppressions part 3 2018-08-16 09:57:01 -04:00
Qubad786
2d24482cbf Add batching support 2018-07-20 17:31:47 +05:00
Qubad786
c7c2f808e7 Add a field to transcript migration data model to track number of runs
include command run tracking information in logging
2018-06-25 20:42:23 +05:00
Qubad786
5b49f17dfe migrate video transcripts for both published and draft structure 2018-06-25 16:16:59 +05:00
Qubad786
6f5fa33d97 Fix celery chord issues, improve logging for splunk reports generations. 2018-06-25 16:16:59 +05:00