Commit Graph

66 Commits

Author SHA1 Message Date
Muhammad Soban Javed
bd601cf3a6 Update celery routing for celery 4+ (#25567)
* Update celery routing

- Used routing function instead of class
- Move task queues dictionary to Django settings
- Removed routing_key parameter
- Refactored routing for singleton celery instantiation

Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com>
2020-12-16 13:40:47 +05:00
Kyle McCormick
4dda73d797 [BD-14] Migrate all environments to use database-backed organizations (#25153)
* Install `organizations` app into LMS and Studio non-optionally.
* Add toggle `ORGANIZATIONS_AUTOCREATE` to Studio.
* Remove the `FEATURES["ORGANIZATIONS_APP"]` toggle.
* Use the new `organizations.api.ensure_organization` function to
  either validate or get-or-create organizations, depending
  on the value of `ORGANIZATIONS_AUTOCREATE`,
  when creating course runs and V2 content libraries.
  We'll soon use it for V1 content libraries as well.
* Remove the `util.organizations_helpers` wrapper layer
  that had to exist because `organizations` was an optional app.
* Add `.get_library_keys()` method to the Split modulestore.
* Add Studio management command for backfilling organizations tables
  (`backfill_orgs_and_org_courses`).

For full details, see
https://github.com/edx/edx-organizations/blob/master/docs/decisions/0001-phase-in-db-backed-organizations-to-all.rst

TNL-7646
2020-12-02 13:58:40 -05:00
Robert Raposa
8eef18710d set code_owner for celery tasks
ARCHBOM-1260

Co-authored-by: Tim McCormack <tmccormack@edx.org>
2020-11-17 15:33:33 -05:00
Awais Qureshi
7201edb11d Revert "Update routing config" (#25536)" (#25549)" (#25553)" (#25561)
This reverts commit db4c3b1210.
2020-11-11 00:13:47 +05:00
Awais Qureshi
db4c3b1210 Revert "Revert ""Update routing config" (#25536)" (#25549)" (#25553)
This reverts commit c1fe3c3a93.
2020-11-10 23:23:09 +05:00
Kyle McCormick
151bd13666 Use full names for common.djangoapps imports; warn when using old style (#25477)
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
2020-11-10 07:02:01 -05:00
Muhammad Soban Javed
c1fe3c3a93 Revert ""Update routing config" (#25536)" (#25549)
This reverts commit 39a22734c1.
2020-11-09 23:43:47 +05:00
Awais Qureshi
39a22734c1 "Update routing config" (#25536)
* Revert "Revert "Update routing config"

* Removed settings from lms/celery.py and cms/celery.py

* Moved settings import from top-level to function's scopes

Co-authored-by: Soban Javed <iamsobanjaved@gmail.com>
2020-11-09 19:06:55 +05:00
Muhammad Soban Javed
5a2ea1f954 Revert "Update routing config" 2020-11-06 02:05:48 +05:00
Muhammad Soban Javed
bfa8f71f33 Update routing config #25419 2020-11-06 01:00:29 +05:00
Régis Behmo
4586002956 Import waffle classes from edx_toggles instead of waffle_utils
Those classes were ported to edx_toggles. The imports remain in
waffle_utils.__init__ for backward compatibility.
2020-11-03 19:25:37 +01:00
Soban Javed
3206d9cb9a Update celery routing
- Used routing function istead of class
- Move task queues to Djano settings
- Removed routing_key parameter
2020-11-02 15:03:53 +05:00
Kyle McCormick
a3deb6e317 Sort imports in cms/
Command: `isort --recursive cms/`
2020-10-19 09:34:31 -04:00
Kyle McCormick
538dbf696b Import CMS code using fully qualified module names
or, using proper the proper Python 3 relative imports
(e.g., 'from .views import x') which are standard and
unambiguous.
2020-10-19 09:34:31 -04:00
David Ormsbee
168c5deee8 Exclude CCX Courses from search indexing.
Search indexing is prohibitively slow for large CCX courses, even
taking hours in the case of some particularly large ones with
thousands of blocks. Temporarily disabling this functionality until
it can be made more performant (PSRE-288), so that we're not
blocking the workers from doing more latency-sensitive work.

There is a separate effort to put search indexing in its own set
of workers.
2020-09-28 13:58:13 -04:00
Aarif
98af9ce418 remove useless-supression warnings 2020-05-01 19:42:15 +05:00
Aarif
6ee2089077 fixed warnings for wrong-import-order 2020-04-08 23:43:06 +05:00
Tim McCormack
11fc118402 Remove django-celery from transitive deps; remove unused tasks (#23318)
The video thumbnail and transcript tasks were the only things using chord_task from
edx-celeryutils, which in turn was blocking django-celery removal. But, they're no longer used.

Co-authored-by: Diana Huang <diana.k.huang@gmail.com>
2020-03-05 21:57:12 +00:00
Tim McCormack
95d4acad7c Minimize dependency on django-celery (#23193)
- Reimplement `djcelery.common.respect_language` utility function so we
  can drop the dependency.
- Loosen `celery` pinning to a range, which brings in a fix for a
  possible crash bug

Also, pin `path` to 13.1.0 for now in order to continue supporting py35.
2020-02-26 00:46:36 +00:00
Ahtisham Shahid
7a026b7850 created const for update_search_index 2020-01-17 14:29:30 +05:00
Feanil Patel
9cf2f9f298 Run 2to3 -f future . -w
This will remove imports from __future__ that are no longer needed.

https://docs.python.org/3.5/library/2to3.html#2to3fixer-future
2019-12-30 10:35:30 -05:00
Feanil Patel
75eeee0f89 Correct the output file name for exports. 2019-12-19 12:48:13 -05:00
Feanil Patel
50ab035d49 Make the file name not have a 'b' prefix.
We make sure the temp file name is a string because we combine it with
other string components with path later.
2019-12-18 12:58:34 -05:00
Feanil Patel
ca705a2757 Encoding that was breaking import. 2019-10-07 13:49:25 -04:00
noraiz-anwar
67af4b8eb1 fix course import issue 2019-09-26 18:00:21 +05:00
Awais Qureshi
40d9a683e5 PROD-692
Fixing python3
2019-09-17 17:03:56 +05:00
Ayub khan
da9c71557b -Remove unicode and use six.text_type 2019-09-02 16:14:02 +05:00
arbisoft
84cfb8d1ec BOM-392
Fixing encode/decode issues for python3.
2019-08-22 16:45:25 +05:00
Waheed Ahmed
23bc5af0fc Fix course:org link creation upon rerun.
Upon rerun creation from publisher course:organization link
was not creating, due to which org logo was not appearing on
course certificates.

Fixed by creating course:org link upon rerun.

PROD-125
2019-08-05 13:48:50 +05:00
Feanil Patel
63f8167dfc Remove the task that is no longer being referenced.
Second phase for removing the task. The references to it were removed
earlier as a part of DEPR-41
2019-07-26 16:24:05 -04:00
Feanil Patel
1f7b12a38d Remove references to push notifications.
Push notifications was only ever setup to connect to Parse.com a service
that has been discontinued.  Since we haven't replaced the functionality
for a few years now, remove this dead code.

In cms/templates/js/course_info_update.underscore we're allowing content
to not be escaped because this is by design according to the tests.
Long term there should be a better fix for this but for now this is
intended behavior.
2019-07-24 11:45:46 -04:00
asadazam93
eb2422990c Reverted logs 2019-05-03 16:10:17 +05:00
Awais Jibran
3bfab9fd32 Adding more timing logs as course-rerun is slow for mitx-7.00x 2019-03-22 15:41:57 +05:00
Matthew Piatetsky
7df30938c4 fix unicode strings in cms/ 2019-02-04 13:01:51 -05:00
Diana Huang
6572d99e76 Remove all references to datadog from our code. 2019-01-08 15:41:24 -05:00
irfanuddinahmad
527edc5c45 Add video thumbnails management command to scrape youtube thumbnails and update them in edxval 2018-09-04 19:07:03 +05:00
Jeremy Bowman
68902a2f3f TE-2689 Remove useless pylint suppressions part 3 2018-08-16 09:57:01 -04:00
Qubad786
68e760ff29 Move transcript migrations tasks to low priority queue 2018-07-10 19:00:47 +05:00
Kshitij Sobti
491532c2db Update report generation code to include old state for backwards compatibility
Include report data generated by blocks as columns instead of a json dict
2018-07-09 23:56:51 +05:30
Qubad786
c444a51af8 Resolve transcript migrations issues resulted from running it on stage 2018-06-28 18:47:40 +05:00
Qubad786
c7c2f808e7 Add a field to transcript migration data model to track number of runs
include command run tracking information in logging
2018-06-25 20:42:23 +05:00
Qubad786
5b49f17dfe migrate video transcripts for both published and draft structure 2018-06-25 16:16:59 +05:00
Qubad786
6f5fa33d97 Fix celery chord issues, improve logging for splunk reports generations. 2018-06-25 16:16:59 +05:00
Qubad786
9f820c9e71 Fix tests 2018-06-07 15:57:45 +05:00
Qubad786
981be5db57 Fix migrations of external video component transcripts and beautify/refatoring 2018-06-07 15:57:45 +05:00
irfanuddinahmad
2146d7219c fixed migrate command's aws settings issues 2018-06-07 15:57:45 +05:00
irfanuddinahmad
0195e9720d added admin site settings 2018-04-23 19:47:40 +05:00
irfanuddinahmad
3e426aca91 New command to migrate transcripts to S3 2018-04-12 16:57:23 +05:00
bmedx
46cb89e55d Fixes to CMS failures in Django 1.11 tests 2018-01-23 14:11:06 -05:00
Alex Dusenbery
1602a0e66c EDUCATOR-1817 | Add traceback info on course import/export exceptions. 2017-11-29 10:10:58 -05:00