Commit Graph

69 Commits

Author SHA1 Message Date
Feanil Patel
8e7144ae2e revert: "test: Update query counts."
This reverts commit c2eabf6cca.

We are changing this from a waffle setting to a django setting so we can
undo this query count bump.
2021-04-02 14:42:55 -04:00
Feanil Patel
c2eabf6cca test: Update query counts.
The new WaffleFlag.is_enabled check in middleware increased query counts
for multiple tests.
2021-03-25 12:14:53 -04:00
Usama Sadiq
afa102e125 refactor: ran pyupgrade on lms/djangoapps/course_api (#26735) 2021-03-09 12:54:42 +05:00
Aarif
a332aac1f6 replaced unittest assertions pytest assertions (#26541) 2021-02-19 12:27:11 +05:00
Jawayria
e9b167e1fd Applied pylint-amnesty to course_api 2021-02-04 13:22:45 +05:00
Diana Huang
afeea75d9e Add changes to support ES7 and edx-search 2.0 back into the tree. 2020-11-17 13:23:13 -05:00
Diana Huang
8d652e6a4b Revert "Merge pull request #25515 from edx/diana/edx-search"
This reverts commit 9714d7412f, reversing
changes made to 5bc7c3996b.
2020-11-13 13:16:16 -05:00
Diana Huang
33d0a18d56 Update edx-platform to use Elasticsearch 7.
This reverts commit b73f755487.
2020-11-13 11:13:01 -05:00
Kyle McCormick
151bd13666 Use full names for common.djangoapps imports; warn when using old style (#25477)
* Generate common/djangoapps import shims for LMS
* Generate common/djangoapps import shims for Studio
* Stop appending project root to sys.path
* Stop appending common/djangoapps to sys.path
* Import from common.djangoapps.course_action_state instead of course_action_state
* Import from common.djangoapps.course_modes instead of course_modes
* Import from common.djangoapps.database_fixups instead of database_fixups
* Import from common.djangoapps.edxmako instead of edxmako
* Import from common.djangoapps.entitlements instead of entitlements
* Import from common.djangoapps.pipline_mako instead of pipeline_mako
* Import from common.djangoapps.static_replace instead of static_replace
* Import from common.djangoapps.student instead of student
* Import from common.djangoapps.terrain instead of terrain
* Import from common.djangoapps.third_party_auth instead of third_party_auth
* Import from common.djangoapps.track instead of track
* Import from common.djangoapps.util instead of util
* Import from common.djangoapps.xblock_django instead of xblock_django
* Add empty common/djangoapps/__init__.py to fix pytest collection
* Fix pylint formatting violations
* Exclude import_shims/ directory tree from linting
2020-11-10 07:02:01 -05:00
Feanil Patel
b73f755487 Revert "Feanil/edx search update (#25406)" (#25412)
This reverts commit 581b20056d.
2020-10-22 15:53:57 -04:00
Feanil Patel
581b20056d Feanil/edx search update (#25406)
OeX_ES-44:  Update edx-search elasticsearch.


* removes redundant doc_types from the lms/cms, replaces courseware
index with doc_types to two indices
* code polishing

* Use a new config variable for ES7 deployment. (#2226)

Co-authored-by: Golub-Sergey <1.golub.sergey.1@gmail.com>
Co-authored-by: Diana Huang <diana.k.huang@gmail.com>
2020-10-22 14:18:19 -04:00
Saleem Latif
43d0a67fa1 Request for field to specify locale in Django enterprise accounts 2020-10-20 17:56:22 +05:00
Troy Sankey
83a0327b92 Also exclude libraries from the list_course_keys API 2020-09-25 11:53:35 -04:00
Troy Sankey
e3cd6d55c4 Bypass calls to has_access() when listing courses
When listing courses for which a user has "staff" level access, bypass
calls to has_access() and use a hopefully more performant and simpler
access pattern that uses only a single ORM query and far less processing
per course.

Caution: this short-circuit implementation does not handle org-level
access grants.
2020-09-25 10:44:03 -04:00
Michael Terry
3030efec78 AA-177: Add masquerading for course home MFE
- Looks at masquerading config for dates, outline, metadata, and
  celebration APIs in course_home_api / courseware_api.
- Consolidates and cleans up places we check whether masquerading
  gives us full access to a course.
2020-06-29 12:37:47 -04:00
adeelehsan
98cc0b24db Fixed pagination.
Before getting data validate the page number
on the current number of pages
2020-03-11 10:04:04 +05:00
Robert Raposa
e2cc7fa348 fix failing tests 2020-02-27 09:00:01 -05:00
Robert Raposa
9fa5443379 add course_ids api
Adds a course_ids api that can filter by user role, since the courses
api could not perform well enough for this, and returned much more
data than we need.

Additionally, adds a LazyPageNumberPagination to provide more accurate
counts in the pagination response when using LazySequence with the
queryset.

BOM-897
2020-01-31 18:02:16 -05:00
Feanil Patel
79d097d3c8 Remove the role parameter from the courses api.
BOM-1228

The api is already not very performant and trying to limit it to only
show courses where you are staff causes the code to iterate over almost
all the courses and times out before it returns any results to the user.

The plan is to build a different api for the thing we need that will
just provide the course IDs for courses where you are staff and sholud
be much faster.
2020-01-29 16:38:19 -05:00
aarif
79bf089406 updated the django-waffle to support django 2.2
made changes to test with previous version of django-waffle

updated the query count to test

testing with version 0.13

testing with version 0.14

testing with version 0.15

added version 0.14

updated the django-waffle version to use 0.18

updated the django-waffle version to use 0.16

updated the query counts to pass tests

ran make upgrade

updated the django-waffle to support django 2.2

made changes to test with previous version of django-waffle

updated the query count to test

testing with version 0.13

testing with version 0.14

testing with version 0.15

added version 0.14

updated the django-waffle version to use 0.18

updated the django-waffle version to use 0.16

updated the query counts to pass tests

removed the pdb statements

ran make upgrade
2020-01-08 14:11:42 +05:00
Feanil Patel
2df8b8226b Merge pull request #22643 from edx/feanil/2to3_asserts
Run `2to3 -f asserts . -w` on edx-platform.
2019-12-30 12:13:42 -05:00
Feanil Patel
9cf2f9f298 Run 2to3 -f future . -w
This will remove imports from __future__ that are no longer needed.

https://docs.python.org/3.5/library/2to3.html#2to3fixer-future
2019-12-30 10:35:30 -05:00
Feanil Patel
70294b0ad2 Run 2to3 -f asserts . -w on edx-platform.
https://docs.python.org/3.5/library/2to3.html#2to3fixer-asserts
2019-12-30 10:22:19 -05:00
amitvadhel
b819024d0e Fixes for encoding (Python 3.5) 2019-09-03 23:08:46 +03:00
Julia Eskew
a2dac21843 Add optional role parameter to course API endpoint. (#21059)
* Add optional role parameter to course API endpoint.

* fixup! Add optional role parameter to course API endpoint.
2019-08-02 12:52:30 -04:00
Stu Young
090b8ed011 incr-285 (#20592)
* run python modernize

* run isort

* quality
2019-05-16 12:42:56 -04:00
Calen Pennington
d23e888316 Revert "Reapply "Improve the performance of access checking for a specific user in CourseViewList by prefetching data""
This reverts commit 6059e21bff.
2019-03-21 14:26:00 -04:00
Calen Pennington
b89ce526ca Revert "Batch queries to fetch all courses to prevent over-fetching during paging operations"
This reverts commit 544c7f7eff.
2019-03-21 14:26:00 -04:00
Calen Pennington
544c7f7eff Batch queries to fetch all courses to prevent over-fetching during paging operations 2019-03-19 10:30:33 -04:00
Calen Pennington
6059e21bff Reapply "Improve the performance of access checking for a specific user in CourseViewList by prefetching data"
This reverts commit d8cd6135a6.
2019-03-18 15:24:40 -04:00
Calen Pennington
d8cd6135a6 Revert "Improve the performance of access checking for a specific user in CourseViewList by prefetching data"
This reverts commit a2a340bcd0.
2019-03-14 10:31:24 -04:00
Calen Pennington
5d3c9b36df Merge pull request #19964 from cpennington/fbe-query-perf
Sort in python and add an index to improve StackedConfigModel.current…
2019-03-11 12:22:59 -04:00
Calen Pennington
a2a340bcd0 Improve the performance of access checking for a specific user in CourseViewList by prefetching data 2019-03-11 10:37:53 -04:00
Matthew Piatetsky
74bcc29417 Remove FEATURE_BASED_ENROLLMENT_GLOBAL_KILL_FLAG 2019-03-07 15:46:33 -05:00
Matthew Piatetsky
49ae6b9d73 Remove CONTENT_TYPE_GATING flag 2019-03-07 15:46:33 -05:00
David Ormsbee
76620e0bf2 Add cookie size monitoring New Relic metrics.
This adds middleware that will create custom parameter metrics in
New Relic to track the size of all the cookies being received for
our domain. The custom fields are "cookies_total_size" and a
separate named parameter for every cookie size, e.g.
"cookies.csrftoken.size".

This is intended to help us track cookie growth and better diagnose
issues where users lose their sessions. It is toggled by the
'request_utils.capture_cookie_sizes' Waffle Flag.
2019-02-22 18:18:41 -05:00
Michael Youngstrom
3221c2b91f Remove lms pytest shards 2019-02-12 11:03:46 -05:00
Calen Pennington
829227ade3 Merge pull request #19663 from cpennington/fbe-improve-course-api-performance
Fbe improve course api performance
2019-02-06 11:01:25 -05:00
Calen Pennington
43d975a92a Select related CourseOverview.image_set when loading the course_api 2019-02-06 10:11:16 -05:00
Calen Pennington
419657d7a5 Prefetch course modes used in has_access calls in course api 2019-02-06 10:11:15 -05:00
Calen Pennington
6d22866e78 Cache org-site lookups in the RequestCache 2019-02-06 10:11:15 -05:00
Calen Pennington
6c596f7c49 Allow courses api to return data incrementally
Prior to this commit, the course api (/api/courses/v1/courses/)
performed all the work necessary to return all courses available
to the user, and then only actually returned on page's worth of those
courses.

With this change, the api now does the work incrementally, computing
only the data needed to fetch the courses up to and including the page
being returned. This still increases approximately linearly as
the page number accessed being increases, but should be more cache-friendly.
One side effect of this is that the max_page reported by pagination
will be an overestimate (it will include pages that are removed due
to a users access restrictions).

This change also changes the sort-order of courses being returned by the
course_api. By sorting by course-id, rather than course-number, we
can sort in the database, rather than in Python, and defer loading data
from the end of the list until it is requested.

REVMI-90
2019-02-06 10:11:15 -05:00
Calen Pennington
9ff9c33f59 Add a test that shows how bad course api query counts are 2019-02-06 10:11:15 -05:00
Matthew Piatetsky
6e81c84d8b fix unicode strings in lms/ part 1 2019-02-05 15:15:02 -05:00
Calen Pennington
805f32a357 Revert "Allow courses api to return data incrementally" 2019-01-25 12:56:57 -05:00
Calen Pennington
2527efd822 Prefetch CourseOverview.image_set when loading the course_api 2019-01-25 10:30:09 -05:00
Calen Pennington
0d273e9506 Prefetch course modes used in has_access calls in course api 2019-01-25 10:30:09 -05:00
Calen Pennington
a842921e1f Cache org-site lookups in the RequestCache 2019-01-25 10:30:09 -05:00
Calen Pennington
a3541d6e46 Allow courses api to return data incrementally
Prior to this commit, the course api (/api/courses/v1/courses/)
performed all the work necessary to return all courses available
to the user, and then only actually returned on page's worth of those
courses.

With this change, the api now does the work incrementally, computing
only the data needed to fetch the courses up to and including the page
being returned. This still increases approximately linearly as
the page number accessed being increases, but should be more cache-friendly.
One side effect of this is that the max_page reported by pagination
will be an overestimate (it will include pages that are removed due
to a users access restrictions).

This change also changes the sort-order of courses being returned by the
course_api. By sorting by course-id, rather than course-number, we
can sort in the database, rather than in Python, and defer loading data
from the end of the list until it is requested.

REVMI-90
2019-01-25 10:30:09 -05:00
Calen Pennington
ee75db2703 Add a test that shows how bad course api query counts are 2019-01-24 22:54:01 -05:00