OAuth docs, including decisions

openedx/core/djangoapps/oauth_dispatch/docs/README.rst

@@ -0,0 +1,102 @@
+OAuth Dispatch App (OAuth2 Provider Interface)
+----------------------------------------------
+
+The OAuth Dispatch app is the topmost interface to `OAuth2`_ provider
+functionality. See decisions_ for its historical journey.
+
+.. _OAuth2: https://tools.ietf.org/html/rfc6749
+.. _decisions: decisions/
+
+Background
+----------
+
+This section provides a few highlights on the code to provide a
+high-level perspective on where different aspects of the OAuth2 flow
+reside. For additional information, see `Open edX Authentication`_.
+
+.. _Open edX Authentication: https://openedx.atlassian.net/wiki/spaces/PLAT/pages/160912480/Open+edX+Authentication
+
+Provider code
+~~~~~~~~~~~~~
+
+* The oauth_dispatch_ app provides the top-most entry points to the OAuth2
+  Provider views.
+
+  * Its `validator module`_ ensures Restricted Applications only receive expired
+    tokens.
+
+  * Its `Access Token View`_ returns JWTs as access tokens when a JWT token_type
+    is requested.
+
+  * It uses an edX custom JwtBuilder_ implementation to create the JWT.
+
+* The JwtBuilder_ uses the pyjwkest_ library for implementation of `JSON Web
+  Signature (JWS)`_ and other crypto to build and sign JWT tokens.
+
+.. _oauth_dispatch: https://github.com/edx/edx-platform/tree/master/openedx/core/djangoapps/oauth_dispatch
+.. _validator module: https://github.com/edx/edx-platform/blob/master/openedx/core/djangoapps/oauth_dispatch/dot_overrides/validators.py
+.. _Access Token View: https://github.com/edx/edx-platform/blob/d21a09828072504bc97a2e05883c1241e3a35da9/openedx/core/djangoapps/oauth_dispatch/views.py#L89
+.. _JwtBuilder: https://github.com/edx/edx-platform/blob/d21a09828072504bc97a2e05883c1241e3a35da9/openedx/core/lib/token_utils.py#L15
+.. _pyjwkest: https://github.com/IdentityPython/pyjwkest
+.. _JSON Web Signature (JWS): https://tools.ietf.org/html/draft-ietf-jose-json-web-signature-41
+
+Clients & REST API Clients code
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* edX services, including LMS, use the edx-rest-api-client_ library
+  to make OAuth2 client requests and REST API calls.
+
+  * Built on top of slumber_, the edx-rest-api-client_ provides
+    a utility to retrieve an access token from the LMS. Its Auth_
+    classes create appropriate HTTP Authorization headers with
+    *Bearer* or *JWT* insertions as needed.
+
+  * It makes use of the PyJWT_ library for cryptographically creating
+    JWT tokens.
+    
+    * **Note:** Creation of JWT tokens in our system should only be done
+      by the OAuth Provider. This will break once we use *asymmetric* signing
+      keys, for which remote services will not have the private keys.
+
+.. _edx-rest-api-client: https://github.com/edx/edx-rest-api-client
+.. _slumber: https://github.com/samgiles/slumber
+.. _Auth: https://github.com/edx/edx-rest-api-client/blob/master/edx_rest_api_client/auth.py
+.. _PyJWT: https://github.com/jpadilla/pyjwt
+
+Authentication by REST endpoints
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Recently created edX REST endpoints use the `Django Rest Framework (DRF)`_.
+  The REST endpoint declares which type(s) of authentication it supports
+  or defaults to the *DEFAULT_AUTHENTICATION_CLASSES* value in DRF's 
+  *REST_FRAMEWORK* Django setting.
+
+* edX REST endpoints that support JWTs as access tokens declare the custom
+  edX JwtAuthentication_ class in its DRF authentication_classes_ scheme.
+
+  * JwtAuthentication_ is implemented in the edx-drf-extensions_ library.
+
+  * JwtAuthentication_ extends the JSONWebTokenAuthentication_ class
+    implemented in the django-rest-framework-jwt_ library.
+
+  * JwtAuthentication_ is used to authenticate an API request only
+    if it is listed in the endpoint's authentication_classes_ and the
+    request's Authorization header specifies "JWT" instead of "Bearer".
+
+  * **Note:** The Credentials service has its own implementation of 
+    JwtAuthentication_ and should be converted to use the common
+    implementation in edx-drf-extensions_.
+
+* **Note:** There is also an auth-backends_ repo that should eventually
+  go away once Open ID Connect is no longer used. The only remaining
+  user of its EdXOpenIdConnect_ class is the edx-analytics-dashboard_.
+
+.. _Django Rest Framework (DRF): https://github.com/encode/django-rest-framework
+.. _JwtAuthentication: https://github.com/edx/edx-drf-extensions/blob/1db9f5e3e5130a1e0f43af2035489b3ed916d245/edx_rest_framework_extensions/authentication.py#L153
+.. _authentication_classes: http://www.django-rest-framework.org/api-guide/authentication/#setting-the-authentication-scheme
+.. _edx-drf-extensions: https://github.com/edx/edx-drf-extensions
+.. _django-rest-framework-jwt: https://github.com/GetBlimp/django-rest-framework-jwt
+.. _JSONWebTokenAuthentication: https://github.com/GetBlimp/django-rest-framework-jwt/blob/0a0bd402ec21fd6b9a5f715d114411836fbb2923/rest_framework_jwt/authentication.py#L71
+.. _auth-backends: https://github.com/edx/auth-backends
+.. _EdXOpenIdConnect: https://github.com/edx/auth-backends/blob/31c944289da0eec7148279d7ada61553dbb61f9e/auth_backends/backends.py#L63
+.. _edx-analytics-dashboard: https://github.com/edx/edx-analytics-dashboard

openedx/core/djangoapps/oauth_dispatch/docs/decisions/0001-record-architecture-decisions.rst

@@ -0,0 +1,32 @@
+1. Record Architecture Decisions
+--------------------------------
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+We would like to keep a historical record on the architectural
+decisions we make with this app as it evolves over time.
+
+Decision
+--------
+
+We will use Architecture Decision Records, as described by 
+Michael Nygard in `Documenting Architecture Decisions`_
+
+.. _Documenting Architecture Decisions: http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions
+
+Consequences
+------------
+
+See Michael Nygard's article, linked above.
+
+References
+----------
+
+* https://resources.sei.cmu.edu/asset_files/Presentation/2017_017_001_497746.pdf
+* https://github.com/npryce/adr-tools/tree/master/doc/adr

openedx/core/djangoapps/oauth_dispatch/docs/decisions/0002-migrate-to-dot.rst

@@ -0,0 +1,55 @@
+2. Migrate to Django OAuth Toolkit
+----------------------------------
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+The edX LMS uses the `Django OAuth Provider (DOP)`_ library in order to support
+the OAuth2_ provider protocol. However, that library is now unsupported and
+deprecated by the Django community. Additionally, the edX mobile apps, which are
+OAuth2 clients of the LMS, need the capability to refresh tokens, which DOP does
+not support for `Public Client`_ types that use `Password Credentials grant`_.
+
+.. _OAuth2: https://tools.ietf.org/html/rfc6749
+.. _Public Client: https://tools.ietf.org/html/rfc6749#section-2.1
+.. _Password Credentials grant: https://tools.ietf.org/html/rfc6749#section-4.3
+.. _Django OAuth Provider (DOP): https://github.com/caffeinehit/django-oauth2-provider
+
+Decision
+--------
+
+Moving forward, we will use the `Django OAuth Toolkit (DOT)`_ library and remove
+our use of DOP. 
+
+.. _Django OAuth Toolkit (DOT): https://github.com/evonove/django-oauth-toolkit
+
+Consequences
+------------
+
+Pluses
+~~~~~~
+
+* The `Django documentation recommends DOT`_ for OAuth 2.0 support, so there
+  should be ample support for it in the community.
+
+* DOT uses the well maintained and recommended OAuthLib_ library for the basic
+  OAuth flow so it has a solid crypto foundation.
+
+* DOT is extensible, including its various polymorphic classes and configurable
+  settings.
+
+* DOT seems to have the basic OAuth2 features that we will need for the
+  foreseeable future, including refresh tokens and scopes.
+
+.. _Django documentation recommends DOT: http://www.django-rest-framework.org/api-guide/authentication/#django-oauth-toolkit
+.. _OAuthLib: https://github.com/idan/oauthlib
+
+Minuses
+~~~~~~~
+
+* We need to remove all usages of DOP before we can remove the library.

openedx/core/djangoapps/oauth_dispatch/docs/decisions/0003-use-jwt-as-oauth-tokens-remove-openid-connect.rst

@@ -0,0 +1,211 @@
+3. Use JWT as OAuth2 Tokens; Remove OpenID Connect
+--------------------------------------------------
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+The edX system has external OAuth2 client applications, including edX Mobile apps
+and external partner services. In addition, there are multiple edX microservices
+that are OAuth2 Clients of the LMS.
+
+Some of the internal microservice clients require `OpenID Connect`_ features.
+Specifically, they make use of the `ID Token`_ extension to get user profile
+details from the LMS via the OAuth protocol. The ID Token can also be forwarded
+from one microservice to another, allowing the recipient microservice to
+validate the identity of the token's owner without needing to reconnect with a
+centralized LMS.
+
+We have integrated our fork of DOP_ with support for OpenID Connect. So, an
+access_token request with a DOP client::
+
+    curl -X POST -d "client_id=abc&client_secret=def&grant_type=client_credentials" http://localhost:18000/oauth2/access_token/
+
+includes an id_token field::
+
+    {
+        "access_token": <RANDOMLY-GENERATED-ACCESS-TOKEN>,
+        "id_token": <BASE64-ENCODED-ID-TOKEN>,
+        "expires_in": 31535999,
+        "token_type": "Bearer",
+        "scope": "profile openid email permissions"
+    }
+
+where the value of BASE64-ENCODED-ID-TOKEN decodes to::
+
+    {
+        "family_name": "User1",
+        "administrator": false,
+        "sub": "foo",
+        "iss": "http://localhost:18000/oauth2",
+        "user_tracking_id": 1234,
+        "preferred_username": "user1",
+        "name": "User 1",
+        "locale": "en",
+        "given_name": "User 1",
+        "exp": 1516757075,
+        "iat": 1516753475,
+        "email": "user1@edx.org",
+        "aud": "bar"
+    }
+
+However, OpenID Connect is a large standard with many features and is not supported by
+the DOT_ implementation.
+
+.. _OpenID Connect: http://openid.net/connect/
+.. _ID Token: http://openid.net/specs/openid-connect-core-1_0.html#IDToken
+.. _DOP: https://github.com/caffeinehit/django-oauth2-provider
+.. _DOT: https://github.com/evonove/django-oauth-toolkit
+
+Decision
+--------
+
+Remove our dependency on OpenID Connect since we don't really need all its
+features and it isn't supported by DOT. Instead, support `JSON Web Token (JWT)`_,
+which is a simpler standard and integrates well with the OAuth2 protocol.
+
+.. _JSON Web Token (JWT): https://jwt.io/
+
+The simplest approach is to allow OAuth2 clients to request JWT tokens in place
+of randomly generated Bearer tokens. JWT tokens contain user information,
+replacing the need for OpenID's ID Tokens altogether.
+
+JWT Token
+~~~~~~~~~
+
+JWT tokens will be signed but not encrypted. We will not encrypt them as we
+want the requesting Application and relying parties to be able to parse the
+JWT for relevant information (like the user's name, etc).
+
+The edX Authorization server (LMS) will selectively include data in the
+JWT based on requested scopes (by the Application) and authorized scopes (by
+the user). For example:
+
++--------------------------------+--------------------------+--------------------------------------------+ 
+| Application requests Scope     | User authorizes Scope    | Authzn server (LMS) includes in JWT Payload|
++================================+==========================+============================================+
+| none                           | n/a                      | - *preferred_username*: user's username    |
+|                                |                          | - *sub*: user's anonymous id               |
++--------------------------------+--------------------------+--------------------------------------------+ 
+| **'email'**                    | **'email'**              | - *email*: user's email address            |
++--------------------------------+--------------------------+--------------------------------------------+ 
+| **'profile'**                  | **'profile'**            | - *name*: user's name in their edX profile |
+|                                |                          | - *family_name*: user's last name          |
+|                                |                          | - *given_name*: user's first name          |
+|                                |                          | - *administrator*: whether user is_staff   |
++--------------------------------+--------------------------+--------------------------------------------+ 
+| **'profile'**                  | user does not authorize  | - profile data not provided                |
++--------------------------------+--------------------------+--------------------------------------------+ 
+
+JWT Authentication Library
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Use the open source `Django Rest Framework JWT library`_ as the backend
+implementation for JWT token type authentication.
+
+.. _Django Rest Framework JWT library: https://getblimp.github.io/django-rest-framework-jwt/
+
+Requesting JWT Tokens
+~~~~~~~~~~~~~~~~~~~~~
+
+An OAuth2 client requesting a JWT token_type::
+
+    curl -X POST -d "client_id=abc&client_secret=def&grant_type=client_credentials&token_type=jwt" http://localhost:18000/oauth2/access_token/
+
+would now receive::
+
+    {
+        "access_token": <BASE64-ENCODED-JWT>,
+        "token_type": "JWT",
+        "expires_in": 31535999,
+        "scope": "read write profile email"
+    }
+
+where the value of BASE64-ENCODED-JWT decodes to what the BASE64-ENCODED-ID-TOKEN
+decodes to. There would no longer be a separate id_token field, but the
+access_token will now contain the data that would have been in the id_token.
+
+**Note:** In order to use the JWT token type to access an API, the Authorization
+header needs to specify "JWT" instead of "Bearer"::
+
+    curl -H "Authorization: JWT <BASE64-ENCODED-JWT>" http://localhost:18000/api/user/v1/me
+
+Requesting Bearer Tokens
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+OAuth2 Clients that are not interested in receiving JWT tokens may continue to
+use the default Bearer token type::
+
+    curl -X POST -d "client_id=abc&client_secret=def&grant_type=client_credentials" http://localhost:18000/oauth2/access_token/
+
+which returns::
+
+    {
+        "access_token": <RANDOMLY-GENERATED-ACCESS-TOKEN>,
+        "token_type": "Bearer",
+        "expires_in": 36000,
+        "scope": "read write profile email"
+    }
+
+**Note:** In order to use the Bearer token type to access an API, the Authorization
+header needs to specify "Bearer"::
+
+    curl -H "Authorization: Bearer <RANDOMLY-GENERATED-ACCESS-TOKEN>" http://localhost:18000/api/user/v1/me
+
+Alternatives
+------------
+
+Our implementation of OAuth2+JWT should not be confused with the `IETF standard for
+OAuth JWT Assertions`_, which is for a different purpose entirely. It uses JWTs as
+a replacement for an assertion_ in the OAuth handshake. That is, it uses the JWT
+as a means to *get an OAuth token* (instead of using traditional `OAuth2 grant
+types`_, which require *client-secrets* or *passwords*). 
+
+Our implementation, however, returns a JWT *in place of an OAuth token*. The
+Authorization server (LMS) creates/signs a JWT that binds information about the
+requesting application and the authorizing user. This self-contained token can
+then be validated/used by any relying party (microservice/API) for granting access.
+
+If we did eventually support the `IETF standard for OAuth JWT Assertions`_, a client
+Application would not send its *client secret* over-the-wire when requesting OAuth
+Tokens. Instead, it would use the once out-of-band exchanged *client secret* to sign
+its own JWT. This would be a stronger story for authenticating client Application
+requests.
+
+.. _IETF standard for OAuth JWT Assertions: https://tools.ietf.org/html/rfc7523#section-2.1
+.. _assertion: https://tools.ietf.org/html/rfc7521
+.. _OAuth2 grant types: https://tools.ietf.org/html/rfc6749#section-4
+
+Consequences
+------------
+
+Pluses
+~~~~~~
+
+* The long-term design of the system will be simpler by using simpler
+  protocols and frameworks, such as JWT as access tokens.
+
+* OAuth Clients obtain basic identity information within the JWT access
+  token without needing to hit an extra user info endpoint.
+
+* Any microservice can validate the JWT as an assertion without making an
+  extra round trip to the LMS.
+
+* Although there is no RFC or IETF standard for our use of OAuth+JWT, we
+  are using a relatively maintained and used `open source library`_ for our
+  implementation.
+
+.. _open source library: https://getblimp.github.io/django-rest-framework-jwt
+
+Minuses
+~~~~~~~
+
+* Token invalidation and single Logout become more difficult.
+
+* During the transition period, there will be multiple implementations,
+  which may result in confusion and a more complex system. The shorter
+  we keep the transition period, the better.

openedx/core/djangoapps/oauth_dispatch/docs/decisions/0004-oauth-dispatch-as-router.rst

@@ -0,0 +1,54 @@
+4. OAuth Dispatch as Router
+---------------------------
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+Although we decided to transition from DOP to DOT and from OpenID Connect to
+JWT Tokens, we are not able to update all of our clients and microservices
+rapidly. In the meantime, the Mobile team wants to move forward with DOT to
+support refresh tokens for the mobile apps.
+
+Decision
+--------
+
+Start using DOT for new OAuth2 clients and newer versions of the Mobile app,
+while supporting older DOP clients until all clients are updated to using
+DOT.
+
+The OAuth Dispatch app will function as a content-based router to the multiple
+`OAuth2`_ provider implementations that will need to exist within the platform
+during the transition phase. The app will route incoming OAuth2 REST requests
+based on the value of the client_id field in the request. If the client_id
+identifies an Application in DOT, then the request is routed to the DOT library.
+Otherwise, the request is routed to the DOP library.
+
+Once we fully execute the `transition plan from DOP to DOT`_, we will continue
+to use and maintain this app as it will also contain edx-specific customizations
+that we will add over time. At that point, the app will no longer act as a
+router but will retain its proxy functionality to DOT.
+
+.. _OAuth2: https://tools.ietf.org/html/rfc6749
+.. _transition plan from DOP to DOT: https://openedx.atlassian.net/wiki/spaces/OpenDev/pages/327778541/OAuth+2.0+Roadmap
+
+Consequences
+------------
+
+Pluses
+~~~~~~
+
+* The OAuth Dispatch app will provide an intermediary interface to the underlying
+  implementation(s), which shields the rest of the platform from changes in the
+  underlying libraries.
+
+Minuses
+~~~~~~~
+
+* The LMS' security would be impacted if there are security vulnerabilities found
+  in either DOP or DOT. Since DOP is no longer supported, security issues may not
+  be addressed.

openedx/core/djangoapps/oauth_dispatch/docs/decisions/0005-restricted-application-for-SSO.rst

@@ -0,0 +1,69 @@
+5. Restricted Application for SSO
+---------------------------------
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+External edX clients would like to use edX as an Identity Provider and verify
+the edX identity of a user. The OAuth2 protocol's Authorization Code grant type
+already supports this behavior and our OAuth2 clients can use it for this
+purpose. However, we want to issue *scoped* access tokens to external edX 
+clients so end users can limit what API calls the clients make on their behalf.
+
+The OAuth2 standard for controlling the use of access tokens is `Access Token
+Scopes`_. With Scopes, an end user explicitly authorizes the actions that an
+OAuth2 client is allowed to perform with the issued access token. Scopes
+allow us to support SSO and issue access tokens on behalf of users, knowing
+the users have approved the usage of the tokens. 
+
+However, the edX platform has not enabled wider usage of Scopes by API
+endpoints, beyond the basic values of 'email' and 'profile'. It would be a
+major undertaking to update all of our microservices to fully support Scopes,
+while keeping the system up and running.
+
+.. _Access Token Scopes: https://tools.ietf.org/html/rfc6749#section-3.3
+
+Decision
+--------
+
+Implement a new model in oauth_dispatch to selectively designate DOT Applications
+as "Restricted Applications". For those external clients that want SSO capability
+with edX as an Identity Provider, configure them as a "Restricted Application".
+Although these applications can still request access tokens via the usual
+Authorization Code grant protocol, issue them only **expired** access tokens
+so they cannot make unauthorized calls to our API endpoints.
+
+Consequences
+------------
+
+Pluses
+~~~~~~
+
+* It is a minimal effort to introduce a new "Restricted Application" model
+  and update the oauth_dispatch logic to create expired tokens.
+
+* SSO OAuth2 clients can now verify the identity of edX users when they 
+  successfully receive an OAuth2 access token in the Authorization Code grant
+  type handshake.
+
+* If they make the access token request with token_type=jwt, they receive
+  a JSON Web Token (JWT) with basic information about the user's identity,
+  including their username and edX Anonymous ID.
+
+* If they include 'email' scope in their authorization request and the user
+  approves, the JWT will include the user's email address as well.
+
+* If they include 'profile' scope in their authorization request and the user
+  approves, the JWT will include the user's full name and whether they have
+  staff access.
+
+Minuses
+~~~~~~~
+
+* Returning expired tokens adds additional technical debt on top of the
+  debt incurred by not implementing scopes.

openedx/core/djangoapps/oauth_dispatch/docs/decisions/0006-enforce-scopes-in-LMS-APIs.rst

@@ -0,0 +1,209 @@
+6. Enforce Scopes in LMS APIs
+-----------------------------
+
+Status
+------
+
+Proposed
+
+Context
+-------
+
+Although external edX clients, as Restricted Applications, can use edX
+as an Identity Provider, they cannot successfully make any API calls on
+behalf of users. As explained in 0005-restricted-application-for-SSO_,
+edX prevents successful API calls since our API endpoints do not enforce
+OAuth scopes.
+
+For additional background information on the current implementation,
+see the README_.
+
+.. _0005-restricted-application-for-SSO: 0005-restricted-application-for-SSO.rst
+.. _README: ../README.rst
+
+Decisions
+---------
+
+Add support for enforcing OAuth2 scopes by making the following advancements
+simultaneously.
+
+1. Define and configure new OAuth2 Scopes for accessing API resources
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* For now, we will start with an initial set of OAuth2 Scopes based on
+  immediate API needs. See 0007-include-organizations-in-tokens_ for
+  initial examples.
+
+* OAuth2 clients should be frugal about limiting the scopes they request
+  in order to:
+
+  * keep the data payload small
+  * keep the UX of the user approval form reasonable
+  * follow principle of least privilege
+
+2. Add a version number in the OAuth2 token payload
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As a preemptive step, set a version number field (= 1) in the OAuth2 token
+payload.
+
+3. Restricted Applications receive *unexpired* JWTs, signed with a *new key*
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* We will no longer return expired *JWTs as access tokens* to Restricted
+  Applications. We will sign them with a *new key* that is not shared with 
+  unprotected microservices.
+
+* API endpoints that are exposed by other microservices and that
+  support OAuth2 requests are vulnerable to exploitation until
+  they are also updated to enforce scopes.
+
+* We do not want a lock-step deployment across all of our microservices.
+  We want to enable these changes without blocking on updating all 
+  microservices.
+
+* We do not want to issue unexpired *Bearer tokens* to Restricted
+  Applications since they will be accepted by unprotected microservices.
+  There's no way to retroactively inform existing microservices
+  to reject scope-limiting *Bearer tokens*.
+
+* On the other hand, existing unprotected microservices will reject
+  *JWT tokens* signed with new keys that they do not know about. We will
+  make the new keys available to a microservice only after they
+  have been updated to enforce OAuth Scopes.
+
+  * edx_rest_framework_extensions.settings_ supports having a list of
+    JWT_ISSUERS instead of just a single one.
+
+  * The `edx-platform settings`_ will be updated to have a list of
+    JWT_ISSUERS instead of a single JWT_ISSUER in its settings (example_).
+    A separate settings field will keep track of which is the new issuer
+    key that is to be used for signing tokens for Restricted Application.
+
+  * oauth_dispatch.views.AccessTokenView.dispatch_ will be updated to
+    pass the new JWT key to JwtBuilder_, but only if
+
+    * the requested token_type is *"jwt"* and
+    * the access token is associated with a Restricted Application.
+
+  * oauth_dispatch.validators_ will be updated to return *unexpired*
+    JWT tokens for Restricted Applications, but ONLY if:
+
+    * the token_type in the request equals *"jwt"* and
+    * a `feature toggle (switch)`_ named "oauth2.unexpired_restricted_applications" is enabled.
+
+.. _edx_rest_framework_extensions.settings: https://github.com/edx/edx-drf-extensions/blob/1db9f5e3e5130a1e0f43af2035489b3ed916d245/edx_rest_framework_extensions/settings.py#L73
+.. _edx-platform settings: https://github.com/edx/edx-platform/blob/master/lms/envs/docs/README.rst
+.. _example: https://github.com/edx/edx-drf-extensions/blob/1db9f5e3e5130a1e0f43af2035489b3ed916d245/test_settings.py#L51
+.. _oauth_dispatch.views.AccessTokenView.dispatch: https://github.com/edx/edx-platform/blob/d21a09828072504bc97a2e05883c1241e3a35da9/openedx/core/djangoapps/oauth_dispatch/views.py#L100
+.. _oauth_dispatch.validators: https://github.com/edx/edx-platform/blob/master/openedx/core/djangoapps/oauth_dispatch/dot_overrides/validators.py
+
+4. Associate Available Scopes with Applications
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* In order to allow open edX operators to a priori limit the
+  types of access an Application can request, we will allow them
+  to configure Application-specific "available scopes".
+
+* Introduce a new data model that associates available scopes with
+  DOT Applications.
+
+* Introduce a new Scopes Backend that extends DOT's SettingsScopes_
+  backend and overrides the implementation of get_available_scopes_.
+
+* The new backend will query the new data model to retrieve
+  available scopes.
+
+.. _get_available_scopes: https://github.com/evonove/django-oauth-toolkit/blob/2129f32f55cda950ef220c130dc7de55bea29caf/oauth2_provider/scopes.py#L17
+.. _SettingsScopes: https://github.com/evonove/django-oauth-toolkit/blob/2129f32f55cda950ef220c130dc7de55bea29caf/oauth2_provider/scopes.py#L39
+
+5. Associate Available Organizations with Applications
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* See 0007-include-organizations-in-tokens_ for decisions on this.
+
+6. Introduce a new Permission class to enforce scopes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* The new `custom Permission`_ class will extend DOT's TokenHasScope_
+  Permission class.
+
+* The TokenHasScope_ permission allows API endpoints to declare the
+  scopes that they require in a *required_scopes* class variable.
+
+* The permission class will verify that the scopes in the provided JWT
+  are a proper superset of the scopes required by the requested view.
+
+* For now, the permission class will skip this verification if the
+  application is not a Restricted Application or if the token_type
+  was not a JWT token.
+
+  * **Note:** This will be an issue when microservices want to verify
+    scopes. Determining whether an access token is associated with a 
+    Restricted Application is an LMS-specific capability. Given this,
+    we may need to include a field in the token that indicates whether
+    it was issued to a Restricted Application.
+
+* If the scopes verify, the permission class will update the request
+  object with any organization values found in the token in an attribute
+  called *allowed_organizations*. The view can then limit its access
+  and resources by the allowed organizations.
+
+* In order to have higher confidence that we don't inadvertently miss
+  protecting any API endpoints, add the new Permission class to the
+  `REST_FRAMEWORK's DEFAULT_PERMISSION_CLASSES`_ setting.
+
+  * **Note:** Many of our API endpoints currently override this default
+    by setting the *permission_classes* field on their own View or ViewSet.
+    So in addition to setting this default value, we will update all
+    (15 or so) places that include JwtAuthentication_ in their
+    *authentication_classes* field.
+
+* In case of an unexpected failure with this approach in production,
+  use a `feature toggle (switch)`_ named "oauth2.enforce_token_scopes".
+  When the switch is disabled, the new Permission class fails verification
+  of all Restricted Application requests.
+
+* **Note:** We currently have both `function-based Django views`_ and
+  class-based `Django Rest Framework (DRF)`_ views in the platform.
+
+  * Authorization enforcement using Django Permission classes is
+    supported only for DRF views. DRF does provide a `Python decorator`_
+    to add DRF support to function-based views.
+    
+  * Only DRF enhanced views support JWT based authentication in our
+    system. They do so via the DRF-based JwtAuthentication_ class.
+    So we can **safely assume** that all JWT-supporting API endpoints
+    can be protected via DRF's Permission class.
+     
+.. _custom Permission: http://www.django-rest-framework.org/api-guide/permissions/#custom-permissions
+.. _TokenHasScope: https://github.com/evonove/django-oauth-toolkit/blob/50e4df7d97af90439d27a73c5923f2c06a4961f2/oauth2_provider/contrib/rest_framework/permissions.py#L13
+.. _`REST_FRAMEWORK's DEFAULT_PERMISSION_CLASSES`: http://www.django-rest-framework.org/api-guide/permissions/#setting-the-permission-policy
+.. _function-based Django views: https://docs.djangoproject.com/en/2.0/topics/http/views/
+.. _Django Rest Framework (DRF): http://www.django-rest-framework.org/
+.. _Python decorator: http://www.django-rest-framework.org/tutorial/2-requests-and-responses/#wrapping-api-views
+.. _JwtAuthentication: https://github.com/edx/edx-drf-extensions/blob/1db9f5e3e5130a1e0f43af2035489b3ed916d245/edx_rest_framework_extensions/authentication.py#L153
+
+Consequences
+------------
+
+* Putting these changes behind a feature toggle allows us to decouple 
+  release from deployment and disable these changes in the event of
+  unexpected issues. 
+  
+  * Minimizing the places that the feature toggle is checked (at the
+    time of returning unexpired tokens and at the time of validating
+    requests), minimizes the complexity of the code.
+
+* By associating Scopes with DOT Applications and not Restricted 
+  Applications, we can eventually eliminate Restricted Applications
+  altogether. Besides, they were introduced as a temporary concept
+  until Scopes were fully rolled out.
+
+* Microservices will continue to have limited scope support. We are
+  consciously deciding to not address them at this time. When we do,
+  we will also want to simplify and consolidate their OAuth-related
+  logic and code.
+
+.. _feature toggle (switch): https://openedx.atlassian.net/wiki/spaces/OpenDev/pages/40862688/Feature+Flags+and+Settings+on+edx-platform#FeatureFlagsandSettingsonedx-platform-Case1:Decouplingreleasefromdeployment
+.. _0007-include-organizations-in-tokens: 0007-include-organizations-in-tokens.rst

openedx/core/djangoapps/oauth_dispatch/docs/decisions/0007-include-organization-in-tokens.rst

@@ -0,0 +1,189 @@
+7. Include Organization in Tokens
+---------------------------------
+
+Status
+------
+
+Proposed
+
+Context
+-------
+
+Status of Organizational Access to edX APIs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+External edX applications would like to make server-to-server API
+calls via the Client Credentials grant type to access data. However,
+our APIs typically return data only to global staff users who
+effectively have administrative read access to the system. This
+all-or-nothing capability is unsatisfactory to meet the needs of
+edX partner organizations.
+
+Additionally, some organizations create their own web portals for
+their learners, using edX as an identity provider and as the underlying
+LMS. For various reasons (?), they would like to present edX data to
+their learners on their own portal. Currently, they cannot access a
+learner's data using our APIs.
+
+Our API endpoints do not have more flexible capabilities since they
+do not have reliably sufficient information to limit/filter API results
+according to the organizational affiliation of the requesting application.
+
+Organizational Types in the edX System
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the edX system, the 2 most prevalent organizational relationships
+are:
+
+* **Organization as a Content Provider**
+
+  * This is a partner organization that provides content for a course,
+    program, etc. Typically, such an organization will want to access
+    data for all learners enrolled in their courses. They may choose to
+    do so either via
+
+    a. bulk APIs using the *Client Credentials grant type* (e.g., to
+       synchronize their own data in a background process) or 
+
+    b. a user-specific API on behalf of a logged-in user via the 
+       *Authorization grant type* and *edX as the identity provider*
+       (e.g., to display user-specific data on their own portal).
+
+* **Organization as a User Provider**
+
+  * This is an enterprise organization that registers users onto the
+    edX system, typically via an SSO-enabled portal, but with the
+    *organization (not edX) as the identity provider*. Such an
+    organization will also want to access data for all its users.
+    However, it is not an immediate requirement to support data
+    access by this organization type at this time.
+
+Decisions
+---------
+
+In order to allow DOT Applications to access data for their own organization
+without inadvertently or maliciously gaining access to data for other
+organizations, (1) applications need to be linked to their own organizations,
+(2) organization information needs to be cryptographically bound with
+issued tokens, (3) the authorization approval form needs to present the
+organization information and (4) organization limitations need to be
+embedded in the scopes.
+
+1. Associate Available Organizations with DOT Applications
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Create a configurable Application-specific "available organizations"
+  setting, which is akin to Application-specific "available scopes"
+  (as described in 0006-enforce-scopes-in-LMS-APIs_.
+
+* Introduce a new data model that associates available organizations
+  with DOT Applications.
+
+* The new data model will have a Foreign Key to the Organization_ table.
+  It will essentially be a many-to-many relationship between Organizations
+  and DOT Applications.
+
+* The new data model will also have a column for specifying organization
+  type: *content_provider* or *user_provider*. Initially, we will only
+  use *content_provider*.
+
+2. Organization Information in OAuth Tokens
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* The organization associated with the Application will be included
+  in the JWT tokens requested by the Application.
+
+* JwtBuilder_'s *build_token* functionality will be extended to include
+  the organization value in the token's payload. This payload is
+  cryptographically signed and so binds and limits the scopes in the
+  token to the organization.
+
+* Since the organization value is inside the token, any relying party
+  that receives the token (including a microservice) will be able to
+  enforce scopes as limited to the organization.
+
+.. _0006-enforce-scopes-in-LMS-APIs: 0006-enforce-scopes-in-LMS-APIs.rst
+.. _Organization: https://github.com/edx/edx-organizations/blob/fa137881be9b7d330062bc32655a00c68635cfed/organizations/models.py#L14
+.. _JwtBuilder: https://github.com/edx/edx-platform/blob/d3d64970c36f36a96d684571ec5b48ed645618d8/openedx/core/lib/token_utils.py#L15
+
+3. Organization Information in Authorization Approval Form
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When the interstitial authorization approval form is presented to the
+user for granting access to a DOT Application, if the Application is
+associated with an Organization, the Organization value(s) should be
+presented to the user. This will make it clear to the user that the
+granted access is limited to the Organization's affiliations.
+
+4. Embed Organization Limitation Types in Scopes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Since individual API endpoints need to enforce both scopes and their
+  corresponding organization limitations as included in the token, the
+  scopes themselves should indicate whether or not organization limitations
+  apply.
+
+* In the event that we add additional types of organization limits in
+  the token, we would introduce new scopes that enforce the new
+  types of limits.
+
+  * This allows us to introduce new types of limits while being assured
+    that pre-existing API endpoints will remain protected. Since the
+    pre-existing endpoint is unaware of the new scope, it will
+    prevent access until it is updated to support the new type of 
+    organization limit.
+
+Scopes Examples
+---------------
+
+Here is an initial list of scopes that we may support. Notice how some
+enforce organization limits and others don't. When configuring a DOT
+Application, the edX operator decides how much access the application
+is permitted.
+
++-------------------------------+----------------------------------------------------------------+ 
+| Scope                         | Allowed access                                                 |
++===============================+================================================================+
+| certificates:read             | Retrieve any certificate                                       |
++-------------------------------+----------------------------------------------------------------+ 
+| certificates:read:content_org | Retrieve certificates for courses provided by the organization |
++-------------------------------+----------------------------------------------------------------+ 
+| grades:read                   | Retrieve any grade                                             |
++-------------------------------+----------------------------------------------------------------+ 
+| grades:read:content_org       | Retrieve grades for courses provided by the organization       |
++-------------------------------+----------------------------------------------------------------+ 
+| enrollments:read              | Retrieve any enrollment information                            |
++-------------------------------+----------------------------------------------------------------+ 
+| enrollments:read:content_org  | Retrieve enrollments for courses provided by the organization  |
++-------------------------------+----------------------------------------------------------------+ 
+
+**Note:** Each of these scopes can be used in a server-to-server
+API call (via Client Credentials) or an API call on behalf of a
+single user (via Authorization Code).
+
+Consequences
+------------
+
+* By associating Organizations with DOT Applications and not Restricted
+  Applications, we can eventually eliminate Restricted Applications
+  altogether.
+
+* By including the organization value in the token, any relying party
+  that receives the token (including a microservice) will be able to
+  enforce the scopes as limited to the organization.
+
+* Including the organization limitation types in the scope allows for
+  a secure path forward to introduce new types of limitations in the
+  future. It also makes it clearer to API endpoints what needs to be
+  enforced.
+
+References
+----------
+
+* Examples of Scopes in other web systems
+
+  * https://developer.github.com/apps/building-oauth-apps/scopes-for-oauth-apps/
+  * https://developers.google.com/identity/protocols/googlescopes
+  * https://api.slack.com/scopes
+  * https://developer.spotify.com/web-api/using-scopes/
+  * https://developer.atlassian.com/server/hipchat/hipchat-rest-api-scopes/

openedx/core/djangoapps/oauth_dispatch/docs/how_tos/testing_manually.rst

@@ -0,0 +1,124 @@
+Manually Testing OAuth2 Provider implementation
+-----------------------------------------------
+
+This document explains how to manually test the open edX LMS' OAuth2 Provider
+implementation. In order to verify that it correctly implements the
+`OAuth2 standard`_, use a publicly available 3rd party standard OAuth2 client.
+The steps here show how to use `Google's OAuth2 Playground`_ as the client for
+testing the `Authorization Code grant type`_. However, similar steps can be used
+to test other grant types if they are substituted in the appropriate places.
+
+1. Create an OAuth2 (DOT) Application in the LMS.
+
+   i. Create or decide which LMS user will be associated with the OAuth2 application. In production, the user should be a "service user" that is distinct from "LMS end-users" that login to the system.
+
+   ii. Go to http://localhost:18000/admin/oauth2_provider/application/add/ to create a new Application.
+
+   iii. Enter a value for the "Name" field - a documentary value that uniquely describes this OAuth2 Application.
+
+   iv. Enter a value for the "User" field - from Step 1i.
+
+   v. Fill in the following required fields for the Authorization Code grant type:
+
+      - Authorization grant type: Authorization code
+      - Client type: Confidential
+      - Redirect uris: https://developers.google.com/oauthplayground
+
+   vi. The "Client id" and "Client secret" values are automatically randomly generated. You will later need these values in Step 4(iii) below to provide to the OAuth2 client.
+
+   vii. Keep the "Skip authorization" checkbox deselected in order to test the interstitial approval form in the Authorization Code protocol.
+
+   viii. Click Save.
+
+2. Optional. Make the new Application a `Restricted Application`_ if you are testing Restricted Application features.
+
+   i. Go to http://localhost:18000/admin/oauth_dispatch/restrictedapplication/add/
+
+   ii. Find and select the new Application you created in the dropdown.
+
+   iii. Click Save.
+
+3. Create a publicly accessible URL to the LMS if you are testing on devstack. This step is needed to support the redirecting handshake in the Authorization Code protocol from Google's server back to localhost.
+
+   i. Install `localtunnel`_:
+
+        npm install -g localtunnel
+
+   ii. Run localtunnel so it assigns a unique external url that proxies requests to the LMS on localhost:
+
+        lt --port 18000
+
+   iii. Copy the URL displayed in the terminal as you'll need it in Step 4(iii) to provide to the OAuth2 client.
+
+4. Configure Google's OAuth2 Playground
+
+   i. Go to https://developers.google.com/oauthplayground
+
+   ii. Click on the settings wheel on the right to configure the OAuth2 client.
+
+   iii. Enter the following values:
+
+        - OAuth flow: Server-side
+        - OAuth endpoints: custom
+        - Authorization endpoint: <URL_FROM_STEP_3(iii)>/oauth2/authorize/
+        - Token endpoint: <URL_FROM_STEP_3(iii)>/oauth2/access_token/
+        - Access token location: Authorization header w/ Bearer prefix
+        - Access type: Online
+        - Force prompt: Consent screen
+        - OAuth Client ID: <VALUE_FROM_STEP_1(vi)>
+        - OAuth Client secret: <VALUE_FROM_STEP_1(vi)>
+
+        .. image:: ../images/oauth_playground_config.png
+
+   iv. Click "Close".
+
+5. Initiate OAuth2 Authorization Code flow
+
+   i. Go to Step 1 on the left side of the OAuth2 Playground.
+
+   ii. In the "Input your own scopes" box, enter space-delimited requested scopes:
+
+       .. image:: ../images/oauth_playground_scopes.png
+
+   iii. Click the "Authorize APIs" button to initiate the OAuth2 Authorization Code protocol.
+
+   iv. Follow through any interstitial steps that are required. Specifically,
+
+       - If you aren't already logged in as an end-user on the LMS instance, you will be prompted to do so.
+
+       - If that LMS end-user hasn't already approved the requested scopes for the OAuth2 Application, then you will be prompted to do so.
+
+       - Once these interstitials are completed, the LMS will redirect back to the OAuth2 client (playground), assuming the redirect URL was correctly entered in Step 1(v).
+
+       - If successful, the LMS would have responded back to the OAuth2 client with a temporary Authorization Code.
+
+6. Exchange the Authorization Code for an Access Token.
+
+   i. Go to Step 2 on the left side of the OAuth2 Playground. You will notice a random value in the "Authorization code" field, which was returned back by the LMS.
+
+      .. image:: ../images/oauth_playground_step2.png
+
+   ii. Click on the "Exchange authorization code for tokens" button.
+
+   iii. Note: the Authorization Code is temporary and short-lived.
+
+   iv. If successful, the LMS would have responded with a "Refresh token" and an "Access token".
+
+       .. image:: ../images/oauth_playground_tokens.png
+
+7. Call an LMS API with the Access Token.
+
+   i. Go to Step 3 on the left side of the OAuth2 Playground. In the Request URI field, enter any LMS URL that supports OAuth2 authentication. Remember the base URL should be the URL from Step 3(iii).
+
+      .. image:: ../images/oauth_playground_request_uri.png
+
+   ii. Click on the "Send the request" button.
+
+   iii. Verify the LMS response on the right side of the OAuth2 Playground.
+
+
+.. _OAuth2 standard: https://tools.ietf.org/html/rfc6749
+.. _Google's OAuth2 Playground: https://developers.google.com/oauthplayground
+.. _Authorization Code grant type: https://tools.ietf.org/html/rfc6749#section-4.1
+.. _Restricted Application: https://github.com/edx/edx-platform/blob/dd136b457bc8a25892445fc4362ce02838179472/openedx/core/djangoapps/oauth_dispatch/models.py#L12
+.. _localtunnel: https://localtunnel.github.io/www/

openedx/core/djangoapps/oauth_dispatch/dot_overrides/validators.py

@@ -76,8 +76,8 @@ class EdxOAuth2Validator(OAuth2Validator):
 
     def _authenticate(self, username, password):
         """
-        Authenticate the user, allowing the user to identify themself either by
-        username or email
+        Authenticate the user, allowing the user to identify themselves either
+        by username or email
         """
 
         authenticated_user = authenticate(username=username, password=password)
@@ -93,8 +93,11 @@ class EdxOAuth2Validator(OAuth2Validator):
 
     def save_bearer_token(self, token, request, *args, **kwargs):
         """
-        Ensure that access tokens issued via client credentials grant are associated with the owner of the
-        ``Application``.
+        Ensure that access tokens issued via client credentials grant are
+        associated with the owner of the ``Application``.
+
+        Also, update the `expires_in` value in the token response for
+        RestrictedApplications.
         """
         grant_type = request.grant_type
         user = request.user
@@ -120,7 +123,7 @@ class EdxOAuth2Validator(OAuth2Validator):
             utc_now = datetime.utcnow().replace(tzinfo=utc)
             expires_in = (access_token.expires - utc_now).total_seconds()
 
-            # assert that RestriectedApplications only issue expired tokens
+            # assert that RestrictedApplications only issue expired tokens
             # blow up processing if we see otherwise
             assert expires_in < 0