build: Add missing Celery task decorators, and add CI check for it (#33154)

test_root/semgrep/README.rst

@@ -0,0 +1,21 @@
+Semgrep linters
+###############
+
+Linting rules for use with `semgrep`_ during CI checks on PRs.
+
+Status
+******
+
+This is an experimental approach to developing new linting rules. Semgrep provides by-example structural matching that can be easier to write and maintain than procedural code inspecting ASTs. If the approach works out, we can expand our use of Semgrep; if it becomes a problem for some reason, we can switch to adding pylint rules in edx-lint.
+
+Ignoring failures
+*****************
+
+If you need to tell semgrep to ignore a block of code, put a ``# nosemgrep`` comment on or before the first matched line.
+
+Documentation for writing new rules:
+
+- https://semgrep.dev/docs/writing-rules/rule-syntax/
+- https://semgrep.dev/docs/writing-rules/pattern-syntax/
+
+.. _semgrep: https://github.com/returntocorp/semgrep

test_root/semgrep/celery-code-owner.yml

@@ -0,0 +1,104 @@
+rules:
+  - id: celery-missing-code-owner-function
+    # We can't link directly to the howto doc in question because
+    # semgrep has a bug around long lines:
+    # https://github.com/returntocorp/semgrep/issues/8608
+    #
+    # Here's the intended URL, for reference:
+    # https://edx.readthedocs.io/projects/edx-django-utils/en/latest/monitoring/how_tos/add_code_owner_custom_attribute_to_an_ida.html#handling-celery-tasks
+    message: |
+      Celery tasks need to be decorated with `@set_code_owner_attribute`
+      (from the `edx_django_utils.monitoring` module) in order for us
+      to correctly track code-owners for errors and in other monitoring.
+
+      For more information, see the Celery section of "Add Code_Owner
+      Custom Attributes to an IDA" in the Monitoring How-Tos of
+      <https://edx.readthedocs.io/projects/edx-django-utils>.
+    languages:
+      - python
+    patterns:
+      # Find functions with decorators containing the substring "task"
+      # in their name. This might end up with false positives, but
+      # there are a lot of variations on how we decorate Celery tasks.
+
+      # This pattern should match all decorators, whether or not
+      # they're called as a function (both `@foo(...)` and `@foo`)
+      # and whether or not there are other decorators above or below.
+      - pattern-either:
+        - pattern: |
+            @$TASK
+            def $F(...):
+              ...
+        - pattern: |
+            @$TASK(...)
+            def $F(...):
+              ...
+
+      # Restrict the decorators of interest to just ones with "task"
+      # in the name.
+      - metavariable-pattern:
+          metavariable: $TASK
+          patterns:
+            - pattern-regex: >-
+                [^\(]*task(\(|$)
+
+      # Filter out all of the properly annotated functions, leaving
+      # just the ones of interest.
+      - pattern-not: |
+          @set_code_owner_attribute
+          def $F(...):
+            ...
+      # This is an alternative approach that we have needed in rare cases.
+      - pattern-not: |
+          def $F(...):
+            ...
+            set_code_owner_attribute_from_module(...)
+
+    severity: WARNING
+
+  # This is like celery-missing-code-owner-function but for the `run`
+  # method of Task classes.
+  - id: celery-missing-code-owner-class
+    message: |
+      Celery task classes need to decorate their `run` method with
+      `@set_code_owner_attribute` (imported from `edx_django_utils.monitoring`)
+      in order for us to correctly track code-owners for errors and in other
+      monitoring. Alternatively, the `run` method can call
+      `set_code_owner_attribute_from_module`.
+
+      For more information, see the Celery section of "Add Code_Owner
+      Custom Attributes to an IDA" in the Monitoring How-Tos of
+      <https://edx.readthedocs.io/projects/edx-django-utils>.
+    languages:
+      - python
+    patterns:
+      - pattern: |
+          class $C(..., $SUPER, ...):
+            def run(...):
+              ...
+      - metavariable-pattern:
+          metavariable: $SUPER
+          patterns:
+            - pattern-regex: "Task$"
+
+      - pattern-not: |
+          class $C(..., $SUPER, ...):
+
+            @set_code_owner_attribute
+            def run(...):
+              ...
+
+      - pattern-not: |
+          class $C(..., $SUPER, ...):
+
+            @set_code_owner_attribute
+            def run(...):
+              ...
+      - pattern-not: |
+          class $C(..., $SUPER, ...):
+
+            def run(...):
+              ...
+              set_code_owner_attribute_from_module(...)
+
+    severity: WARNING