Edx-arch-experiments

~~~~~~~~~~~~~~~~~~~~
Changed
_______
* Completes code owner monitoring updates, which drops owner theme and finalizes the code owner span tags. See doc and ADR updates for more details.

* The code_owner_theme_2 tag was dropped altogether.
* The temporary suffix (_2) was removed from other span tags.
* The code_owner (formerly code_owner_2) tag no longer includes the theme name.
* The new name for the django setting is CODE_OWNER_TO_PATH_MAPPINGS (formerly CODE_OWNER_MAPPINGS).
* The django setting CODE_OWNER_THEMES was dropped.
* Updates the generate_code_owner_mappings.py script accordingly.

~~~~~~~~~~~~~~~~~~~~
Removed
-------
- Removes CodeOwnerMonitoringMiddleware, in favor of using new signals sent from edx-django-utils's MonitoringSupportMiddleware.

Added
-----
* Adds search script datadog_search.py, for searching Datadog monitors and dashboards.

~~~~~~~~~~~~~~~~~~~~
Added
-----
* Added Datadog monitoring app which adds code owner monitoring. This is the first step in moving code owner code from edx-django-utils to this plugin.

* Adds near duplicate of code owner middleware from edx-django-utils.
* Adds code owner span tags for celery using Datadog span processing of celery.run spans.
* Uses temporary span tags names using ``_2``, like ``code_owner_2``, for rollout and comparison with the original span tags.
* Span tag code_owner_2_module includes the task name, where the original code_owner_module does not. In both cases, the code owner is computed the same, because it is based on a prefix match.

~~~~~~~~~~~~~~~~~~~~
Removed
-------
* Deleted Datadog diagnostics plugin app and middleware, which are no longer in use in edxapp.

~~~~~~~~~~~~~~~~~~~~
Added
-----
* Datadog diagnostics middleware can now attempt to close anomalous spans. Can be enabled via Waffle flag ``datadog.diagnostics.close_anomalous_spans`` (controlled separately from logging feature).

~~~~~~~~~~~~~~~~~~~~
Changed
-------
* Datadog diagnostics now logs ancestor spans when an anomaly is encountered, up to a limit of 10 (controlled by new Django setting ``DATADOG_DIAGNOSTICS_LOG_SPAN_DEPTH``). Spans are logged in full and on separate lines, so this logging is now much more verbose; consider only enabling this logging for short periods. Log format of first line has also changed slightly.