Ddtrace

Upgrade Notes

- CI Visibility: `DD_CIVISIBILITY_ITR_ENABLED` now defaults to true, and the Datadog API (configured via the Datadog dashboard) now determines whether code coverage and test skipping are enabled.
- CI Visibility: the CI Visibility service is no longer enabled when the initial query to the Datadog test service settings API fails due to a 403 status code.

New Features

- botocore: Adds optional feature to propagate context between producers and consumers for AWS SQS, AWS SNS, and AWS Kinesis via <span class="title-ref">DD_BOTOCORE_PROPAGATION_ENABLED</span> environment variable. Adds optional feature to disable tracing of AWS SQS <span class="title-ref">poll()</span> operation and AWS Kinesis 'get_records()' operation when no data is consumed via <span class="title-ref">DD_BOTOCORE_EMPTY_POLL_ENABLED</span> environment variable.

- tracing: Adds new tag <span class="title-ref">python_main_package</span> containing the name of the main package of the application. profiling: Adds new tag <span class="title-ref">python_main_package</span> containing the name of the main package of the application.

- ASM: API Security schema collection is now officially supported for Django, Flask and FastAPI. It can be enabled in the tracer using environment variable DD_API_SECURITY_ENABLED=true It will only be active when ASM is also enabled.

- elasticsearch: This allows custom tags to be set on Elasticsearch spans via the Pin interface.

- botocore: This introduces tracing support for bedrock-runtime operations.
See [the docs](https://ddtrace.readthedocs.io/en/stable/integrations.html#botocore) for more information.

- datastreams: this change adds kombu auto-instrumentation for datastreams monitoring. tracing: this change adds the `DD_KOMBU_DISTRIBUTED_TRACING` flag (default `True`)

- Vulnerability Management for Code-level (IAST): Add support for CMDi in langchain.

- botocore: Add the ability to inject trace context into the input field of botocore stepfunction start_execution and start_sync_execution calls.

- Removes another place where we always load instrumentation telemetry, even if it is disabled

- tracing: This introduces the ability to disable tracing at runtime based on configuration values sent from the Datadog frontend. Disabling tracing in this way also disables instrumentation telemetry.

- tracing: Adds support for remote configuration of `DD_TRACE_HEADER_TAGS`

- tracing: Add support for remote configuration of trace-logs correlation.

- grpc/grpc_aio: reports the available target host in client spans as `network.destination.ip` if only an IP is available, `peer.hostname` otherwise.

- span: Adds a public api for setting span links

- starlette,fastapi: Trace background tasks using span links

Bug Fixes

- ASM: This fix resolves an issue where an exception would be logged while parsing an empty body JSON request.

- CI Visibility: fixes an issue where coverage data for suites could be lost for long-running test sessions, reducing the possibility of skipping tests when using the Intelligent Test Runner.

- IAST: Don't split AST Assign nodes since it's not needed for propagation to work.

- ASM: This fix resolves an issue where suspicious request blocking on request data was preventing API Security to collect schemas in FastAPI, due to route not being computed.

- ASM: This fix resolves an issue where ASM custom blocking actions with a redirect action could cause the server to drop the response.

- Fixed an incompatible version requirements for one of the internal dependencies that could have caused an exception to be raised at runtime with Python 3.12.

- data_streams: This change fixes a bug leading to lag being reported as 1 offset instead of 0 offsets.

- IAST: fixes import overhead when IAST is disabled.

- Fix an incomplete support for pkg_resouces that could have caused an exception on start-up.

- Fix an issue that caused an exception to be raised when trying to access resource files via `pkg_resources`.

- Fix for an import issue that caused the pytest plugin to fail to properly initialize a test session and exit with an import exception.

- openai: This fixes a bug that prevents logs from being correlated with traces in the Datadog UI.

- langchain: This fixes a bug that prevents logs from being correlated with traces in the Datadog UI.

- openai: This fix resolves an issue where an internal OpenAI method <span class="title-ref">SyncAPIClient.\_process_response</span>
was not being patched correctly and led to to an AttributeError while patching.

- profiling: handle a potential system error that may be raised when running a Celery-based application with CPython 3.11.

- Fixed an issue that could have caused an exception as a result of a concurrent access to some internal value cache.

- tracing: Ensures span links are serialized with the expected traceflag when `DD_TRACE_API_VERSION=v0.4`

- ASM: This fix resolves an issue where IP Headers configured by the user in the environment could not work for frameworks handling requests with case insensitive headers like FastAPI.

- Vulnerability Management for Code-level (IAST): Fixes a bug in the `str` aspect where encoding and errors arguments were not honored correctly.

- Vulnerability Management for Code-level (IAST): Fix an unhandled ValueError in `ast_function` thrown in some cases (i.e. Numpy arrays when converted to bool).

- opentelemetry: Ensures that span links are serialized in a json-compatible representation.

- Pin importlib_metadata to 6.5.0 to avoid its issue 455 (<https://github.com/python/importlib_metadata/issues/455>).

- profiler: Fixes a sigabrt when shutdown occurs during an upload

- otel: Ensures all otel sampling decisions are consistent with Datadog Spans. This prevents otel spans in a distributed trace from being sampled differently than Datadog spans in the same trace.

- tracing: Fix an issue where remote configuration values would not be reverted when unset in the UI.

- tracing: Ensures hostnames are reported in statsd metrics if `DD_TRACE_REPORT_HOSTNAME=True` (default value is `False`).

Other Changes

- setup: pins the default macOS deployment target to 10.14.
- tracing: Updates the default value of `DD_TRACE_PROPAGATION_STYLE` from `tracecontext,datadog` to `datadog,tracecontext`. With this change w3c tracecontext headers will be parsed before datadog headers. This change is backwards compatible and should not affect existing users.

---

New Features

- aiohttp: add <span class="title-ref">split_by_domain</span> config to split service name by domain
- CI Visibility: Adds code coverage lines covered tag for `pytest` and `unittest`.
- aiohttp: Adds http.route tag to `aiohttp.request` spans.
- bottle: Adds http.route tag to `bottle.request` spans.
- falcon: Adds http.route tag to `falcon.request` spans.
- molten: Adds http.route tag to `molten.request` spans.
- Adds distributed tracing for confluent-kafka integration. Distributed tracing connects Kafka consumer spans with Kafka producer spans within the same trace if a message is valid. To enable distributed tracing, set the configuration: `DD_KAFKA_DISTRIBUTED_TRACING_ENABLED=True` for both the consumer and producer service.
- ASM: This introduces (experimental) api security support for fastAPI. Flask and Django were already supported in 2.4.0. Support schema computation on all addresses (requests and responses) and scanner support for pii, credentials and payment data.
- CI Visibility: introduces a CI visibility-specific logger (enabled for the `pytest` plugin), enabled by setting the `DD_CIVISIBILITY_LOG_LEVEL` environment variable (with the same level names as Python logging levels).
- CI Visibility: allows for waiting for the git metadata upload to complete before deciding whether or not to enable coverage (based on API response).
- Further lazy loads telemetry_writer so that it is not running when explicitly disabled. Users must explicitly set "DD_INSTRUMENTATION_TELEMETRY_ENABLED=false".
- tracer: Add support for remotely configuring trace tags.

Bug Fixes

- loguru: Ensures log correlation is enabled when the root logger is initialized. Previously, log correlation was only enabled when a new sink was added.
- Fix compatibility with other tools that try to infer the type of a Python object at runtime.
- tracing: Fixes a bug that prevents span links from being visualized in the Datadog UI.
- tracing: Resolves span encoding errors raised when span links do not contain expected types
- ASM: This fix resolves an issue where custom event boolean properties were not reported as <span class="title-ref">true</span> and <span class="title-ref">false</span> like other tracers but as <span class="title-ref">True</span> and <span class="title-ref">False</span>.
- Vulnerability Management for Code-level (IAST): Ensure that Cookies vulnerabilities report only the cookie name.
- langchain: This fix resolves an `get_openai_token_cost_for_model` import error in langhcain version 0.0.351 or later.
- ASM: This fix resolves an issue where IAST could cause circular dependency at startup.
- tracing: Ensures all fields in `ddtrace.context.Context` are picklable.
- pytest: This fix resolves an issue where the <span class="title-ref">--no-cov</span> flag did not take precedence over the <span class="title-ref">--cov</span> flag when deciding whether to report code coverage on spans.
- rq: Fixed a bug where the RQ integration would emit a warning when setting `job.status` span tag.

---

Upgrade Notes

- <div id="remove-unsupported-pylons">

This removes the `pylons` integration, which does not support Python 3.

</div>

Deprecation Notes

- aioredis: The aioredis integration is deprecated and will be removed in a future version. As an alternative to the aioredis integration, you can use the redis integration with redis\>=4.2.0.

New Features

- ASM: dependency telemetry metrics now will only report dependencies actually in use (imported) and will also report new imported modules periodically.

- ASM: This introduces Threat Monitoring and Blocking on FastAPI.
- IP Blocking and all input addresses are supported on requests and responses

\- Custom Blocking This does not contain user blocking specific features yet.

- tracing: Introduces support for OpenTracing Baggage Items with HTTP Propagation. Enable this support by `DD_TRACE_PROPAGATION_HTTP_BAGGAGE_ENABLED=true`. The `Context._set_baggage_item` and `Context._get_baggage_item` internal methods are provided for manual modifications to the Baggage Items. These API changes are subject to change.

- dynamic instrumentation: Add support for more built-in container types, such as `defaultdict`, `frozenset`, `OrderedDict` and `Counter`.

- Vulnerability Management for Code-level (IAST): Adds Python 3.12 compatibility

- Optionally lazy loads and disables Instrumentation Telemetry. Users must explicitly set "DD_INSTRUMENTATION_TELEMETRY_ENABLED=false".

- tracer: Add support for remotely setting the trace sample rate from the Datadog UI. This functionality is enabled by default when using `ddtrace-run` and library injection. To enable it when using the library manually, use `ddtrace.config.enable_remote_config()`.

Bug Fixes

- tracer: tag spans that have been sampled due to an Agent sampling configuration.
- lambda: This change disables the use of `multiprocessing.queue` in Lambda, because it is not supported in Lambda
- langchain: This fix resolves a crash that could occur during embedding when no embeddings are found.
- Fix a regression with the support for gevent that could have occurred if some products, like ASM, telemetry, were enabled.
- kafka: Resolves `TypeError` raised by serializing producers and deserializing consumers when the `message.key` tag is set on spans.
- dynamic instrumentation: Fix an issue that caused the instrumented application to fail to start if a non-standard module was imported.
- openai: This fix resolves an issue where tagging image inputs in the chat completions endpoint resulted in attribute errors.
- openai: This fix resolves an issue where requesting raw API responses from openai\>=1.0 resulted in attribute errors while tagging.
- profiling: Fix an issue that prevented threading locks from being traced when using gevent.
- profiling: Fix a segmentation fault with CPython 3.12 when sampling thread stacks.
- pylibmc: Fixes an issue where using `ddtrace-run` or `ddtrace.patch_all()` with `DD_TRACE_ENABLED=False` would break with get, gets, and get_multi operations on pylibmc Clients.
- tracing: This fix resolves an issue where concurrent mutations to the `context._meta` dict caused <span class="title-ref">RuntimeError: dictionary changed size during iteration</span>.
- django: Resolves `AttributeError` raised by traced `StreamingHttpResponse`.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where certain aspects incorrectly expected at least one argument, leading to an IndexError when none were provided. The solution removes this constraint and incorporates regression tests for stability assurance.
- Vulnerability Management for Code-level (IAST): Cookies vulnerabilities are only reported if response cookies are insecure.
- Vulnerability Management for Code-level (IAST): Fix propagation error on `.format` string method.
- requests: Updates the resource names of `requests.requests` spans to include the method and path of the request.
- propagation: This fix resolves an issue where a `Context` generated from extracted headers could lack a span_id or trace_id, leading `SpanLink` encoding errors.
- psycopg: This fix resolves an issue where a circular import of the psycopg library could cause a crash during monkeypatching.
- psycopg: This fix resolves an issue where exceptions originating from asynchronous Psycopg cursors were not propagated up the call stack.
- redis: This fix resolves an issue where the yaaredis and aredis integrations imported code from the redis integration, causing a circular import error.
- tracing: Resolves trace encoding errors raised when `DD_TRACE_API_VERSION` is set to `v0.5` and a BufferFull Exception is raised by the TraceWriter. This fix ensures span fields are not overwritten and reduces the frequency of 4XX errors in the trace agent.

Other Changes

- tracing: Upgrades the trace encoding format to v0.5. This change improves the performance of encoding and sending spans.

---

New Features

- propagation: 128-bit trace ids are now used by default for propagation. Previously the default was 64-bit. This change is backwards compatible with tracers that still use 64-bit trace ids and should not cause any breaking behavior.
- Adds DSM `pathway.hash` tag to spans when DSM is enabled. This allows traces from the instrumented service to show up in the DSM traces tab.
- propagation: When the tracer is configured to extract and inject `tracecontext`, the tracer will propagate the tracestate values from other vendors so long as the traceparent trace-id matches the first found trace context, regardless of propagator configuration order. To disable this behavior `DD_TRACE_PROPAGATION_EXTRACT_FIRST=true` can be set.
- opentelemetry: Map reserved OpenTelemetry attributes to Datadog span model.
- opentelemetry: datadog operation name from semantic conventions
- propagation: If a valid context is extracted from headers, and the following extracted trace context's `trace_id`s do not match the valid context's, then add a span link to the root span to represent the broken propagation.
- tracing: This change treats spans that terminated with `sys.exit(0)` as successful non-error spans.
- tracing: This introduces the `DD_TRACE_SPAN_TRACEBACK_MAX_SIZE` environment variable, allowing the maximum size of tracebacks included on spans to be configured.

Bug Fixes

- CI Visibility: fixes the fact that the GITHUB_SERVER_URL environment variable was not being sanitized for credentials
- dynamic instrumentation: Needs to update the pubsub instance when the application forks because the probe mechanism should run in the child process. For that, DI needs the callback as the method of an instance of Debugger, which lives in the child process.
- CI Visibility: Fixes an issue where a `ValueError` was raised when using different path drives on Windows
- Fixes an issue where ddtrace could not be installed from source when using `setuptools>=69` due to a change in the license field.
- tracing: Fixes an issue where the thread responsible for sending traces is killed due to concurrent dictionary modification.
- structlog: Fixes `TypeError` raised when ddtrace log processor is configured with a tuple
- Vulnerability Management for Code-level (IAST): Generates cookies vulnerabilities report if IAST is enabled. Before this fix, Cookies vulnerabilities were only generated if both IAST and Appsec were enabled.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where, at AST patching to replace code with IAST aspects, passing the original function/method as an extra parameter for accurate patching unintentionally triggers side effects in methods obtained from an expression (like `decode` in `file.read(n).decode()`), resulting in unexpected multiple calls to the expression (`file.read(n)` in the example).
- Vulnerability Management for Code-level (IAST): This fix eliminates some reference leaks and C-API usage when IAST reports a vulnerability and calls `get_info_frame`.
- kafka: This fix resolves an issue where calls to `confluent_kafka`'s `produce` method with `key=None` would cause an exception to be raised.
- tracing: This fix resolves an issue where ddtrace's signal handlers could cause Flask apps not to respond correctly to SIGINT.
- logging: A log handler is automatically added to the ddtrace logger upon ddtrace import, when not using ddtrace-run. This can lead to duplicate logging if users add additional loggers and do not explicitly modify the ddtrace logger. This fix adds a feature flag that can be used to toggle this behavior off `DD_TRACE_LOG_STREAM_HANDLER` which defaults to `true`.

---

Upgrade Notes

- The `wrapt` and `psutil` packages are vendored to help users avoid building these packages if wheels were not available for a given platform. This reverses a change released in v2.0.0.

New Features

- CI Visibility: adds ITR support for `unittest`
- CI Visibility: adds start/end line support for `pytest` test spans
- CI Visibility: adds start/end line source file data to `unittest` test spans
- aiohttp: This introduces basic tracing of streaming responses that stay open long after the <span class="title-ref">on_prepare</span> signal has been sent.
- CI Visibility: introduce pytest hooks for modifying the module, suite, and test naming logic
- CI Visibility: add support for AWS Codepipeline to CI env var gathering
- datastreams: this change adds message payload size metrics and aggregations for Kafka.
- structlog: Wraps get_logger function in order to add datadog injection processor regardless of configuration
- openai: This adds support for openai v1.
- Source Code: filters Git repo URLs from env vars and setuptools
- logbook: This introduces log correlation for the logbook library. Refer to `logbook-docs <ddtrace.contrib.logbook>` for more details.
- loguru: This introduces log correlation for the loguru library. Refer to `loguru-docs <ddtrace.contrib.loguru>` for more details.
- openai: This adds support for tagging function call arguments when using OpenAI's function calling feature.
- Adds ARM64 support for Single-Step instrumentation
- structlog: This introduces log correlation for the structlog library. Refer to `structlog-docs <ddtrace.contrib.structlog>` for more details.
- celery: Adds Python 3.11 and 3.12 support for the celery integration.

Known Issues

- ASM: fix a body read problem on some corner case where passing empty content length makes wsgi.input.read() blocks.

Bug Fixes

- Application Security Management (ASM): fix a body read error when `Transfer-Encoding: chunked` header is sent
- CI Visibility: fixes an issue where class-based test methods with the same name across classes would be considered duplicates, and cause one (or more) tests to be dropped from results, by adding `--ddtrace-include-class-name` as an optional flag (defaulting to false) to prepend the class name to the test name.
- CI Visibility: fixes a crash where the unittest integration would try to enable coverage when tests are run even if the Intelligent Test Runner is not enabled.
- data_streams: This fix resolves an issue where tracing would crash if a kafka client produced a message with no key or value.
- CI: fixes an issue which prevented the library from filtering user credentials for SSH Git repository URLs
- dynamic instrumentation: fix an issue that caused function probes on the same module to fail to instrument and be reported in the `ERROR` status in the UI if the module was not yet imported.
- Use a unique default service name across all the products provided by the library when one is not given via the configuration interface.
- sampling: This fix reverts a refactor which affected how the tracer handled the trace-agent's recommended trace sampling rates, leading to an unintended increase in traces sampled.
- tracing: Fixes a msgpack import error when `DD_TRACE_API` is set to `v0.5`
- fix(profiling): numeric type exception in memalloc When pushing allocation samples, an exception was being thrown due to a float being passed instead of an integer. We now cast the ceiled value to an integer.
- CI Visibility: fixes `unittest` data not being initialized properly
- CI Visibility: fixes an issue where just importing <span class="title-ref">unittest</span> enabled CIVisibility and potentially caused unexpected logs and API requests
- Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where custom functions or methods could be replaced by aspects with differing argument numbers, causing runtime errors as a result. Furthermore, it addresses a case during patching where the module is inadvertently passed as the first argument to the aspect.
- Vulnerability Management for Code-level (IAST): Fix potential string id collisions that could cause false positives with non tainted objects being marked as tainted.
- IAST: This fix resolves an issue where JSON encoder would throw an exception while encoding a tainted dict or list.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where SimpleJSON encoder would throw an exception while encoding a tainted dict or list.
- ASM: add support for psycopg2 adapt mechanism to LazyTaintList, preventing a ProgrammingError when using psycopg2 with IAST.
- tracing: This fix resolves an issue where unserializable tracer attributes caused crashes when `DD_TRACE_DEBUG` was set.
- This fix resolves an issue where `confluent_kafka`'s `SerializingProducer` and `DeserializingConsumer` classes were incorrectly patched, causing crashes when these classes are in use with Datadog patching.
- langchain: This fix resolves an issue with tagging pydantic <span class="title-ref">SecretStr</span> type api keys.
- lib injection: Fix permissions error raised when non-root users copy single step instrumentation files.
- redis: The Datadog Agent removes command arguments from the resource name. However there are cases, like compressed keys, where this obfuscation cannot correctly remove command arguments. To safeguard that situation, the resource name set by the tracer will only be the command (e.g. SET) with no arguments. To retain the previous behavior and keep arguments in the span resource, with the potential risk of some command arguments not being fully obfuscated, set `DD_REDIS_RESOURCE_ONLY_COMMAND=false`.

Other Changes

- tags: Previously `DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH` had a max limit setting of 512 characters. This change removes that limit but keeps the default at 512.

---

Prelude

The Datadog APM Python team is happy to announce the release of v2.0.0 of ddtrace. This release drops support for Python 2.7, 3.5, and 3.6. This release adds support for Python 3.12.

<div class="important">

<div class="title">

Important

</div>

If you are on version of Python not supported by v2, we will continue to maintain the ddtrace v1 with bug fixes.

</div>

<div class="note">

<div class="title">

Note

</div>

Before upgrading to v2.0.0, we recommend users install `ddtrace~=1.20.0` and enable deprecation warnings. All removals to the library interface and environment variables in v2 were deprecated in the 1.x release line.

</div>

<div class="note">

<div class="title">

Note

</div>

The changes to environment variables apply only to the configuration of the ddtrace library and not the Datadog Agent.

</div>

Upgrading summary

Functionality changes

The default logging configuration functionality of ddtrace has been changed to avoid conflicting with application logging configurations. `DD_CALL_BASIC_CONFIG` has been removed and the ddtrace logger will log to stdout by default, or a log file as specified using `DD_TRACE_LOG_FILE`.

Setting the environment variable `DD_TRACE_PROPAGATION_STYLE='b3'`, which previously enabled `b3multi` now enables `b3 single header`. `b3 single header` still works but is deprecated for `b3`. Simplified: `b3` used to enable `b3multi`, but now enables `b3 single header` to better align with Opentelemetry's terms.

Removed deprecated environment variables

These environment variables have been removed. In all cases the same functionality is provided by other environment variables and replacements are provided as recommended actions for upgrading.

| Variable | Replacement | Note |
|--------------------------------------------|--------------------------------------------|-----------------------------------------------------|
| `DD_GEVENT_PATCH_ALL` | None | `📝<remove-dd-gevent-patch-all>` |
| `DD_AWS_TAG_ALL_PARAMS` | None | `📝<remove-aws-tag-all-params>` |
| `DD_REMOTECONFIG_POLL_SECONDS` | `DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS` | `📝<rename-remote-config-poll-seconds>` |
| `DD_CALL_BASIC_CONFIG` | None | `📝<remove-basic-config>` |
| `DD_TRACE_OBFUSCATION_QUERY_STRING_PATERN` | `DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP` | `📝<remove-trace-obfuscation-query-string-pattern>` |

Removed deprecated library interfaces

These methods and module attributes have been removed. Where the same functionality is provided by a different public method or module attribute, a recommended action is provided for upgrading. In a few limited cases, because the interface was no longer used or had been moved to the internal interface, it was removed and so no action is provided for upgrading.

| Module | Method/Attribute | Note |
|-----------------------------------|---------------------------------|----------------------------------------------------|
| `ddtrace.constants` | `APPSEC_ENABLED` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_JSON` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_VERSION` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_ERRORS` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_LOADED` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_EVENT_RULE_ERROR_COUNT` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_DURATION` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_DURATION_EXT` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_TIMEOUTS` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_WAF_VERSION` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_ORIGIN_VALUE` | `📝<remove-appsec-private-constants>` |
| | `APPSEC_BLOCKED` | `📝<remove-appsec-private-constants>` |
| | `IAST_JSON` | `📝<remove-appsec-private-constants>` |
| | `IAST_ENABLED` | `📝<remove-appsec-private-constants>` |
| | `IAST_CONTEXT_KEY` | `📝<remove-appsec-private-constants>` |
| `ddtrace.contrib.fastapi.patch` | `span_modifier` | `📝<remove-fastapi-starlette-span-modifier>` |
| | `aggregate_resources` | `📝<remove-fastapi-starlette-aggregate-resources>` |
| `ddtrace.contrib.starlette.patch` | `span_modifier` | `📝<remove-fastapi-starlette-span-modifier>` |
| | `aggregate_resources` | `📝<remove-fastapi-starlette-aggregate-resources>` |
| | `get_resource` | `📝<remove-fastapi-starlette-span-modifier>` |
| `ddtrace.contrib.grpc.constants` | `GRPC_PORT_KEY` | `📝<remove-grpc-port-key>` |
| `ddtrace.ext.cassandra` | `ROW_COUNT` | `📝<remove-cassandra-row-count>` |
| `ddtrace.ext.mongo` | `ROWS` | `📝<remove-mongo-row-count>` |
| `ddtrace.ext.sql` | `ROWS` | `📝<remove-sql-row-count>` |
| `ddtrace.filters` | `TraceCiVisibilityFilter` | `📝<remove-trace-ci-visibility-filter>` |
| `ddtrace.tracer` | `DD_LOG_FORMAT` | `📝<remove-dd-log-format>` |

Upgrade Notes

- <div id="remove-dd-gevent-patch-all">

`DD_GEVENT_PATCH_ALL` is removed. There is no special configuration necessary to make ddtrace work with gevent if using ddtrace-run.

</div>

- <div id="remove-aws-tag-all-params">

`DD_AWS_TAG_ALL_PARAMS` is removed. The boto/botocore/aiobotocore integrations no longer collect all API parameters by default.

</div>

- <div id="rename-remote-config-poll-seconds">

`DD_REMOTECONFIG_POLL_SECONDS` is removed. Use the environment variable `DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS` instead.

</div>

- <div id="remove-appsec-private-constants">

`APPSEC_ENABLED`, `APPSEC_JSON`, `APPSEC_EVENT_RULE_VERSION`, `APPSEC_EVENT_RULE_ERRORS`, `APPSEC_EVENT_RULE_LOADED`, `APPSEC_EVENT_RULE_ERROR_COUNT`, `APPSEC_WAF_DURATION`, `APPSEC_WAF_DURATION_EXT`, `APPSEC_WAF_TIMEOUTS`, `APPSEC_WAF_VERSION`, `APPSEC_ORIGIN_VALUE`, `APPSEC_BLOCKED`, `IAST_JSON`, `IAST_ENABLED`, `IAST_CONTEXT_KEY` are removed. This should not affect existing code as these deprecated ASM constants were meant for private use only.

</div>

- <div id="remove-fastapi-starlette-span-modifier">

`ddtrace.contrib.starlette.get_resource`, `ddtrace.contrib.starlette.span_modifier`, and `ddtrace.contrib.fastapi.span_modifier` are removed. The starlette and fastapi integrations now provide the full route and not just the mounted route for sub-applications.

</div>

- <div id="remove-fastapi-starlette-aggregate-resources">

`ddtrace.contrib.starlette.config['aggregate_resources']` and `ddtrace.contrib.fastapi.config['aggregate_resources']` are removed. The starlette and fastapi integrations no longer have the option to `aggregate_resources`, as it now occurs by default.

</div>

- <div id="remove-grpc-port-key">

`ddtrace.contrib.grpc.constants.GRPC_PORT_KEY` is removed. Use `ddtrace.ext.net.TARGET_PORT` instead.

</div>

- <div id="remove-cassandra-row-count">

`ddtrace.ext.cassandra.ROW_COUNT` is removed. Use `ddtrace.ext.db.ROWCOUNT` instead.

</div>

- <div id="remove-mongo-row-count">

`ddtrace.ext.mongo.ROW_COUNT` is removed. Use `ddtrace.ext.db.ROWCOUNT` instead.

</div>

- <div id="remove-sql-row-count">

`ddtrace.ext.sql.ROW_COUNT` is removed. Use `ddtrace.ext.db.ROWCOUNT` instead.

</div>

- <div id="remove-trace-ci-visibility-filter">

`ddtrace.filters.TraceCiVisibilityFilter` is removed.

</div>

- <div id="remove-dd-log-format">

`ddtrace.tracer.DD_LOG_FORMAT` is removed. As an alternative, please follow the log injection formatting as provided in the [log injection docs](https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#update-log-format).

</div>

- <div id="remove-basic-config">

`DD_CALL_BASIC_CONFIG` is removed. There is no special configuration necessary to replace `DD_CALL_BASIC_CONFIG`. The ddtrace logger will log to stdout by default or additionally to a file specified by `DD_TRACE_LOG_FILE`.

</div>

- <div id="remove-trace-obfuscation-query-string-pattern">

`DD_TRACE_OBFUSCATION_QUERY_STRING_PATTERN` is removed. Use `DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP` instead.

</div>

New Features

- Adds support for Python 3.12.

Known Issues

- aiohttp: Python 3.12 is not supported.
- aiohttp-jinja: Python 3.12 is not supported.
- aiobotocore: Python 3.12 is not supported.
- asm: IAST for Python 3.12 is not supported.
- flask-caching: Python 3.12 is not supported.
- openai/langchain: Python 3.12 is not supported.
- opentelemetry-api: Python 3.12 is not supported.
- opentracing: Python 3.12 is not supported.
- pyramid: Python 3.12 is not supported.
- pynamodb: Python 3.12 is not supported.
- redis/redis-py-cluster: Python 3.12 is not supported.

---