Ddtrace

Bug Fixes

- ASM: This fix resolves an issue where django login failure events may send wrong information of user existence.
- Code Security: fix setting the wrong source on map elements tainted from <span class="title-ref">taint_structure</span>.
- datastreams: Changed DSM processor error logs to debug logs for a statement which is retried. If all retries fail, the stack trace is included
- Code Security: Fixes an issue where the AST patching process fails when the origin of a module is reported as None, raising a `FileNotFoundError`.
- CI Visibility: fixes an issue where tests were less likely to be skipped due to ITR skippable tests requests timing out earlier than they should
- internal: This fix resolves an issue where importing the `ddtrace.contrib.botocore.services` module would fail raising an ImportError
- starlette: Fix a bug that crashed background tasks started from functions without a <span class="title-ref">\_\_name\_\_</span> attribute
- Code Security: Fixed an issue with AES functions from the pycryptodome package that caused the application to crash and stop.
- Code Security: This fix addresses an issue where tainting objects may fail due to context not being created in the current span.
- Code Security: Some native exceptions were not being caught correctly by the python tracer. This fix remove those exceptions to avoid fatal error executions.
- ASM: This fix removes unrequired API security metrics.
- structlog: Fixes error where multiple loggers would duplicate processors. Also adds processors injection when resetting to defaults.

---

Bug Fixes

- propagation: This fix resolves an issue where the sampling decision-maker tag in tracestate propagation headers was clobbered by a default value.
- langchain: Ensures langchain vision APIs are correctly instrumented
- ASM: This fix resolves an issue where the asgi middleware could crash with a RuntimeError "Unexpected message received".
- kafka: This fix resolves an issue where `None` messages from confluent-kafka could cause crashes in the Kafka integration.


---

Upgrade Notes

- CI Visibility: `DD_CIVISIBILITY_ITR_ENABLED` now defaults to true, and the Datadog API (configured via the Datadog dashboard) now determines whether code coverage and test skipping are enabled.
- CI Visibility: the CI Visibility service is no longer enabled when the initial query to the Datadog test service settings API fails due to a 403 status code.

New Features

- botocore: Adds optional feature to propagate context between producers and consumers for AWS SQS, AWS SNS, and AWS Kinesis via <span class="title-ref">DD_BOTOCORE_PROPAGATION_ENABLED</span> environment variable. Adds optional feature to disable tracing of AWS SQS <span class="title-ref">poll()</span> operation and AWS Kinesis 'get_records()' operation when no data is consumed via <span class="title-ref">DD_BOTOCORE_EMPTY_POLL_ENABLED</span> environment variable.

- tracing: Adds new tag <span class="title-ref">python_main_package</span> containing the name of the main package of the application. profiling: Adds new tag <span class="title-ref">python_main_package</span> containing the name of the main package of the application.

- ASM: API Security schema collection is now officially supported for Django, Flask and FastAPI. It can be enabled in the tracer using environment variable DD_API_SECURITY_ENABLED=true It will only be active when ASM is also enabled.

- elasticsearch: This allows custom tags to be set on Elasticsearch spans via the Pin interface.

- botocore: This introduces tracing support for bedrock-runtime operations.
See [the docs](https://ddtrace.readthedocs.io/en/stable/integrations.html#botocore) for more information.

- datastreams: this change adds kombu auto-instrumentation for datastreams monitoring. tracing: this change adds the `DD_KOMBU_DISTRIBUTED_TRACING` flag (default `True`)

- Vulnerability Management for Code-level (IAST): Add support for CMDi in langchain.

- botocore: Add the ability to inject trace context into the input field of botocore stepfunction start_execution and start_sync_execution calls.

- Removes another place where we always load instrumentation telemetry, even if it is disabled

- tracing: This introduces the ability to disable tracing at runtime based on configuration values sent from the Datadog frontend. Disabling tracing in this way also disables instrumentation telemetry.

- tracing: Adds support for remote configuration of `DD_TRACE_HEADER_TAGS`

- tracing: Add support for remote configuration of trace-logs correlation.

- grpc/grpc_aio: reports the available target host in client spans as `network.destination.ip` if only an IP is available, `peer.hostname` otherwise.

- span: Adds a public api for setting span links

- starlette,fastapi: Trace background tasks using span links

Bug Fixes

- ASM: This fix resolves an issue where an exception would be logged while parsing an empty body JSON request.

- CI Visibility: fixes an issue where coverage data for suites could be lost for long-running test sessions, reducing the possibility of skipping tests when using the Intelligent Test Runner.

- IAST: Don't split AST Assign nodes since it's not needed for propagation to work.

- ASM: This fix resolves an issue where suspicious request blocking on request data was preventing API Security to collect schemas in FastAPI, due to route not being computed.

- ASM: This fix resolves an issue where ASM custom blocking actions with a redirect action could cause the server to drop the response.

- Fixed an incompatible version requirements for one of the internal dependencies that could have caused an exception to be raised at runtime with Python 3.12.

- data_streams: This change fixes a bug leading to lag being reported as 1 offset instead of 0 offsets.

- IAST: fixes import overhead when IAST is disabled.

- Fix an incomplete support for pkg_resouces that could have caused an exception on start-up.

- Fix an issue that caused an exception to be raised when trying to access resource files via `pkg_resources`.

- Fix for an import issue that caused the pytest plugin to fail to properly initialize a test session and exit with an import exception.

- openai: This fixes a bug that prevents logs from being correlated with traces in the Datadog UI.

- langchain: This fixes a bug that prevents logs from being correlated with traces in the Datadog UI.

- openai: This fix resolves an issue where an internal OpenAI method <span class="title-ref">SyncAPIClient.\_process_response</span>
was not being patched correctly and led to to an AttributeError while patching.

- profiling: handle a potential system error that may be raised when running a Celery-based application with CPython 3.11.

- Fixed an issue that could have caused an exception as a result of a concurrent access to some internal value cache.

- tracing: Ensures span links are serialized with the expected traceflag when `DD_TRACE_API_VERSION=v0.4`

- ASM: This fix resolves an issue where IP Headers configured by the user in the environment could not work for frameworks handling requests with case insensitive headers like FastAPI.

- Vulnerability Management for Code-level (IAST): Fixes a bug in the `str` aspect where encoding and errors arguments were not honored correctly.

- Vulnerability Management for Code-level (IAST): Fix an unhandled ValueError in `ast_function` thrown in some cases (i.e. Numpy arrays when converted to bool).

- opentelemetry: Ensures that span links are serialized in a json-compatible representation.

- Pin importlib_metadata to 6.5.0 to avoid its issue 455 (<https://github.com/python/importlib_metadata/issues/455>).

- profiler: Fixes a sigabrt when shutdown occurs during an upload

- otel: Ensures all otel sampling decisions are consistent with Datadog Spans. This prevents otel spans in a distributed trace from being sampled differently than Datadog spans in the same trace.

- tracing: Fix an issue where remote configuration values would not be reverted when unset in the UI.

- tracing: Ensures hostnames are reported in statsd metrics if `DD_TRACE_REPORT_HOSTNAME=True` (default value is `False`).

Other Changes

- setup: pins the default macOS deployment target to 10.14.
- tracing: Updates the default value of `DD_TRACE_PROPAGATION_STYLE` from `tracecontext,datadog` to `datadog,tracecontext`. With this change w3c tracecontext headers will be parsed before datadog headers. This change is backwards compatible and should not affect existing users.

---

New Features

- aiohttp: add <span class="title-ref">split_by_domain</span> config to split service name by domain
- CI Visibility: Adds code coverage lines covered tag for `pytest` and `unittest`.
- aiohttp: Adds http.route tag to `aiohttp.request` spans.
- bottle: Adds http.route tag to `bottle.request` spans.
- falcon: Adds http.route tag to `falcon.request` spans.
- molten: Adds http.route tag to `molten.request` spans.
- Adds distributed tracing for confluent-kafka integration. Distributed tracing connects Kafka consumer spans with Kafka producer spans within the same trace if a message is valid. To enable distributed tracing, set the configuration: `DD_KAFKA_DISTRIBUTED_TRACING_ENABLED=True` for both the consumer and producer service.
- ASM: This introduces (experimental) api security support for fastAPI. Flask and Django were already supported in 2.4.0. Support schema computation on all addresses (requests and responses) and scanner support for pii, credentials and payment data.
- CI Visibility: introduces a CI visibility-specific logger (enabled for the `pytest` plugin), enabled by setting the `DD_CIVISIBILITY_LOG_LEVEL` environment variable (with the same level names as Python logging levels).
- CI Visibility: allows for waiting for the git metadata upload to complete before deciding whether or not to enable coverage (based on API response).
- Further lazy loads telemetry_writer so that it is not running when explicitly disabled. Users must explicitly set "DD_INSTRUMENTATION_TELEMETRY_ENABLED=false".
- tracer: Add support for remotely configuring trace tags.

Bug Fixes

- loguru: Ensures log correlation is enabled when the root logger is initialized. Previously, log correlation was only enabled when a new sink was added.
- Fix compatibility with other tools that try to infer the type of a Python object at runtime.
- tracing: Fixes a bug that prevents span links from being visualized in the Datadog UI.
- tracing: Resolves span encoding errors raised when span links do not contain expected types
- ASM: This fix resolves an issue where custom event boolean properties were not reported as <span class="title-ref">true</span> and <span class="title-ref">false</span> like other tracers but as <span class="title-ref">True</span> and <span class="title-ref">False</span>.
- Vulnerability Management for Code-level (IAST): Ensure that Cookies vulnerabilities report only the cookie name.
- langchain: This fix resolves an `get_openai_token_cost_for_model` import error in langhcain version 0.0.351 or later.
- ASM: This fix resolves an issue where IAST could cause circular dependency at startup.
- tracing: Ensures all fields in `ddtrace.context.Context` are picklable.
- pytest: This fix resolves an issue where the <span class="title-ref">--no-cov</span> flag did not take precedence over the <span class="title-ref">--cov</span> flag when deciding whether to report code coverage on spans.
- rq: Fixed a bug where the RQ integration would emit a warning when setting `job.status` span tag.

---

Upgrade Notes

- <div id="remove-unsupported-pylons">

This removes the `pylons` integration, which does not support Python 3.

</div>

Deprecation Notes

- aioredis: The aioredis integration is deprecated and will be removed in a future version. As an alternative to the aioredis integration, you can use the redis integration with redis\>=4.2.0.

New Features

- ASM: dependency telemetry metrics now will only report dependencies actually in use (imported) and will also report new imported modules periodically.

- ASM: This introduces Threat Monitoring and Blocking on FastAPI.
- IP Blocking and all input addresses are supported on requests and responses

\- Custom Blocking This does not contain user blocking specific features yet.

- tracing: Introduces support for OpenTracing Baggage Items with HTTP Propagation. Enable this support by `DD_TRACE_PROPAGATION_HTTP_BAGGAGE_ENABLED=true`. The `Context._set_baggage_item` and `Context._get_baggage_item` internal methods are provided for manual modifications to the Baggage Items. These API changes are subject to change.

- dynamic instrumentation: Add support for more built-in container types, such as `defaultdict`, `frozenset`, `OrderedDict` and `Counter`.

- Vulnerability Management for Code-level (IAST): Adds Python 3.12 compatibility

- Optionally lazy loads and disables Instrumentation Telemetry. Users must explicitly set "DD_INSTRUMENTATION_TELEMETRY_ENABLED=false".

- tracer: Add support for remotely setting the trace sample rate from the Datadog UI. This functionality is enabled by default when using `ddtrace-run` and library injection. To enable it when using the library manually, use `ddtrace.config.enable_remote_config()`.

Bug Fixes

- tracer: tag spans that have been sampled due to an Agent sampling configuration.
- lambda: This change disables the use of `multiprocessing.queue` in Lambda, because it is not supported in Lambda
- langchain: This fix resolves a crash that could occur during embedding when no embeddings are found.
- Fix a regression with the support for gevent that could have occurred if some products, like ASM, telemetry, were enabled.
- kafka: Resolves `TypeError` raised by serializing producers and deserializing consumers when the `message.key` tag is set on spans.
- dynamic instrumentation: Fix an issue that caused the instrumented application to fail to start if a non-standard module was imported.
- openai: This fix resolves an issue where tagging image inputs in the chat completions endpoint resulted in attribute errors.
- openai: This fix resolves an issue where requesting raw API responses from openai\>=1.0 resulted in attribute errors while tagging.
- profiling: Fix an issue that prevented threading locks from being traced when using gevent.
- profiling: Fix a segmentation fault with CPython 3.12 when sampling thread stacks.
- pylibmc: Fixes an issue where using `ddtrace-run` or `ddtrace.patch_all()` with `DD_TRACE_ENABLED=False` would break with get, gets, and get_multi operations on pylibmc Clients.
- tracing: This fix resolves an issue where concurrent mutations to the `context._meta` dict caused <span class="title-ref">RuntimeError: dictionary changed size during iteration</span>.
- django: Resolves `AttributeError` raised by traced `StreamingHttpResponse`.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where certain aspects incorrectly expected at least one argument, leading to an IndexError when none were provided. The solution removes this constraint and incorporates regression tests for stability assurance.
- Vulnerability Management for Code-level (IAST): Cookies vulnerabilities are only reported if response cookies are insecure.
- Vulnerability Management for Code-level (IAST): Fix propagation error on `.format` string method.
- requests: Updates the resource names of `requests.requests` spans to include the method and path of the request.
- propagation: This fix resolves an issue where a `Context` generated from extracted headers could lack a span_id or trace_id, leading `SpanLink` encoding errors.
- psycopg: This fix resolves an issue where a circular import of the psycopg library could cause a crash during monkeypatching.
- psycopg: This fix resolves an issue where exceptions originating from asynchronous Psycopg cursors were not propagated up the call stack.
- redis: This fix resolves an issue where the yaaredis and aredis integrations imported code from the redis integration, causing a circular import error.
- tracing: Resolves trace encoding errors raised when `DD_TRACE_API_VERSION` is set to `v0.5` and a BufferFull Exception is raised by the TraceWriter. This fix ensures span fields are not overwritten and reduces the frequency of 4XX errors in the trace agent.

Other Changes

- tracing: Upgrades the trace encoding format to v0.5. This change improves the performance of encoding and sending spans.

---

New Features

- propagation: 128-bit trace ids are now used by default for propagation. Previously the default was 64-bit. This change is backwards compatible with tracers that still use 64-bit trace ids and should not cause any breaking behavior.
- Adds DSM `pathway.hash` tag to spans when DSM is enabled. This allows traces from the instrumented service to show up in the DSM traces tab.
- propagation: When the tracer is configured to extract and inject `tracecontext`, the tracer will propagate the tracestate values from other vendors so long as the traceparent trace-id matches the first found trace context, regardless of propagator configuration order. To disable this behavior `DD_TRACE_PROPAGATION_EXTRACT_FIRST=true` can be set.
- opentelemetry: Map reserved OpenTelemetry attributes to Datadog span model.
- opentelemetry: datadog operation name from semantic conventions
- propagation: If a valid context is extracted from headers, and the following extracted trace context's `trace_id`s do not match the valid context's, then add a span link to the root span to represent the broken propagation.
- tracing: This change treats spans that terminated with `sys.exit(0)` as successful non-error spans.
- tracing: This introduces the `DD_TRACE_SPAN_TRACEBACK_MAX_SIZE` environment variable, allowing the maximum size of tracebacks included on spans to be configured.

Bug Fixes

- CI Visibility: fixes the fact that the GITHUB_SERVER_URL environment variable was not being sanitized for credentials
- dynamic instrumentation: Needs to update the pubsub instance when the application forks because the probe mechanism should run in the child process. For that, DI needs the callback as the method of an instance of Debugger, which lives in the child process.
- CI Visibility: Fixes an issue where a `ValueError` was raised when using different path drives on Windows
- Fixes an issue where ddtrace could not be installed from source when using `setuptools>=69` due to a change in the license field.
- tracing: Fixes an issue where the thread responsible for sending traces is killed due to concurrent dictionary modification.
- structlog: Fixes `TypeError` raised when ddtrace log processor is configured with a tuple
- Vulnerability Management for Code-level (IAST): Generates cookies vulnerabilities report if IAST is enabled. Before this fix, Cookies vulnerabilities were only generated if both IAST and Appsec were enabled.
- Vulnerability Management for Code-level (IAST): This fix resolves an issue where, at AST patching to replace code with IAST aspects, passing the original function/method as an extra parameter for accurate patching unintentionally triggers side effects in methods obtained from an expression (like `decode` in `file.read(n).decode()`), resulting in unexpected multiple calls to the expression (`file.read(n)` in the example).
- Vulnerability Management for Code-level (IAST): This fix eliminates some reference leaks and C-API usage when IAST reports a vulnerability and calls `get_info_frame`.
- kafka: This fix resolves an issue where calls to `confluent_kafka`'s `produce` method with `key=None` would cause an exception to be raised.
- tracing: This fix resolves an issue where ddtrace's signal handlers could cause Flask apps not to respond correctly to SIGINT.
- logging: A log handler is automatically added to the ddtrace logger upon ddtrace import, when not using ddtrace-run. This can lead to duplicate logging if users add additional loggers and do not explicitly modify the ddtrace logger. This fix adds a feature flag that can be used to toggle this behavior off `DD_TRACE_LOG_STREAM_HANDLER` which defaults to `true`.

---