Ddtrace

Prelude

profiling: Code provenance is a feature that enhances the "My code" experience in the Datadog UI by allowing the tracer to report packaging metadata about installed source files. This information is used to distinguish between user and third-party code.

New Features

- aws: Adds span tags for consistency with tags collected by Datadog for AWS metrics and logs.

- botocore: Adds the ability to control which botocore submodules will be patched.

- ASM: Send WAF metrics over telemetry

- pytest: This introduces test suite and module level visibility for the pytest integration. Pytest test traces will now include test session, test module, test suite, and test spans, which correlate to pytest session, pytest package, pytest module, and pytest test functions respectively.

- redis: Introducing redis command span tag max length configuration for `aioredis<aioredis>`, `aredis<aredis>`, `redis<redis>`, `rediscluster<rediscluster>`, and `yaaredis<yaaredis>` integrations.

- profiling: Code provenance is enabled by default.

- OpenAI: Add integration with support for metrics, logs and traces from
OpenAI requests. See [the docs](https://ddtrace.readthedocs.io/en/stable/integrations.html#openai) for more information.

Bug Fixes

- dependencies: Resolves an issue where ddtrace installs an incompatible version of cattrs when Python 3.6 is used.

- tracing: Resolves an issue where `DD_TRACE_<INTEGRATION>_ENABLED=False` could not be used to disable the following integrations when `ddtrace-run` was used: flask, django, bottle, falcon, and pyramid.

- asgi: Ensures `error.message` and `error.stack` tags are set when an exception is raised in a route.

- appsec: Fixes an encoding error when we are unable to cleanup the AppSec request context associated with a span.

- ASM: Fixes encoding error when using AppSec and a trace is partial flushed.

- CI Visibility: This fix resolves an issue where the tracer was doing extra requests if the `DD_CIVISIBILITY_ITR_ENABLED` env var was not set.

- CI Visibility: This fix resolves an issue where the API call would fail because it is reporting a null service name

- bootstrap: fixed an issue with the behavior of `ddtrace.auto` that could have caused incompatibilities with frameworks such as `gevent` when used as a programmatic alternative to the `ddtrace-run` command.

- django: Fixed a bug that prevented a Django application from starting with celery and gevent workers if `DJANGO_SETTINGS_MODULE` was not explicitly set.

- tracing: Fixes a cryptic encoding exception message when a span tag is not a string.

- ASM: fix extract_body for Django such that users of Django Rest Framework can still use custom parsers.

- flask: Remove patching for Flask hooks `app.before_first_request` and `bp.before_app_first_request` if Flask version \>= 2.3.0.

- gevent: Fix a bug that caused traceback objects to fail to pickle when using gevent.

- OpenAI: Resolved an issue where OpenAI API keys set in individual requests rather than as an environment variable caused an error in the integration.

- profiler: Fixed a bug that caused segmentation faults in applications that use protobuf as a runtime dependency.

- redis: Resolves an issue where the aioredis/aredis/yaaredis integrations cross-imported a helper method from the redis integration, which triggered redis patching before the redis integration was fully loaded.

- wsgi: Resolves an issue where accessing the `__len__` attribute on traced wsgi middlewares raised a TypeError

- django: Adds catch to guard against a ValueError, AttributeError, or NotImplementedError from being thrown when evaluating a django cache result for `db.row_count` tag.

- lib-injection: Ensure local package is installed. Previously the package
could still be pulled from the internet causing application slowdowns.

- kafka: Fixes `TypeError` raised when arbitrary keyword arguments are passed to `confluent_kafka.Consumer`

- profiler: Fix support for latest versions of protobuf.

- psycopg: Resolves an issue where an AttributeError is raised when `psycopg.AsyncConnection` is traced.

- sanic: Resolves `sanic_routing.exceptions.InvalidUsage` error raised when gevent is installed or `DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE` is set to True.

- elasticsearch: This fix resolves an issue where the tracer would throw an error when patching unsupported versions of elasticsearch (\> 8.0). Patching is now skipped if an unsupported version is detected.

Other Changes

- span: Increases the traceback limit in `error.stack` tags from 20 to 30
- aws_lambda: Logs warnings and exceptions on cold start only.

---

New Features

- psycopg: This release adds support for the new psycopg3 package. This new integration has all the same tracing functionality as the previous psycopg2-binary package, with added support for new methods including async connection and async cursor classes. The release also adds support for using Django\>=4.2 with psycopg3 integrated tracing.

Bug Fixes

- algoliasearch: This fix resolves an issue where non-text search query arguments caused Type Errors when being added as tags.

- ASM: fix calling <span class="title-ref">set_user</span> without a created span raising a <span class="title-ref">ValueError</span>.

- django: Adds fix for bug where Django cache return object throws an error if it does not implement `__bool__()`.

- kafka: Previously instantiating a subclass of kafka's Producer/Consumer classes would result in attribute errors due to patching the Producer/Consumer classes with an ObjectProxy. This fix resolves this issue by making the traced classes directly inherit from kafka's base Producer/Consumer classes.

- profiling: Fixed a regression in the memory collector that caused it to fail to cleanly re-initialize after a fork, causing error messages to be logged.

- logging: Ensure that the logging module can report thread information, such as thread names, correctly when a framework like gevent is used that requires modules cleanup.

- ASM: This fix resolves an issue where path parameters for the Flask framework were handled at response time instead of at request time for suspicious request blocking. This close a known issue opened in 1.10.0.

- lib-injection: Switch installation to install from included wheels. Prior,
the wheels were merged together which caused conflicts between versions of dependencies based on Python version.

- tracer: Handle exceptions besides `ImportError` when integrations are loaded.

Other Changes

- ASM: Add information about Application Security config values on <span class="title-ref">ddtrace-run --info</span>.
- otel: Fixes code formatting in api docs

---

New Features

- tracing: Adds support for 128 bit trace ids for b3 and w3c distributing tracing headers.
- pytest: Adds the `DD_CIVISIBILITY_AGENTLESS_ENABLED` environment variable to configure the `CIVisibility` service to use an agent-less test reporting `CIVisibilityWriter`. Note that the `CIVisibility` service will use regular agent reporting by default.
- sci: Extracts and sends git metadata from environment variables `DD_GIT_REPOSITORY_URL`, `DD_GIT_COMMIT_SHA`, or from the python package specified in the `DD_MAIN_PACKAGE`. This feature can be disabled by setting `DD_TRACE_GIT_METADATA_ENABLED=False`.
- otel: Adds support for the [OpenTelemetry Tracing API](https://opentelemetry.io/docs/reference/specification/trace/api/). Please refer to the `docs <ddtrace.opentelemetry>` for more details.

Bug Fixes

- tracing: Ensure datadog headers propagate 128 bit trace ids when `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=False`
- aws_lambda: Fix AttributeError raised when `ddtrace.patch_all()`, or `ddtrace.patch(aws_lambda=True)`, is set on user handler.
- aws_lambda: Fix AttributeError raised when extracting context from arguments.
- aws_lambda: Fix AttributeError raised when callable handlers are traced.
- dynamic instrumentation: Fixed an issue with expressions in metric probes that prevented them from being evaluated.
- Prevent exceptions when autoreloading modules that directly or indirectly import ddtrace with the iPython autoreload extension.
- profiling: Corrects accounting of wall and CPU time for gevent tasks within the main Python thread.
- profiling: Fixed an issue with the memory collector where a segmentation fault could occur during shutdown.
- lib-injection: The ddtrace package is now provided via the Docker image rather than relying on a run-time `pip install`. This solves issues like containers blocking network requests, installation overhead during application startup, permissions issues with the install.

---

Deprecation Notes

- ASM: Several deprecated ASM constants that were added to the public API will be removed. This should not affect existing code as they were meant for private use only.

New Features

- tracing: Adds support for 128 bit trace ids. To generate and propagate 128 bit trace ids using Datadog distributed tracing headers set the following configuration: `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=True`. Support for B3 and W3C distributed tracing headers will be added in a future change.
- aiohttp: Add missing component meta tag to aiohttp server spans.
- redis: Adds tracing support for <span class="title-ref">redis.cluster.RedisCluster</span>.
- celery: Adds automatic tracing of the `celery.beat` scheduling service to the `celery` integration.
- kafka: Adds instrumentation support for `confluent-kafka>=1.7`. See the `confluent-kafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#kafka>` documentation for more information.
- dynamic instrumentation: introduced support for dynamic span probes.
- Adds source code integration with setuptools build metadata. This enables traces and profiles to be automatically tagged with git metadata to track deployments in Datadog.

Bug Fixes

- tracing: This fix resolves an issue where making a sampling decision before the `env` span tag had been set caused sample rate data from the Datadog Agent to be ignored.
- ASM: make `track_custom_event()` also set `appsec.events.<custom_event>.track` which was missing.
- django: Fixes an issue where `http.route` was only set if `use_handler_resource_format` and `use_legacy_resource_format` were set to `False`.
- tracing: This fix resolves an issue where a very long string as a span attribute would cause that span not to be delivered. It replaces string span attributes larger than DD_TRACE_WRITER_BUFFER_SIZE_BYTES (which as of this version defaults to 8388608) with a small string containing debug information and not containing any of the original attribute string.
- ASM: Resolves installation issues with compiling native code on Windows and unknown platforms.
- aws_lambda: Fixes a `RecursionError` which is raised when aws lambda signal handlers are wrapped infinitely. This caused lambdas to crash on startup.
- botocore: Fix TypeError raised by injecting trace context into Kinesis messages.
- dynamic instrumentation: Fix a bug where the dynamic instrumentation would stop injecting function probes after the first failed one.
- dynamic instrumentation: This change fixes a bug whereby probes that have been disabled/removed from the front-end would not be removed by the client library.
- futures: Resolves an issue that prevents tasks from being submitted to a thread pool executor when gevent is used (e.g. as a worker class for gunicorn or celery).
- propagation: This fix resolves an issue where previously W3C tracestate propagation could not handle whitespace. With this fix whitespace is now removed for incoming and outgoing requests.
- httplib: Fixes an issue with patching of http client upon import
- Ensure DD_REMOTE_CONFIGURATION_ENABLED environment variable disables remote config if set to False

Other Changes

- aws_lambda: Updates how <span class="title-ref">DD_APM_FLUSH_DEADLINE_MILLISECONDS</span> is used. Previously, we would set the deadline as the environment variable value, if set. Now, when the remaining time in an AWS Lambda invocation is less than <span class="title-ref">DD_APM_FLUSH_DEADLINE_MILLISECONDS</span>, the tracer will attempt to submit the current active spans and all finished spans. the value in the environment variable is used to subtract from the deadline. The default is still 100ms.

---

Prelude

Application Security Management (ASM) has added Django support for blocking malicious users using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Deprecation Notes

- dbapi: `ddtrace.ext.mongo.ROWS` is deprecated. Use `ddtrace.ext.db.ROWCOUNT` instead.

New Features

- starlette: Add http.route tag to `starlette.request` spans.
- fastapi: Add http.route tag to `fastapi.request` spans.
- ASM: Add support for one click blocking of user ids with the Django framework using Remote Configuration Management.
- ASM: This introduces the "suspicious request blocking" feature for Django and Flask.

Known Issues

- ASM: There is a known issue with the flask support for any rule blocking on `server.request.path_params`. The request will be correctly blocked but the client application will be receiving and processing the suspicious request. Possible workaround: use `server.request.uri.raw` instead, if you want the request to be blocked before entering the flask application.

Bug Fixes

- dbapi: The dbapi integration no longer assumes that a cursor object will have a rowcount as not all database drivers implement rowcount.

- dbm: Support sql queries with the type `byte`.

- elasticsearch: Omit large `elasticsearch.body` tag values that are
greater than 25000 characters to prevent traces from being too large to send.

- aws_lambda: This fix resolves an issue where existing signals were wrapped multiple times.

- profiling: Handles a race condition on process shutdown that would cause an error about a module not being started to occasionally appear in the logs.

- Fix for KeyError exceptions when when <span class="title-ref">ASM_FEATURES</span> (1-click activation) disabled all ASM products. This could cause 1-click activation to work incorrectly in some cases.
- ASM: Solve some corner cases where a Flask blocking request would fail because headers would be already sent.
- ASM: Solve the content-type not always being correct in blocking responses.
- ASM: Ensure the blocking responses have the following tags: <span class="title-ref">http.url</span>, <span class="title-ref">http.query_string</span>, <span class="title-ref">http.useragent</span>, <span class="title-ref">http.method</span>, <span class="title-ref">http.response.headers.content-type</span> and <span class="title-ref">http.response.headers.content-length</span>.
- ASM: fix memory leaks and memory corruption in the interface between ASM and the WAF library
- psycopg2: Fixes a bug with DSN parsing integration.

Other Changes

- remote_config: Change the level of remote config startup logs to debug.

---

Deprecation Notes

- gevent: `DD_GEVENT_PATCH_ALL` is deprecated and will be removed in the next major version. Gevent compatibility is now automatic and does not require extra configuration when running with `ddtrace-run`. If not using `ddtrace-run`, please import `ddtrace.auto` before calling `gevent.monkey.patch_all()`.

Bug Fixes

- aws_lambda: Resolves an exception not being handled, which occurs when no root span is found before a lambda times out.
- gevent: This fix resolves an incompatibility between ddtrace and gevent that caused threads to hang in certain configurations, for example the profiler running in a gunicorn application's gevent worker process.

Other Changes

- ASM: The list of headers for retrieving the IP when Application Security Management is enabled or the
<span class="title-ref">DD_TRACE_CLIENT_IP_ENABLED</span> environment variable is set has been updated. "Via" has been removed as it rarely contains IP data and some common vendor headers have been added. You can also set the environment variable <span class="title-ref">DD_TRACE_CLIENT_IP_HEADER</span> to always retrieve the IP from the header specified as the value.

---