Ddtrace

Prelude

Application Security Management (ASM) has added Django support for blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Application Security Management (ASM) has added Flask support for blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Deprecation Notes

- grpc: Deprecates `ddtrace.contrib.grpc.constants.GRPC_PORT_KEY`. Use `ddtrace.ext.net.TARGET_PORT` instead.
- dbapi: `ddtrace.ext.sql.ROWS` is deprecated. Use `ddtrace.ext.db.ROWCOUNT` instead.
- cassandra: `ddtrace.ext.cassandra.ROW_COUNT` is deprecated. Use `ddtrace.ext.db.ROWCOUNT` instead.

New Features

- Enable traces to be sent before an impending timeout for `datadog_lambda>=4.66.0`. Use `DD_APM_FLUSH_DEADLINE` to override the default flush deadline. The default is the AWS Lambda function configured timeout limit.

- debugger: Add dynamic log probes to that generate a log message and optionally capture local variables, return value and exceptions

- tracing: Add support for enabling collecting of HTTP request client IP addresses as the `http.client_ip` span tag. You can set the `DD_TRACE_CLIENT_IP_ENABLED` environment variable to `true` to enable. This feature is disabled by default.

- ASM: add support for one click blocking of IPs with the Django framework using Remote Configuration Management.

- ASM: add support for one click blocking of IPs with the Flask framework using
Remote Configuration Management.

- ASM: also fetch loopback IPs if client IP fetching is enabled (either via ASM or DD_TRACE_CLIENT_IP_ENABLED).

- ASM: Enable ability to remotely activate and configure ASM features. To enable, check the Python Security page in your account. Note that this is a beta feature.

- profiling: Collects endpoint invocation counts.

- dynamic instrumentation: Python 3.11 is now supported.

- graphene: Adds support for Python 3.11.

- graphql: Adds support for Python 3.11.

- httpx: Add support for `httpx<0.14.0,>=0.9.0`.

- tracer/span: Add `Span.finish_with_ancestors` method to enable the abrupt
finishing of a trace in cases where the trace or application must be immediately terminated.

Known Issues

- remote config: There is a known issue with remote configuration management (RCM) when paired with gevent which can cause child processes to deadlock. If you are experiencing issues, we recommend disabling RCM with `DD_REMOTE_CONFIGURATION_ENABLED=false`. Note, this will disable one click activation for ASM.
- gunicorn: ddtrace-run does not work with gunicorn. To instrument a gunicorn application, follow the instructions [here](https://ddtrace.readthedocs.io/en/latest/integrations.html#gunicorn).

Bug Fixes

- fastapi: Previously, custom fastapi middlewares configured after application startup were not traced. This fix ensures that all fastapi middlewares are captured in the <span class="title-ref">fastapi.request</span> span.

- tracing: Pads trace_id and span_ids in b3 headers to have a minimum length of 16.

- Fix full stacktrace being sent to the log on remote config connection errors.

- httpx: Only patch `httpx.AsyncClient` for `httpx>=0.11.0`.

- tracing: This fix resolves an issue with the encoding of traces when using the v0.5 API version with the Python optimization option flag `-O` or the `PYTHONOPTIMIZE` environment variable.

- pylons: This fix resolves an issue where `str.decode` could cause critical unicode decode errors when ASM is enabled. ASM is disabled by default.

- gevent: This fix resolves incompatibility under 3.8\>=Python\<=3.10 between `ddtrace-run` and applications that depend on `gevent`, for example `gunicorn` servers. It accomplishes this by keeping copies that have not been monkey patched by `gevent` of most modules used by `ddtrace`. This "module cloning" logic can be controlled by the environment variable `DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE`. Valid values for this variable are "1", "0", and "auto". "1" tells `ddtrace` to run its module cloning logic unconditionally, "0" tells it never to run that logic, and "auto" tells it to run module cloning logic *only if* `gevent` is accessible from the application's runtime. The default value is "0".

- lib-injection: Use package versions published to PyPI to install the
library. Formerly the published image was installing the package from source using the tagged commit SHA which resulted in slow and potentially failing installs.

- profiler: Handles potential `AttributeErrors` which would arise while collecting frames during stack unwinding in Python 3.11.

- remote config: ensure proper validation of responses from the agent.

---

Upgrade Notes

- ASM: libddwaf upgraded to version 1.6.1 using a new library loading mechanism
- profiling: upgrades the profiler to support the `v2.4` backend API for profile uploads, using a new request format.

Deprecation Notes

- `DD_REMOTECONFIG_POLL_SECONDS` environment variable is deprecated and will be removed in v2.0. Please use `DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS` instead.

New Features

- CI Visibility: Add support for CI provider buddy.works

- The component tag has been added for all auto-instrumented spans. The value of the component tag is equal to the name of the integration that produced the span.

- tracing: Adds support for IPv6 agent hostnames for <span class="title-ref">DD_AGENT_HOST</span>.

- elasticsearch: Update `elasticsearch` integration to add support for `opensearch-py`. See [the elasticsearch documentation](https://ddtrace.readthedocs.io/en/stable/integrations.html#elasticsearch) for more information.

- ASM: one click activation enabled by default using Remote Configuration Management (RCM). Set `DD_REMOTE_CONFIGURATION_ENABLED=false` to disable this feature.

- ASM: New Application Security Events Tracking API, starting with the functions `track_user_login_success_event` and
`track_user_login_failure_event` for tracking user logins (it will also internally call `set_user`) and `track_custom_event` for any custom events. You can find these functions in the `ddtrace.appsec.trace_utils` module. Calling these functions will create new tags under the `appsec.events` namespace (`appsec.events.user.login` for logins) allowing you to track these events with Datadog. In the future this will be used to provide protection against account takeover attacks (ATO). Public documentation will be online soon.

- celery: Enhances context tags containing dictionaries so that their contents are sent as individual tags (issue \4771).

- tornado: Support custom error codes: <https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#custom-error-codes>.

- CI Visibility: Support reliably linking tests to the pipeline that executed them.

Known Issues

- profiling: There is currently a known performance regression issue with the profiler's code provenance feature. Note that this feature is disabled by default and will only be enabled if `DD_PROFILING_ENABLE_CODE_PROVENANCE` is set to true.

Bug Fixes

- This fix improves a cryptic error message encountered during some `pip install ddtrace` runs under pip versions \<18.
- dynamic instrumentation: remove unnecessary log line from application start up
- This fix removes unintended url parts in the `http.url` tag.
- botocore: Before this change, the botocore integration stripped newlines from the JSON string encoded in the data blob of Amazon Kinesis records. This change includes a terminating newline if it is present in the decoded data.
- profiling: This fix resolves an issue in Python 3.11 where a PyFrameObject strong reference count was not properly decremented in the stack collector.
- telemetry: This fix resolves an issue when we try to fetch `platform.libc_ver()` on an unsupported system.
- Fix for ValueError when `` is not present in network location but other part of the url.

Other Changes

- profiler: CPU overhead reduction.

---

Prelude

Initial library support has been added for Python 3.11.

<div class="note">

<div class="title">

Note

</div>

Continuous Profiler and Dynamic Instrumentation are not yet compatible and must be disabled in order to use the library with Python 3.11. Support for them will be added in a future release. To track the status, see the [Support Python 3.11](https://github.com/DataDog/dd-trace-py/issues/4149) issue on GitHub.

</div>

Upgrade Notes

- The default propagation style configuration changes to `DD_TRACE_PROPAGATION_STYLE=tracecontext,datadog`. To only support Datadog propagation and retain the existing default behavior, set `DD_TRACE_PROPAGATION_STYLE=datadog`.
- tracer: support for Datadog Agent v5 has been dropped. Datadog Agent v5 is no longer supported since ddtrace==1.0.0. See <https://ddtrace.readthedocs.io/en/v1.0.0/versioning.html#release-support> for the version support.
- Python 3.11: Continuous Profiler and Dynamic Instrumentation must be disabled as they do not current support Python 3.11.
- The configured styles in `DD_TRACE_PROPAGATION_STYLE_EXTRACT` are now evaluated in order to specification. To keep the previous fixed evaluation order, set: `DD_TRACE_PROPAGATION_STYLE_EXTRACT=datadog,b3,b3 single header`.
- tracing: upgrades the default trace API version to `v0.5` for non-Windows systems. The `v0.5` trace API version generates smaller payloads, thus increasing the throughput to the Datadog agent especially with larger traces.
- tracing: configuring the `v0.5` trace API version on Windows machines will raise a `RuntimeError` due to known compatibility issues. Please see <https://github.com/DataDog/dd-trace-py/issues/4829> for more details.

Deprecation Notes

- propagation: Configuration of propagation style with `DD_TRACE_PROPAGATION_STYLE=b3` is deprecated and will be removed in version 2.0.0. Please use the newly added `DD_TRACE_PROPAGATION_STYLE=b3multi` instead.
- aws: The boto, botocore and aiobotocore integrations no longer include all API parameters by default. To retain the deprecated behavior, set the environment variable `DD_AWS_TAG_ALL_PARAMS=1`. The deprecated behavior and environment variable will be removed in v2.0.0.

New Features

- django: add configuration option to allow a resource format like <span class="title-ref">{method} {handler}.{url_name}</span> in projects with Django \<2.2.0
- django: Adds the `DD_DJANGO_INCLUDE_USER_NAME` option to toggle whether the integration sets the `django.user.name` tag.
- Added environment variable `DD_TRACE_PROPAGATION_STYLE` to configure both injection and extraction propagation styles. The configured styles can be overridden with environment variables `DD_TRACE_PROPAGATION_STYLE_INJECT` and `DD_TRACE_PROPAGATION_STYLE_EXTRACT`.
- tracing: This introduces `none` as a supported propagator for trace context extraction and injection. When `none` is the only propagator listed, the corresponding trace context operation is disabled. If there are other propagators in the inject or extract list, the none propagator has no effect. For example `DD_TRACE_PROPAGATION_STYLE=none`
- ASM: now http.client_ip and network.client.ip will only be collected if ASM is enabled.
- tracing: Adds support for W3C Trace Context propagation style for distributed tracing. The `traceparent` and `tracestate` HTTP headers are enabled by default for all incoming and outgoing HTTP request headers. The Datadog propagation style continue to be enabled by default.
- flask: Adds support for streamed responses. Note that two additional spans: `flask.application` and `flask.response` will be generated.
- profiling: Adds support for Python 3.11.
- tracer: added support for Python 3.11.

Bug Fixes

- ASGI: response headers are correctly processed instead of ignored
- Fix issue with `attrs` and `contextlib2` version constraints for Python 2.7.
- CGroup file parsing was fixed to correctly parse container UUID for PCF containers.
- ASM: Do not raise exceptions when failing to parse XML request body.
- ASM: fix a body read problem on some corner case where don't passing the content length makes wsgi.input.read() blocks.
- aws: We are reducing the number of API parameters that the boto, botocore and aiobotocore integrations collect as span tags by default. This change limits span tags to a narrow set of parameters for specific AWS APIs using standard tag names. To opt out of the new default behavior and collect no API parameters, set the environment variable `DD_AWS_TAG_NO_PARAMS=1`. To retain the deprecated behavior and collect all API parameters, set the environment variable `DD_AWS_TAG_ALL_PARAMS=1`.
- tracing: make `ddtrace.context.Context` serializable which fixes distributed tracing across processes.
- django: avoid `SynchronousOnlyOperation` when failing to retrieve user information.
- Remove `forbiddenfruit` as dependency and rollback `wrapt` changes where `forbiddenfruit` was called. IAST: Patch builtins only when IAST is enabled.
- httpx: Fixes an incompatibility from `httpx==0.23.1` when the `URL.raw` property is not available.
- Fix error in patching functions. `forbiddenfruit` package has conflicts with some libraries such as `asynctest`. This conflict raises `AttributeError` exception. See issue \4484.
- tracer: This fix resolves an issue where the rate limiter used for span and trace sampling rules did not reset the time since last call properly if the rate limiter already had max tokens. This fix resets the time since last call always, which leads to more accurate rate limiting.
- Ensure that worker threads that run on start-up are recreated at the right time after fork on Python \< 3.7.
- tracing: This fix resolves an issue where the `DD_SERVICE_MAPPING` mapped service names were not used when updating span metadata with the `DD_VERSION` set version string.
- wsgi: This fix resolves an issue where `BaseException` raised in a WSGI application caused spans to not be submitted.
- library injection: Pin the library version in the library injection image. Prior, the latest version of `ddtrace` would always be installed, regardless of the image version.
- Fix error in the agent response payload when the user disabled ASM in a dashboard using 1-click Remote Configuration.
- flask: add support for flask v2.3. Remove deprecated usages of `flask._app_ctx_stack` and `flask._request_ctx_stack`.
- The specification of `DD_TRACE_PROPAGATION_STYLE_EXTRACT` now respects the configured styles evaluation order. The evaluation order had previously been fixed and so the configured order was ignored.
- tracing: Ensures that encoding errors due to wrong span tag types will be logged. Previously, if non-text span tags were set, this resulted in v0.5 encoding errors to be output to `stderr` instead of to a logger.

Other Changes

- Kubernetes library injection: run commands as non-root user.
- tracing: The value of `ddtrace.constants.PID` has been changed from `system.pid` to `process_id`. All spans will now use the metric tag of `process_id` instead.
- tracing: The exception logged for writing errors no longer includes a long, unhelpful stack trace. The message now also includes the number of traces dropped and the number of retries attempted.

---

Prelude

Application Security Management (ASM) has added support for preventing attacks by blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click activation for ASM is currently in beta.

</div>

Dynamic instrumentation allows instrumenting a running service dynamically to extract runtime information that could be useful for, e.g., debugging purposes, or to add extra metrics without having to make code changes and re-deploy the service. See <https://ddtrace.readthedocs.io/en/stable/configuration.html> for more details.

Upgrade Notes

- Pin \[attrs\](<https://pypi.org/project/attrs/>) dependency to version `>=20` due to incompatibility with \[cattrs\](<https://pypi.org/project/cattrs/>) version `22.1.0`.
- Use `Span.set_tag_str()` instead of `Span.set_tag()` when the tag value is a text type as a performance optimizations in manual instrumentation.

New Features

- ASM: add support for one click activation using Remote Configuration Management (RCM). Set `DD_REMOTE_CONFIGURATION_ENABLED=true` to enable this feature.
- ASM: ip address collection will be enabled if not explicitly disabled and appsec is enabled.
- tracing: HTTP query string tagged by default in http.url tag (sensitive query strings will be obfuscated).
- Django: set <span class="title-ref">usr.id</span> tag by default if <span class="title-ref">request.user</span> is authenticated.
- Introduced the public interface for the dynamic instrumentation service. See <https://ddtrace.readthedocs.io/en/stable/configuration.html> for more details.
- Add `Span.set_tag_str()` as an alternative to the overloaded functionality of `Span.set_tag()` when the value can be coerced to unicode text.
- Enable `telemetry <Instrumentation Telemetry>` collection when tracing is enabled.

Bug Fixes

- ASM: only report actor.ip on attack.
- aioredis: added exception handling for <span class="title-ref">CancelledError</span> in the aioredis integration.
- CI Visibility: fixed AppVeyor integration not extracting the full commit message.
- Add iterable methods on TracedCursor. Previously these were not present and would cause iterable usage of cursors in DB API integrations to fail.
- Fix parsing of the `DD_TAGS` environment variable value to include support for values with colons (e.g. URLs). Also fixed the parsing of invalid tags that begin with a space (e.g. `DD_TAGS=" key:val"` will now produce a tag with label `key`, instead of `key`, and value `val`).
- opentracing: don't raise an exception when distributed tracing headers are not present when attempting to extract.
- sqlite3: fix error when using `connection.backup` method.
- Change dependency from `` backport_ipaddress` to ``ipaddress`. Only install`ipaddress\`\` for Python \< 3.7.
- gevent: disable gevent after fork hook which could result in a performance regression.
- profiling: restart automatically on all Python versions.
- profiling: fixes an issue with Gunicorn child processes not storing profiling events.
- wsgi: when using more than one nested wsgi traced middleware in the same app ensure wsgi spans have the correct parenting.

Other Changes

- tracing: add http.route tag to root span for Flask framework.

---

New Features

- graphene: add support for `graphene>=2`. [See the graphql documentation](https://ddtrace.readthedocs.io/en/stable/integrations.html#graphql) for more information.
- Add support for aiobotocore 1.x and 2.x.
- ASM: add user information to traces.
- ASM: collect http client_ip.
- ASM: configure the sensitive data obfuscator.
- ASM: Detect attacks on Pylons body.
- ASM: propagate user id.
- ASM: Support In-App WAF metrics report.
- Collect user agent in normalized span tag `http.useragent`.
- ASM: Detect attacks on XML body (for Django, Pylons and Flask).
- Adds support for Lambda profiling, which can be enabled by starting the profiler outside of the handler (on cold start).
- profiler: collect and export the class name for the wall time, CPU time and lock profiles, when available.
- add DD_PYMONGO_SERVICE configuration
- ASM: Redact sensitive query strings if sent in http.url.
- redis: track the connection client_name.
- rediscluster: add service name configuration with `DD_REDISCLUSTER_SERVICE`
- snowflake: add snowflake query id tag to `sql.query` span

Bug Fixes

- aiohttp_jinja2: use `app_key` to look up templates.
- ASM: (flask) avoid json decode error while parsing request body.
- ASM: fix Python 2 error reading WAF rules.
- ASM: reset wsgi input after reading.
- tracing: fix handling of unicode `_dd.origin` tag for Python 2
- tracing: fix nested web frameworks re-extracting and activating HTTP context propagation headers.
- requests: fix split-by-domain service name when multiple `` signs are present in the url
- profiling: internal use of RLock needs to ensure original threading locks are used rather than gevent threading lock. Because of an indirection in the initialization of the original RLock, we end up getting an underlying gevent lock. We work around this behavior with gevent by creating a patched RLock for use internally.
- profiler: Remove lock for data structure linking threads to spans to avoid deadlocks with the trade-off of correctness of spans linked to threads by stack profiler at a given point in time.
- profiling: fix a possible deadlock due to spans being activated unexpectedly.

---

New Features

- graphql: add tracing for `graphql-core>2`. See [the graphql documentation](https://ddtrace.readthedocs.io/en/stable/integrations.html#graphql) for more information.
- ASM: Detect attacks on Django body.
- ASM: Detect attacks on Flask request cookies
- ASM: Detect attacks on Django request cookies
- ASM: Detect attacks on Pylons HTTP query.
- ASM: Detect attacks on Pylons request cookies
- ASM: detect attacks on Pylons path parameters.
- ASM: Report HTTP method on Pylons framework
- ASM: Collect raw uri for Pylons framework.
- AppSec: collect response headers
- ASM: Detect attacks on Flask body.
- ASM: Detect attacks on path parameters
- The profiler now supports Windows.
- The profiler now supports code provenance reporting. This can be enabled by using the `enable_code_provenance=True` argument to the profiler or by setting the environment variable `DD_PROFILING_ENABLE_CODE_PROVENANCE` to `true`.

Bug Fixes

- flask: add support for `flask>=2.2.0`
- Fixed the environment variable used for log file size bytes to be `DD_TRACE_LOG_FILE_SIZE_BYTES` as documented.
- jinja2: fix handling of template names which are not strings.
- Fixed support for pytest-bdd 6.
- Fixes cases where a pytest test parameter object string representation includes the `id()` of the object, causing the test fingerprint to constantly change across executions.
- wsgi: ignore GeneratorExit Exception in wsgi.response spans
- wsgi: ensures resource and http tags are always set on <span class="title-ref">wsgi.request</span> spans.

Other Changes

- profiler: don't initialize the `AsyncioLockCollector` unless asyncio is
available. This prevents noisy logs messages from being emitted in Python 2.

- docs: Added troubleshooting section for missing error details in the root span of a trace.

---