**Migration Guide: Upgrading from dd-trace-py v2 to v3**
We are happy to announce the release of v3.0.0 of ddtrace. This release drops support for Python 3.7 and many previously-deprecated parts of the library interface.
The 2.x release line enters Maintenance Mode with the release of 3.0.0, meaning that it will only receive bugfix changes on its last few minor releases, and 2.21 will be the last 2.x minor version. See the [versioning policy](https://docs.datadoghq.com/tracing/trace_collection/compatibility/python/#releases) for support level definitions.
---
**Summary of Significant Changes**
* Removed support for Python 3.7 from the library
**CI Visibility**
* The new pytest plugin is now the default.
* Module, suite, and test names are now parsed from `item.nodeid`.
* Test names now include the class name (`TestClass::test_method`) for class-based tests.
* Test skipping is now done at the suite level.
* The `DD_PYTEST_USE_NEW_PLUGIN_BETA` environment variable is no longer used.
**LLM Observability**
* Removed support for `DD_LLMOBS_APP_NAME`.
* The `parameters` argument in `LLMObs.annotate()` has been removed. Use `metadata` instead.
* Dropped support for OpenAI 0.x and Langchain v0.0.x.
**Tracing**
* Removed all deprecated interfaces from integrations.
* `DD_TRACE_PROPAGATION_STYLE=b3 single header` is no longer supported. Use `b3` instead.
* Dropped support for multiple Tracer instances. The global tracer instance `ddtrace.tracer` must be used.
* Removed deprecated parameters from `Tracer.configure(...)`.
* Dropped support for several deprecated tracing configurations. See “Environment Variables” section below for details.
**Recommended Pre-Upgrade Steps**
Before upgrading to v3, check for deprecation warnings in your application using the latest v2 release (**2.21.0**):
**Detect Deprecation Warnings in Tests**
Enable warnings as errors in `pytest`:
pytest -W "error::ddtrace.DDTraceDeprecationWarning" tests.py
**Detect Deprecation Warnings in Applications**
Set the following environment variable to raise exceptions for deprecated usage:
DD_TRACE_RAISE_DEPRECATIONWARNING=1 ddtrace-run python app.py
Alternatively, enable all deprecation warnings:
PYTHONWARNINGS=all python app.py
**Environment Variables**
The following environment variables have been removed or replaced in v3. Use the table below to update your configuration:
| v2 | Removed in v3 | Replaced/Changed in v3 |
| :---- | :---- | :---- |
| **LLM Observability** | | |
| `DD_LLMOBS_APP_NAME` | | Use `DD_LLMOBS_ML_APP` instead |
| `_DD_LLMOBS_EVALUATOR_SAMPLING_RULES` | | Use `DD_LLMOBS_EVALUATOR_SAMPLING_RULES ` instead |
| `_DD_LLMOBS_EVALUATORS ` | | Use `DD_LLMOBS_EVALUATORS ` instead |
| **Tracing** | | |
| `DD_ANALYTICS_ENABLED` | No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the [ingestion controls documentation](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_mechanisms/?tab=python) for more detail.| |
| `DD_HTTP_CLIENT_TAG_QUERY_STRING` | | Use `DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING`. |
| `DD_TRACE_ANALYTICS_ENABLED` | No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the [ingestion controls documentation](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_mechanisms/?tab=python) for more detail. | |
| `DD_TRACE_METHODS` (using `[]` notation) | | Must use `:` notation instead, For example: `mod.submod:method1,method2;mod.submod:Class.method1`. ([docs](https://ddtrace.readthedocs.io/en/stable/configuration.html?highlight=dd_trace_methods#DD_TRACE_METHODS)) |
| `DD_TRACE_PROPAGATION_STYLE=”b3 single header”` | | Use `DD_TRACE_PROPAGATION_STYLE=b3` for identical behavior |
| `DD_TRACE_SAMPLE_RATE` | | Use `DD_TRACE_SAMPLING_RULES` ([docs](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_mechanisms/?tab=python#in-tracing-libraries-user-defined-rules)). |
| `DD_TRACE_SPAN_AGGREGATOR_RLOCK` | No replacement (feature removed). Config variable is now a no-op. | |
| **CI Visibility** | | |
| `DD_PYTEST_USE_NEW_PLUGIN_BETA` | No replacement (new pytest plugin is not in beta anymore). | |
Use the following patterns to identify the deprecated environment variables in a code base:
git grep -P -e "DD_LLMOBS_APP_NAME" \
-e "_DD_LLMOBS_EVALUATOR_SAMPLING_RULES" \
-e "_DD_LLMOBS_EVALUATORS" \
-e "DD_TRACE_PROPAGATION_STYLE=.*b3 single header" \
-e "DD_TRACE_SAMPLE_RATE" \
-e "DD_TRACE_API_VERSION=v0.3" \
-e "DD_ANALYTICS_ENABLED" \
-e "DD_TRACE_ANALYTICS_ENABLED" \
-e "DD_HTTP_CLIENT_TAG_QUERY_STRING" \
-e "DD_TRACE_SPAN_AGGREGATOR_RLOCK" \
-e "DD_TRACE_METHODS=.*\[\]"
Replace deprecated settings with the recommended alternatives where applicable.
**NOTE**: The changes to environment variables apply only to the configuration of the dd-trace-py library and not the Datadog Agent.
**Legacy Interfaces**
The following methods and module attributes have been removed or changed. Unless noted, removal means that the functionality pointed to by the removed interface has been disabled.
| v2 | v3 |
| :---- | :---- |
| **Tracing** | |
| `Tracer.configure( enabled=foo, hostname=foo, port=foo, uds_path=foo, https=foo, sampler=foo, wrap_executor=foo, priority_sampling=foo, settings=foo, dogstatsd_url=foo, writer=foo, partial_flush_enabled=foo, partial_flush_min_spans=foo, api_version=foo )` | `Tracer.configure()` |
| `ddtrace.opentracer.tracer.Tracer()._dd_tracer` | `ddtrace.tracer` |
| `ddtrace.tracer.sampler.rules[].choose_matcher(lambda: True)` | `ddtrace.tracer.sampler.rules[].choose_matcher(“foo”)` Callables and regex patterns are no longer allowed as arguments to `choose_matcher` |
| `Span.sampled` | `Span.context.sampling_priority > 0` |
| **LLM Observability** | |
| `LLMObs.annotate(parameters=foo)` | `LLMObs.annotate(metadata=foo)` |
**Python 3.13 Support**
Much of the library’s functionality has been compatible with Python 3.13 since version 2.20. Some pieces of functionality do not yet work with Python 3.13. Here’s what does not work under Python 3.13 as of dd-trace-py 3.0. 'Not tested' indicates that while the feature may be compatible with Python 3.13, its functionality has not been verified, as it is not included in our automated test coverage.
* `dd-trace-py` doesn’t work on Windows with Python 3.13
* Appsec Threat Detection is not tested against Django, Flask, or FastAPI with 3.13
* Automatic Service Naming is not tested with 3.13
* The following products are not tested with 3.13:
* Code Coverage
* Appsec IAST
* Data Streams Monitoring
* CI Visibility
* Continuous Profiling
* The following integrations are not tested with 3.13:
* `anthropic`
* `consul`
* `freezegun`
* `gevent`
* `google_generativeai`
* `gunicorn`
* `langchain`
* `mysqlclient`
* `opentracing`
* `psycopg`
* `psycopg2`
* `pymysql`
* `pytest`
* `pytest-bdd`
* `pytest-benchmark`
* `sanic`
* `selenium`
* `sqlite3`
* `starlette`
* `tornado`
* `vertexai`
---
By following these guidelines, you can transition to dd-trace-py v3 gradually, ensuring minimal disruption to your services while taking advantage of the latest features and improvements. Happy migrating!
---
New Features
- Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.
- ASM
- Introduces use of meta struct for appsec event reports. This will improve performances. ASM will now require at least agent version 7.35.0 to work as expected.
- Code Security (IAST)
- Introduces Code Security support to PyMySQL and MySQLDB integrations.
- XSS detection for Django applications and Jinja2 (Flask and FastAPI applications), which will be displayed on your DataDog Vulnerability Explorer dashboard. See the [Application Vulnerability Management](https://docs.datadoghq.com/security/application_security/vulnerability_management/) documentation for more information about this feature.
- Implements the detection of the Stacktrace-Leak vulnerability for Django, Flask and FastAPI.
- LLM Observability
- `valkey`: Adds automatic instrumentation of the Valkey package. Thank you [AhmadMasry](<https://github.com/AhmadMasry>)!
- `openai`: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.
- Introduces an integration with the [RAGAS](https://docs.ragas.io/en/stable/) evaluation framework to continuously monitor the performance of context-augmented LLM generations in production.
The integration supports evaluating LLM inferences with the following RAGAS metrics:
- [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/): measures if the LLM response is faithful to the provided context.
- [Answer Relevancy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/): measures how relevant the LLM response is to the user input.
- [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/): measures how effectively the context is used in the generated response.
To learn more, see the [LLM Observability evaluations guide](https://docs.datadoghq.com/llm_observability/submit_evaluations/).
Upgrade Notes
- CI Visibility
- Official release of the new version of the pytest plugin, introducing the following features:
- [Auto Test Retries](https://docs.datadoghq.com/tests/flaky_test_management/auto_test_retries)
- [Early Flake Detection](https://docs.datadoghq.com/tests/flaky_test_management/early_flake_detection)
- Improved coverage collection for [Test Impact Analysis](https://docs.datadoghq.com/tests/test_impact_analysis) (formerly Intelligent Test Runner), using an internal collection method instead of [coverage.py](https://github.com/nedbat/coveragepy), with improved dependency discovery.
**NOTE:** this new version of the plugin introduces breaking changes:
- `module`, `suite`, and `test` names are now parsed from the `item.nodeid` attribute
- test names now include the class for class-based tests
- Test skipping by Test Impact Analysis (formerly Intelligent Test Runner) is now done at the suite level, instead of at the test level
A beta version of the plugin had been available since v2.18.0, and could be enabled via the `DD_PYTEST_USE_NEW_PLUGIN_BETA` environment variable. The new version is now the default, and the environment variable is not used anymore.
- LLM Observability
- `langchain`: Removes the `langchain.tokens.total_cost` span metric for OpenAI calls. For continued cost estimation of OpenAI calls, enable [LLM Observability](https://docs.datadoghq.com/llm_observability/).
- `langchain`: Removes prompt-completion log sampling from the LangChain integration. To continue logging prompt completions, enable LLM Observability.
- `langchain`: Removes integration metrics from the LangChain integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.
- `langchain`: removed tracing support for `langchain.chains.base.Chains`. To maintain support, migrate to using Langchain's LCEL instead.
-` langchain`: Drops tracing and LLM Observability support for Langchain v0.0.x versions.
- `openai`: Removes prompt-completion log sampling from the OpenAI integration. To continue logging prompt completions, enable LLM Observability.
- `openai`: Removes integration metrics from the OpenAI integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.
- `openai`: Drops tracing and LLM Observability support for OpenAI 0.x versions. To maintain support, upgrade to `openai>=1.0`.
- Support for the deprecated environment variable `DD_LLMOBS_APP_NAME` has been removed.
- Removes the deprecated `parameters` argument from `LLMObs annotate()`. Use `metadata` instead.
- Profiling
- Bumps libdatadog dependency to v15.0.0.
- Tracing
- configurations: Drops support for deprecated tracing configurations. The following configurations are no longer supported:
- `DD_TRACE_SAMPLE_RATE`, use `DD_TRACE_SAMPLING_RULES` instead.
- `DD_TRACE_API_VERSION`=v0.3, the default `v0.5` version is used instead.
- `DD_ANALYTICS_ENABLED`, Datadog Analytics is no longer supported.
- `DD_TRACE_ANALYTICS_ENABLED`, Datadog Analytics is no longer supported.
- `DD_HTTP_CLIENT_TAG_QUERY_STRING`, `DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING` should be used instead.
- `DD_TRACE_SPAN_AGGREGATOR_RLOCK`, disabling the span aggregator rlock is no longer supported.
- `aws_lambda`: Drops support for `aws_lambda<=6.105.0`.
- configurations: Drops support for configuring `DD_TRACE_METHODS` with the '[]' notation. Ensure `DD_TRACE_METHODS` use the ':' notation instead".
- integrations: Removes deprecated interfaces from all integrations.
- opentracing: Removes the deprecated `ddtracer` parameter from `ddtrace.opentracer.tracer.Tracer()`.
- propagation: Setting `DD_TRACE_PROPAGATION_STYLE` to `b3 single header` is no longer supported. Use `b3` instead.
- sampling: Drops support for configuring sampling rules using functions and regex in the `ddtrace.tracer.sampler.rules[].choose_matcher(...)` method and removes the `timestamp_ns` parameter from `ddtrace.internal.rate_limiter.RateLimiter.is_allowed()`.
- span: Removes the deprecated `Span.sampled` property
- Drops support for multiple Tracer instances in the same process. Use `ddtrace.trace.tracer` to access the global tracer instance.
- Removes support for overriding the global tracer in `ddtrace.trace.Pin`
- Removes deprecated parameters from `Tracer.configure(...)` method and removes the `Tracer.sampler` attribute.
- Drops support for multiple tracer instances, `ddtrace.trace.Tracer` can not be reinitialized.
Deprecation Notes
- ASM
- Deprecated env var DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING is now removed. Please use the new env var DD_APPSEC_AUTO_USER_INSTRUMENTATION_MODE instead.
- LLM Observability
- The `_DD_LLMOBS_EVALUATORS` environment variable is deprecated and will be removed in ddtrace 3.0.0. As an alternative to `_DD_LLMOBS_EVALUATORS`, you can use `DD_LLMOBS_EVALUATORS` instead. To migrate, replace `_DD_LLMOBS_EVALUATORS` with `DD_LLMOBS_EVALUATORS`.
- `_DD_LLMOBS_EVALUATOR_SAMPLING_RULES` environment variable is deprecated and will be removed in ddtrace 3.0.0. As an alternative to `_DD_LLMOBS_EVALUATOR_SAMPLING_RULES`, you can use `DD_LLMOBS_EVALUATOR_SAMPLING_RULES` instead. To migrate, replace `_DD_LLMOBS_EVALUATOR_SAMPLING_RULES` with `DD_LLMOBS_EVALUATOR_SAMPLING_RULES`.
- Tracing
- Moves `ddtrace.provider.BaseContextProvider` to `ddtrace.trace.BaseContextProvider`. The `ddtrace.provider` module is deprecated and will be removed in v3.0.0.
- Deprecates the following constants in `ddtrace.constants` module:
- `ANALYTICS_SAMPLE_RATE_KEY`
- `SAMPLE_RATE_METRIC_KEY`
- `SAMPLING_PRIORITY_KEY`
- `SAMPLING_AGENT_DECISION`
- `SAMPLING_RULE_DECISION`
- `SAMPLING_LIMIT_DECISION`
- `ORIGIN_KEY`
- `USER_ID_KEY`
- `HOSTNAME_KEY`
- `RUNTIME_FAMILY`
- `BASE_SERVICE_KEY`
- `SPAN_MEASURED_KEY`
- `KEEP_SPANS_RATE_KEY`
- `MULTIPLE_IP_HEADERS`
- `CONFIG_ENDPOINT_ENV`
- `CONFIG_ENDPOINT_RETRIES_ENV`
- `CONFIG_ENDPOINT_TIMEOUT_ENV`
- Internalizes the `ddtrace.settings.config` module and deprecates the following `ddtrace.config` attributes:
- `http`, use `DD_TRACE_HEADER_TAGS` environment variable instead.
- `http_server`, use `DD_TRACE_HTTP_SERVER_ERROR_STATUSES` environment variable instead.
- `trace_headers`, this attribute is internal to the tracer.
- `header_is_traced`, this attribute is internal to the tracer.
- `convert_rc_trace_sampling_rules`, this attribute is internal to the tracer.
- `enable_remote_configuration`, use `DD_REMOTE_CONFIGURATION_ENABLED` environment variable instead.
- `get_from`, use `ddtrace.trace.Pin` to set instance level configurations.
- configurations: Removes deprecated attributes from `ddtrace.config`.
Bug Fixes
- ASM
- Ensures that no module from ASM are loaded when ASM is disabled or unavailable.
- Resolves an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.
- CI Visibility
- Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.
- Code security (IAST)
- SCA: Ensures that no module from IAST are loaded when IAST is disabled or unavailable.
- Resolves an issue where the usage of `callonce` decorator could trigger an import loop
- LLM Observability
- `openai`: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
- `vertexai`: Resolves an issue with `chat.send_message()` where the content keyword argument was not parsed correctly.
- Resolves an issue where successive calls to `LLMObs.annotate()` on the same span caused overwriting of metrics, metadata, and prompt dictionaries. Now, calling `LLMObs.annotate()` will merge new metrics/metadata/prompt values with any existing ones.
- Resolves an issue where spans were being enqueued to an inactive evaluator runner which caused noisy logs related to the evaluator runner buffer being full.
- Resolves an issue where explicitly only using `LLMObs.enable()` to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
- Resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
- Resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError.
- Profiling
- Fixes an issue on arm64 macOS where the profiler was not able to find native extension modules to enable the following features: `DD_PROFILING_TIMELINE_ENABLED` and `DD_PROFILING_STACK_V2_ENABLED`.
- Fixes SystemError from the memory profiler returning NULL when collecting events
- Tracing
- `aiohttp`: Adds the environment variable `DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK` to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.
- internal: Fix performance overhead of Python distribution parsing for internal telemetry.
- logging: Resolves an an unneeded info log being logged on process exit due to a forksafe hook being unregistered that was never registered to begin with.
Other Changes
- Tracing
- `cassandra`, `cherrypy`, `flask_cache`, `starlette`: Ensures a deprecation warning is not raised when patching these integrations via `ddtrace-run` and `import ddtrace.auto`.
- Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
- Removes the deprecated tracing modules and constants from the `ddtrace` package.
---