Schemathesis

:version:`4.0.0-alpha.5 <v4.0.0-alpha.4...v4.0.0-alpha.5>` - 2025-02-23
-----------------------------------------------------------------------

This release introduces a new phase management system for CLI that simplifies test execution control and separates unit testing into different stages.

Phase configuration changes:

- ``examples`` (formerly ``explicit``): Runs examples specified in the API schema
- ``fuzzing`` (formerly ``generate``): Testing with randomly generated test cases
- ``coverage``: Deterministic testing of schema constraints and boundary values
- ``reuse`` and ``shrink`` remain enabled by default. Disable via ``--generation-database=none`` and ``--no-shrink``.
- ``target`` phase available via ``--generation-maximize=<METRIC>``

**NOTE**: Pytest integration does not currently have a way to disable the coverage phase. Python API support is planned for future releases.

**Added**

- Warning for 4xx-only operations during unit tests to help identify configuration issues.

**Changed**

- Separate ``coverage`` and ``examples`` into independent testing phases.
- Replace ``--hypothesis-phases`` with ``--phases``.
- Do not report ``unsupported_method`` failure if the API returned HTTP 200 on OPTIONS request.
- Add HTTP 406 status to the list of status codes that are expected for negative test cases.
- The experimental ``--experimental-no-failfast`` option has been stabilized as ``--continue-on-failure``.
This option ensures all test cases within a scenario are executed, even if failures occur.

**Fixed**

- **Coverage phase**: Missing test case metadata leading to some failures not being detected.
- **Coverage phase**: Missing parameter overrides.
- **Coverage phase**: Custom auth implementation not applied to test cases.
- **Coverage phase**: Not applying parameter serialization rules during test generation.
- Incorrect quantifiers merging for some patterns. :issue:`2732`
- Showing ``1 errors`` instead of ``1 error`` in CLI output.

**Removed**

- ``--hypothesis-no-phases``.
- ``--exitfirst``. Use ``--max-failures=1`` instead.

:version:`4.0.0-alpha.4 <v4.0.0-alpha.3...v4.0.0-alpha.4>` - 2025-02-03
-----------------------------------------------------------------------

**Added**

- ``--report`` unified reporting system with multiple format support.
- ``--report-dir`` for centralized report storage.
- Display Open API link definition & extraction errors in CLI output.

**Changed**

- Rename ``--generation-optimize`` to ``--generation-maximize``
- Rename ``--generation-mode`` to ``-m/--mode``
- Rename ``--generation-max-examples`` to ``-n/--max-examples``
- Rename ``--junit-xml`` to ``--report=junit``
- Rename ``--cassette-*`` options to ``--report=vcr/har`` with format-specific paths
- Replace ``shrink`` in ``--hypothesis-phases`` with a separate ``--no-shrink`` option
- Simplify help messages for filtering options.
- Add display of selected operations count in CLI.
- Improve state machine generation by prioritizing reliable API entry points over random operations.
- Make operation filtering independent of API base path for more predictable behavior.
- Improve error message for malformed JSON responses.
- Return ``UNRESOLVABLE`` sentinel instead of an empty string when Open API runtime expressions can't be evaluated (e.g., when ``$response.body/id`` is not found)
- **BREAKING**: The ``validate_response`` method in state machines now accepts the same keyword arguments as ``call``.
If you've overridden this method, update its signature to include ``**kwargs``.

**Fixed**

- Handling of multiple API links pointing to the same operation with different parameters.
- **CLI**: Make exact method filters case-insensitive.
- Internal error in coverage phase when a parameter mixes keywords for different types.
- Skip irrelevant checks for "Unspecified HTTP method" coverage scenarios.
- Handle ``verify=False`` properly when specified via ``get_call_kwargs`` on a state machine. :issue:`2713`
- Preserve test data when unit tests are interrupted via CTRL-C.

**Removed**

- ``--cassette-format`` (replaced by ``--report``).

:version:`4.0.0-alpha.3 <v4.0.0-alpha.2...v4.0.0-alpha.3>` - 2025-01-27
-----------------------------------------------------------------------

**Added**

- Add ``LoadingStarted`` & ``LoadingFinished`` to the public API.
- Display the random generator seed in CLI output.

**Changed**

- Improve control over API calls in stateful testing to make test scenarios more diverse.
- Improve error message in CLI when the schema has no Open API links.
- Improve error message in CLI when the schema contains incorrect Open API links.
- Display the number of covered, selected, and total Open API links in stateful testing output.
- Vary random seed in subsequent state machine re-runs to avoid repeating all previous sequences.

**Fixed**

- Handling of complex regex patterns with multiple quantifiers to respect length constraints during test generation.
- Internal error during the coverage phase if negated parameter has no ``type``.
- Internal error in CLI if run with ``PYTHONIOENCODING`` environment variable that is not ``utf8``.
- Overriding of multiple incoming links defined for the same status code in OpenAPI specification.
- Support for API links where operations can form a cycle (e.g., DELETE -> POST -> DELETE).
- Incorrect Open API link selection when target operations are filtered out.
- False positive in the ``ensure_resource_availability`` check.
- Calculating the number of Open API links selected for testing.

:version:`4.0.0-alpha.2 <v4.0.0a1...v4.0.0-alpha.2>` - 2025-01-20
-----------------------------------------------------------------

**Added**

- Improved visibility into Open API link extraction success/failure status :issue:`823`

**Changed**

- Unified test progress display with multi-spinner interface and single progress bar :issue:`2372`
- Optimized stateful testing by generating only required test data for Open API links.
- Cleaner display of schema loading errors.

**Fixed**

- More accurate exception deduplication based on source location instead of messages.
- Stricter validation of ``--include-*`` and ``--exclude-*`` CLI options.

:version:`4.0.0a1 <v3.39.6...v4.0.0a1>` - 2025-01-15
----------------------------------------------------

I'm releasing Schemathesis 4.0.0a1 - the biggest change in the project's history. I've rewritten major parts of the core engine, Python API, and pytest integration from scratch to enable features that were impossible to implement before.
While this means removing some functionality temporarily, it was necessary to clean up four years of accumulated hacks and create a more solid foundation.

This is an alpha release - expect breaking changes and missing features.
If you're using Schemathesis in production, stick with 3.x for now.
The documentation is outdated, and I'll update it as the new architecture stabilizes.

I'd really appreciate your feedback at `this GitHub Discussion <https://github.com/schemathesis/schemathesis/discussions/2677>`_ - it will help shape the path to stable 4.0.
A detailed migration guide and complete changelog will follow.

**Added**

- New test phases system with `--phases` CLI option to control unit & stateful testing.

**Performance**

- Up to 3x faster test execution.
- Up to 15x lower memory usage.

**Changed**

- Schema loaders reorganized with namespaces:
- ``schemathesis.from_uri`` → ``schemathesis.openapi.from_url``
- ``schemathesis.from_pytest_fixture`` → ``schemathesis.pytest.from_fixture``
- Response handling: Custom checks now receive Schemathesis' ``Response`` class instead of ``requests.Response``.
- Payload serialization: Decorators per transport replace single class implementation.
- CLI: Updated header & summary style.
- Sanitization: Direct arguments in ``schemathesis.sanitization.configure`` instead of ``Config`` instance.

**Renamed CLI Options**

- `--data-generation-methods` → `--generation-mode`
- `--targets` → `--generation-optimize`
- `--hypothesis-derandomize` → `--generation-deterministic`
- `--hypothesis-database` → `--generation-database`
- `--hypothesis-seed` → `--generation-seed`
- `--contrib-unique-data` → `--generation-unique-inputs`
- `--hypothesis-max-examples` → `--generation-max-examples`
- `--sanitize-output` → `--output-sanitize`
- `--hypothesis-suppress-health-check` → `--suppress-health-check`

**Removed**

- `aiohttp` integration.
- Old-style stateful runner (new one is now default).
- Schemathesis.io integration & ``--report`` option (local HTML reports coming later).
- FastAPI fixups.
- Python code samples (only cURL now).
- Python 3.8 support.
- Support for ``pytest<7.0``.
- CLI Options: ``--endpoint``, ``--method``, ``--tag``, ``--operation-id``, ``--skip-deprecated-operations``,
``--show-trace``, ``--debug-output-file``, ``--hypothesis-deadline``, ``--hypothesis-report-multiple-bugs``,
``--hypothesis-verbosity``, ``--store-network-log``, ``--pre-run``, ``--dry-run``, ``--contrib-openapi-formats-uuid``,
``--validate-schema``.
- Most loader configuration moved to ``schema.configure`` method.
- ``add_case`` hook.
- ``schemathesis.contrib.unique_data``.
- Single argument ``AuthProvider.get``.
- ``schemathesis.runner.prepare`` (use ``schemathesis.engine.from_schema``).
- ``schemathesis replay`` command.
- Stateful testing summary (coming later).
- ``SCHEMA_ANALYSIS`` experimental feature.

:version:`3.39.9 <v3.39.8...v3.39.9>` - 2025-02-01
--------------------------------------------------

**Fixed**

- Internal error in coverage phase when a parameter is mixing keywords for different types.
- Do not run irrelevant checks on "Unspecified HTTP method" type of coverage scenarios.
- Ignoring ``verify=False`` when specified via ``get_call_kwargs`` on a state machine. :issue:`2713`

**Changed**

- Slightly improve the error message on malformed JSON.