Buildbot

Bug fixes
---------

- Improved statistics capture to avoid negative build duration.
- Improved reliability of "buildbot stop" (3535).
- Cancelled builds now have stop reason included into the state string.
- Fixed `custom_class` change hook checks to allow hook without a plugin.
- Added treq response wrapper to fix issue with missing url attribute.
- Fixed Buildbot Worker being unable to start on Python 2.7 due to issue in a new version of Automat dependency.

Features
--------

- Expanded `ChangeFilter` filtering capabilities:
- New `<attribute>_not_eq` parameters to require no match
- `<attribute>_re` now support multiple regexes
- New `<attribute>_not_re` parameters to require no match by regex
- New `property_<match_type>` parameters to perform filtering on change properties.
- Exposed frontend configuration as implementation-defined JSON document that can be queried separately.
- Added support for custom branch keys to `OldBuildCanceller`. This is useful in Version Control Systems such as Gerrit that have multiple branch names for the same logical branch that should be tracked by the canceller.
- `p4port` argument of the `P4` step has been marked renderable.
- Added automatic generation of commands for Telegram bot without need to send them manually to BotFather.

Deprecations and Removals
-------------------------

- This release includes an experimental web UI written using React framework. The existing web UI is written using AngularJS framework which is no longer maintained. The new web UI can be tested by installing `buildbot-www-react` package and `'base_react': {}` key-value to www plugins. Currently no web UI plugins are supported. The existing web UI will be deprecated on a subsequent Buildbot release and eventually replaced with the React-based web UI on Buildbot 4.0.

Bug fixes
---------

- Fixed handling of last line in logs when Buildbot worker 3.5 and older connects to Buildbot master 3.6 using PB protocol (6632).
- Fixed worker `cpdir` command handling when using PB protocol (6539)

Bug fixes
---------

- Fixed compatibility with Autobahn 22.4.x.
- Fixed a circular import that causes errors in certain cases.
- Fixed issue with :bb:worker:`DockerLatentWorker` accumulating connections with the docker server (6538).
- Fixed documentation build for ReadTheDocs: Sphinx and Python have been updated to latest version.
- Fixed build pending and canceled status reports to GitLab.
- Fixed compatibility of hvac implementation with Vault 1.10.x (6475).
- Fixed a race condition in `PyLint` step that may lead to step throwing exceptions.
- Reporters now always wait for previous report to completing upload before sending another one.
This works around a race condition in GitLab build reports ingestion pipeline (6563).
- Fixed "retry fetch" and "clobber on failure" git checkout options.
- Improved Visual Studio installation path retrieval when using MSBuild and only 'BuildTools' are installed.
- Fixed search for Visual Studio executables by inspecting both `C:\Program Files` and `C:\Program Files (x86)` directories.
- Fixed Visual Studio based steps causing an exception in `getResultSummary` when being skipped.
- Fixed issue where workers would immediately retry login on authentication failure.
- Fixed sending emails when using Twisted 21.2 or newer (5943)

Features
--------

- Implemented support for App password authentication in `BitbucketStatusPush` reporter.
- Cancelled build requests now generate build reports.
- Implemented support for `--no-verify` git option to the `GitCommit` step.
- `HTTPClientService` now accepts full URL in its methods.
Previously only a relative URL was supported.
- Callback argument of class `LineBoundaryFinder` is now optional and deprecated.
- Added `VS2019`, `VS2022`, `MsBuild15`, `MsBuild16`, `MsBuild17` steps.
- Names of transfer related temporary files are now prefixed with `buildbot-transfer-`.
- `buildbot try` now accepts empty diffs and prints a warning instead of rejecting the diff.
- Implemented note event handling in GitLab www hook.

Deprecations and Removals
-------------------------

- Removed support for Python 3.6 from master.
Minimal python version for the master is now 3.7.
The Python version requirements for the worker don't change: 2.7 or 3.4 and newer.
- `buildbot` package now requires Twisted versions >= 18.7.0

Bug fixes
---------

- Improved handling of "The container operating system does not match the host operating system" error on Docker on Windows to mark the build as erroneous so that it's not retried.
- Fixed rare `AlreadyCalledError` exceptions in the logs when worker worker connection is lost at the same time it is delivering final outcome of a command.
- Fixed errors when accessing non-existing build via REST API when an endpoint matching rule with builder filter was present.
- Fixed an error in `CMake` passing options and definitions on the cmake command line.
- Fixed an error when handling command management errors on the worker side (regression since v3.0.0).
- Fixed updating build step summary with mock state changes for MockBuildSRPM and MockRebuild.
- Fixed support for optional `builder` parameter used in RebuildBuildEndpointMatcher (6307).
- Fixed error that caused builds to become stuck in building state until next master restart if builds that were in the process of being interrupted lost connection to the worker.
- Fixed Gerrit change sources to emit changes with proper branch name instead of one containing `refs/heads/` as the prefix.
- Fixed handling of `build_wait_timeout` on latent workers which previously could result in latent worker being shut down while a build is running in certain scenarios (5988).
- Fixed problem on MySQL when using master names or builder tags that differ only by case.
- Fixed timed schedulers not scheduling builds the first time they are enabled with `onlyIfChanged=True` when there are no important changes. In such case the state of the code is not known, so a build must be run to establish the baseline.
- Switched Bitbucket OAuth client from the deprecated 'teams' APIs to the new 'workspaces' APIs
- Fixed errors when killing a process on a worker fails due to any reason (e.g. permission error or process being already exited) (6140).
- Fixed updates to page title in the web UI. Web UI now shows the configured buildbot title within the page title.

Features
--------

- The use of Renderables when constructing payload For `JSONStringDownload` is now allowed.
- Added `alwaysPull` support when using `dockerfile` parameter of `DockerLatentWorker`.
- Base Debian image has been upgraded to Debian Bullseye for the Buildbot master.
- Added rendering support to `docker_host` and `hostconfig` parameters of `DockerLatentWorker`.
- `MailNotifier` reporter now sends HTML messages by default.
- `MessageFormatter` will now use a default subject value if one is not specified.
- The default templates used in message formatters have been improved to supply more information. Separate default templates for html messages have been provided.
- Added `buildbot_title`, `result_names` and `is_buildset` keys to the data passed to `MessageFormatter` instances for message rendering.
- Added `target` support when using `dockerfile` parameter of `DockerLatentWorker`.
- Simplified :bb:cfg:`prioritizeBuilders` default function to make an example easier to customize.
- Buildbot now exposes its internal framework for writing tests of custom build steps. Currently the API is experimental and subject to change.
- Implemented detection of too long step and builder property names to produce errors at config time if possible.

Deprecations and Removals
-------------------------

- Deprecated `subject` argument of `BuildStatusGenerator` and `BuildSetStatusGenerator` status generators. Use `subject` argument of corresponding message formatters.

- Updated Bitbucket API URL for `BitbucketPullrequestPoller`.
- Fixed a crash in `BitbucketPullrequestPoller` (4153).
- Fixed installation of master and worker as Windows service from wheel package (regression since 3.4.0) (6294).
- Fixed occasional exceptions when using Visual Studio steps (5698).
- Fixed rare "Did you maybe forget to yield the method" errors coming from the log subsystem.

Bug fixes
---------

- Database migrations are now handled using Alembic (1.6.0 or newer is required) (5872).
- AMI for latent worker is now set before making spot request to enable dynamically setting AMIs for instantiating workers.
- Fixed `GitPoller` fetch commands timing out on huge repositories
- Fixed a bug that caused Gerrit review comments sometimes not to be reported.
- Fixed a critical bug in the `MsBuild141` step (regression since Buildbot v2.8.0) (6262).
- Implemented renderable support in secrets list of `RemoveWorkerFileSecret`.
- Fixed issues that prevented Buildbot from being used in Setuptools 58 and newer due to dependencies failing to build (6222).

Improved Documentation
----------------------

- Fixed help text for `buildbot create-master` so it states that `--db` option is passed verbatim to `master.cfg.sample` instead of `buildbot.tac`.
- Added documentation of properties available in the formatting context that is presented to message formatters.

Features
--------

- MsBuild steps now handle correctly rebuilding or cleaning a specific project.
Previously it could only be done on the entire solution.
- Implemented support for controlling `filter` option of `git clone`.
- Optimized build property filtering in the database instead of in Python code.
- Implemented support of `SASL PLAIN` authentication to `IRC` reporter.
- The `want_logs` (previously `wantLogs`) argument to message formatters will now imply `wantSteps` if selected.
- Added information about log URLs to message formatter context.
- Implemented a way to ask for only logs metadata (excluding content) in message formatters via `want_logs` and `want_logs_content` arguments.
- Implemented support for specifying pre-processor defines sent to the compiler in the `MsBuild` steps.
- Introduced `HvacKvSecretProvider` to allow working around flaws in `HashiCorpVaultSecretProvider` (5903).
- Implemented support for proxying worker connection through a HTTP proxy.

Deprecations and Removals
-------------------------

- The `wantLogs` argument of message formatters has been deprecated.
Please replace any uses with both `want_logs` and `want_logs_content` set to the same value.
- The `wantProperties` and `wantSteps` arguments of message formatters have been renamed to `want_properties` and `want_steps` respectively.
- Buildbot now requires SQLAlchemy 1.3.0 or newer.