Buildbot

Bug fixes
---------

- Fixed crash in `GerritEventLogPoller` when invalid input is passed (7612)
- Fixed `Build` summary containing non obfuscated `Secret` values present in a failed `BuildStep` summary (7833)
- Fixed data API query using buildername to correctly work with buildername containing spaces and unicode characters (7752)
- Fixed an error when master is reconfigured with new builders and a build finishing at this time, causing the build to never finish.
- Fixed crash on master shutdown trying to insubstantiate build's worker when no worker is assigned to the build (7753)
- Fixed confusing error messages in case of HTTP errors that occur when connecting to Gerrit server.
- Fixed `GitPoller` merge commit processing. `GitPoller` now correctly list merge commit files. (7494)
- Fixed hang in `buildbot stop --clean` when a in progress build was waiting on a not yet started BuildRequest that it triggered.
- Improved error message in case of OAuth2 failures.
- Fixed display of navigation links when the web frontend is displayed in narrow window (7818)
- Fixed inconsistent logs data in reports produced by report generators. In particular, `stepname` key is consistently attached to logs regardless if they come with build steps or with the global `logs` key.
- Fixed a regression where a `ChoiceStringParameter` requires a user selected value (no default value), but the force build form incorrectly displays the first choice as being selected which later causes validation error.
- Fixed logs `/raw` and `/raw_inline` endpoint requiring large memory on master (more than full log size) (3011)
- Fixed sidebar group expander to use different icon for expanded groups.
- Log queries in BuildView (`builders/:builderid/builds/:buildnumber`) have been reduced when logs won't be displayed to the user.
- REST API json responses now correctly provide the `Content-Length` header for non-HEAD requests.
- Buildbot is now compatible with SQLAlchemy v2.0+

Changes
-------

- Buildbot will now add a trailing '/' to the `buildbotURL` and `titleURL` configured values if it does not have one.
- The internal API presented by the database connectors has been changed to return data classes instead of Python dictionaries. For backwards compatibility the classes also support being accessed as dictionaries. The following functions have been affected:

- `BuildDataConnectorComponent` `getBuildData`, `getBuildDataNoValue`, and `getAllBuildDataNoValues` now return a `BuildDataModel` instead of a dictionary.
- `BuildsConnectorComponent` `getBuild`, `getBuildByNumber`, `getPrevSuccessfulBuild`, `getBuildsForChange`, `getBuilds`, `_getRecentBuilds`, and `_getBuild` now return a `BuildModel` instead of a dictionary.
- `BuildRequestsConnectorComponent` `getBuildRequest`, and `getBuildRequests` now return a `BuildRequestModel` instead of a dictionary.
- `BuildsetsConnectorComponent` `getBuildset`, `getBuildsets`, and `getRecentBuildsets` now return a `BuildSetModel` instead of a dictionary.
- `BuildersConnectorComponent` `getBuilder` and `getBuilders` now return a `BuilderModel` instead of a dictionary.
- `ChangesConnectorComponent` `getChange`, `getChangesForBuild`, `getChangeFromSSid`, and `getChanges` now return a `ChangeModel` instead of a dictionary.
- `ChangeSourcesConnectorComponent` `getChangeSource`, and `getChangeSources` now return a `ChangeSourceModel` instead of a dictionary.
- `LogsConnectorComponent` `getLog`, `getLogBySlug`, and `getLogs` now return a `LogModel` instead of a dictionary.
- `MastersConnectorComponent` `getMaster`, and `getMasters` now return a `MasterModel` instead of a dictionary.
- `ProjectsConnectorComponent` `get_project`, `get_projects`, and `get_active_projects` now return a `ProjectModel` instead of a dictionary.
- `SchedulersConnectorComponent` `getScheduler`, and `getSchedulers` now return a `SchedulerModel` instead of a dictionary.
- `SourceStampsConnectorComponent` `getSourceStamp`, `get_sourcestamps_for_buildset`, `getSourceStampsForBuild`, and `getSourceStamps` now return a `SourceStampModel` instead of a dictionary.
- `StepsConnectorComponent` `getStep`, and `getSteps` now return a `StepModel` instead of a dictionary.
- `TestResultsConnectorComponent` `getTestResult`, and `getTestResults` now return a `TestResultModel` instead of a dictionary.
- `TestResultSetsConnectorComponent` `getTestResultSet`, and `getTestResultSets` now return a `TestResultSetModel` instead of a dictionary.
- `UsersConnectorComponent` `getUser`, `getUserByUsername`, and `getUsers` now return a `UserModel` instead of a dictionary.
- `WorkersConnectorComponent` `getWorker`, and `getWorkers` now return a `WorkerModel` instead of a dictionary.
- `Git` step no longer includes `-t` (tags) option when fetching by default. Explicitly enabling with `tags=True` is now required to achieve the same functionality.
- `logCompressionMethod` will default to `zstd` if the `buildbot[zstd]` extra set was installed (otherwise, it default to `gzip` as before).
- Buildbot now requires `treq` package to be installed.
- Buildbot worker will now run process in `JobObject` on Windows, so child processes can be killed if main process dies itself either intentionally or accidentally.
- Worker docker image now uses Debian 12.
- Settings UI has been improved by reducing group header size and adding space between groups.

Features
--------

- `copy-db` script now reads/writes in parallel and in batches. This results in it being faster and having smaller memory footprint
- Added possibility to set `START_TIMEOUT` via environment variable.
- Added data API `/workers/n:workerid/builders` allowing to query the Builders assigned to a worker
- The `db_url` config value can now be a renderable, allowing usage of secrets from secrets providers. eg. `util.Interpolate("postgresql+psycopg2://db_user:%(secret:db_password)sdb_host:db_port/db_name")`
- Added `tooltip` parameter to the forcescheduler, allowing passing help text to web frontend to explain the user what the parameters mean.
- `Git` and `GitPush` steps and `GitPoller` change source now support authentication with username/password. Credentials can be provided through the `auth_credentials` and/or `git_credentials` parameters.
- `Git` step `getDescription` configuration now supports the first-parent and exclude arguments.
- `Git` step now honors the shallow option in fetching in addition to clone and submodules.
- Github change hooks how have access to `full_name` of the repository when rendering GitHub tokens.
- Implemented simpler way to perform HTTP requests via `httpclientservice.HTTPSession`. It does not require a parent service.
- `logCompressionMethod` can now be set to `br` (using brotli, requires the `buildbot[brotli]` extra) or `zstd` (using zstandard, requires the `buildbot[zstd]` extra)
- Buildbot now compress REST API responses with the appropriate `accept-encoding` header is set. Available encodings are: gzip, brotli (requires the `buildbot[brotli]` extra), and zstd (requires the `buildbot[zstd]` extra)
- Added `max_lines` parameter to the shell command, allowing processes to be terminated if they exceed a specified line count.
- The `want_logs_content` argument of message formatters now supports being passed a list of logs for which to load the content.
- Exposed log URLs as `url`, `url_raw`, `url_raw_inline` in the log dictionary generated by report generators.
- `TestBuildStepMixin` now supports testing multiple steps added via `setup_step()` in a single unit test.
- Worker base directory has been exposed as a normal build property called `basedir`.
- Show build and step start and stop times when hovering on duration in build step table.
- The following website URLs now support receiving `buildername` instead of `builderid` to select the builder: `builders/:builderid`, `builders/:builderid/builds/:buildnumber`, and `builders/:builderid/builds/:buildnumber/steps/:stepnumber/logs/:logslug`.
- Human readable time is now shown in addition to timestamp in various debug tabs in the web frontend.
- `ChoiceStringParameter` can now have both `multiple=True` and `strict=False` allowing to create values in the web UI.
- Buildrequests tables in various places in the web UI now have a button to load more items.
- Added a way to configure sidebar menu group expand behavior in web frontend.
- Web UI's Worker view (`workers/{workerId}`) now has a `Builders` tab showing Builders configured on the worker
- Builders view now paginates builders list. Page size can be configured with the setting 'Builders page related settings > Number of builders to show per page'.
- Workers view now paginates workers list. Page size can be configured with the setting 'Workers page related settings > Number of workers to show per page'.
- Workers view now includes a search box to filter on worker's name.

Deprecations and Removals
-------------------------

- Buildbot worker no longer supports Python 3.4, 3.5 and 3.6. Older version of Buildbot worker should be used in case it needs to run on these old versions of Python. Old versions of Buildbot worker are fully supported by Buildbot master.
- `buildbot.db.test_results.TestResultDict` is deprecated in favor of `buildbot.db.test_results.TestResultModel`.
- `buildbot.db.test_result_sets.TestResultSetDict` is deprecated in favor of `buildbot.db.test_result_sets.TestResultSetModel`.
- `buildbot.db.buildrequests.BrDict` is deprecated in favor of `buildbot.db.buildrequests.BuildRequestModel`.
- `buildbot.db.build_data.BuildDataDict` is deprecated in favor of `buildbot.db.build_data.BuildDataModel`.
- `buildbot.db.changes.ChDict` is deprecated in favor of `buildbot.db.changes.ChangeModel`.
- `buildbot.db.masters.MasterDict` is deprecated in favor of `buildbot.db.masters.MasterModel`.
- `buildbot.db.sourcestamps.SsDict` is deprecated in favor of `buildbot.db.sourcestamps.SourceStampModel`.
- `buildbot.db.users.UsDict` is deprecated in favor of `buildbot.db.users.UserModel`.
- The following methods of `httpclientservice.HTTPClientService` have been deprecated: `get`, `delete`, `post`, `put`, `updateHeaders`. Use corresponding methods from `HTTPSession`.
- The `add_logs` argument of `BuildStatusGenerator`, `BuildStartEndStatusGenerator` and `BuildSetStatusGenerator` has been removed. As a replacement, set `want_logs_content` of the passed message formatter.
- The `build_files`, `worker_env` and `worker_version` arguments of `TestBuildStepMixin.setup_step()` have been deprecated. As a replacement, call `TestBuildStepMixin.setup_build()` before `setup_step`.
- The `step` attribute of `TestBuildStepMixin` has been deprecated. As a replacement, call `TestBuildStepMixin.get_nth_step()`.
- Master running with Twisted >= 24.7.0 does not work with buildbot-worker 0.8. Use Twisted 24.3.0 on master if you need to communicate with buildbot-worker 0.8. This may be fixed in the future.
- Buildbot master now requires Twisted 22.1.0 or newer.

Bug fixes
---------

- Fixed missing builder force scheduler route `/builders/:builderid/force/:scheduler`.
- Fixed URL of WGSI dashboards to keep backward compatibility with the old non-React WSGI plugin.
- Fixed display of long property values by wrapping them
- Dropped no longer needed dependency on the ``future`` library

Bug fixes
---------

- Fixed function signature for `CustomAuth.check_credentials`.
- Fixed ReactUI authentication when Buildbot is hosted behind a reverse proxy not at url's root. (7814)
- Made Tags column in Builders page take less space when there are no tags
- Fixed cropped change author avatar image in web UI.
- Fixed pluralization of build count in build summaries in the web UI.
- The change details page no longer requires an additional mouse click to show the change details.
- Fixed showing misleading "Loading" spinner when a change has no builds.
- Fixed too small spacing in change header text in web UI.
- Fixed showing erroneous zero changes count in the web UI when loading changes.
- Cleaned up build and worker tabs in builders view in web UI.
- Fixed links to external URLs in the about pages.
- Fixed missing warnings on old browsers.
- Builds in the home page are now sorted to show the latest one first.
- Fixed loading of plugins settings (e.g. from master's `ui_default_config`)
- Improved visual separation between pending build count and build status in trigger build steps in web UI.

Changes
-------

- Buildbot has migrated to `quay.io` container registry for storing released container images. In order to migrate to the new registry container image name in `FROM` instruction in Dockerfiles needs to be adjusted to `quay.io/buildbot/buildbot-master` or `quay.io/buildbot/buildbot-worker`.
- GitHubStatusPush will now render github tokens right before the request. This allow to update the token in the configuration file without restarting the server, which is useful for Github App installations where tokens are rotated every hour.
- The list of supported browsers has been updated to Chrome >=80, Firefox >= 80, Edge >=80, Safari >= 14, Opera >=67.

Features
--------

- The text displayed in build links is now configurable and can use any build property. It was showing build number or branch and build number before.
- Changes and builds tables in various places in the web UI now have a button to load more items.

Bug fixes
---------

- Fixed ``GitPoller`` when repourl has the port specified (7822)
- Fixed ``ChoiceStringParameter`` fields with ``multiple`` would not store the selected values
- Fixed unnecessary trimming of spaces in logs showed in the web UI (7774)
- Fixed favicon colors on build views in the web UI
- Fixed the icon on the about page in the web UI
- Fixed a regression where builds in waterfall view were no longer linking to the build page.
- Fixed an issue that would cause non-ui www plugins to not be configured (such as buildbot-badges) (7665)

Changes
-------

- Buildbot will now error when configured with multiple services of the same type are configured with the same name (6987)

Bug fixes
---------

- Transfer build steps (`FileUpload`, `DirectoryUpload`,
`MultipleFileUpload`, `FileDownload`, and `StringDownload`) now
correctly remove destination on failure, no longer leaving partial content (2860)
- Fixed ReactUI when Buildbot is hosted behind a reverse proxy not at url's root (7260,
7746)
- Fixed results color shown on builder's header in waterfall view
- Fixed cases where waterfall view could be squashed to a pixel high
- Improved flexibility of ``scaling_waterfall`` setting to support floating-point values for more
condensed view.
- Fixed broken theming in web frontend when not using it via ``base_react`` plugin name.
- Fixed ``/builders/n:builderid/builds/n:build_number/properties`` endpoint returning results
for wrong builds.
- Fixed useless logged ``fatal Exception on DB: log with slug ... already exists in this step``
errors.

Bug fixes
---------

- ``BitbucketServerCoreAPIStatusPush`` now handles epoch time format in events as well as `datetime.datetime`.
- Fixed buildrequest cancel requests being ignored under certain infrequent conditions.
- Fixed an issue in lock handling which caused step locks to be acquired in excess of their configured capacity under certain conditions (5655, 5987).
- ``OldBuildCanceller`` will now cancel builds only if a superseding buildrequest is actually created. Previously it was enough to observe a superseding change even if it did not result in actually running builds.
- Fixed ``OldBuildCanceller`` crashes when sourcestamp with no branch was ingested.
- Fixed ``ChoiceStringParameter`` fields being not present in ForceBuild Form.
- Fixed initialization of default web setting values from config.
- Fixed loading of user saved settings in React web frontend.

Changes
-------

- Added optional ``locks_acquired_at`` argument to ``master.data.updates.set_step_locks_acquired_at()``.
- Master and Worker packages have stopped using the deprecated ``distutils`` package and rely on setuptools. Worker installation now requires setuptools.
- Events between ``GerritChangeSource`` and ``GerritEventLogPoller`` are no longer deduplicated. Use ``GerritChangeSource`` with both SSH and HTTP API configured as a replacement.
- ``GitPoller`` no longer track the ``master`` branch when neither ``branch`` nor ``branches`` arguments are provided. It now track the remote's default branch.
- Improved performance of ``OpenstackWorker`` startup when there are large number of images on the server.
- ``buildbot.www.plugin.Application`` no longer accepts module name as the first parameter. It requires the name of package. In most cases where ``__name__`` was being passed, ``__package__`` is now required.
- Padding of the UI elements in the web UI has been reduced to increase density of presented data.
- Buildbot now requires SQLAlchemy 1.4.0 or newer.
- Old ``importlib_resources`` is no longer used.

Features
--------

- Added ``rebuilt_buildid`` key-value pair to buildsets to be able to keep track on which build is being rebuild.
- Buildbot now tracks total time that has been spent waiting for locks in a build.
- Added ``projectid`` and ``projectname`` properties to Build
- The ``worker_preparation`` dummy step that tracks various build startup overhead has been split into two steps to track worker startup and locks wait times separately.
- Builds now have ``builderid`` property.
- Build request cancellation has been exposed to the Data API.
- Added optional ``started_at`` and ``locks_acquired`` arguments to ``master.data.updates.startStep()``.
- ``buildbot.test.fake.httpclientservice.HTTPClientService`` now can simulate network and processing delays via ``processing_delay_s`` option to ``expect()`` method.
- Added ability to poll HTTP event API of Gerrit server to ``GerritChangeSource``. This has the following advantages compared to simply pointing ``GerritChangeSource`` and ``GerritEventLogPoller`` at the same Gerrit server:

- All events are properly deduplicated
- SSH connection is restarted in case of silent hangs of underlying SSH connection (this may happen even when ServerAliveInterval is used)

- Added ``select_next_worker`` global configuration key which sets default ``nextWorker`` customization hook on all builders.
- Added support for connecting Kubernetes workers to multiple Kubernetes clusters.
- Raw logs downloaded from the web UI now include full identifying information in the filename.
- Raw text logs downloaded from the web UI now include a small header with identifying information.
- The Rebuild button on a Build's view now redirect to the Buildrequest corresponding to the latest rebuild.
- Add a "Workers" tab to the Builder view listing workers that are assigned to this builder (7162)
- Added check for correct argument types to ``BuildStep`` and ``ShellCommand`` build steps and all steps deriving from ``ShellMixin``. This will avoid wrong arguments causing confusing errors in unrelated parts of the codebase.
- Implemented a check for step attempting to acquire the same lock as its build.
- Implement support for customizing ``affinity`` and ``nodeSelector`` fields in Kubernetes pod spec for Kubernetes worker.
- The debug tab in build page now shows previous build that has been built on the same worker for the same builder. This helps debugging any build directory cleanup problems in custom Buildbot setups.
- Add support for case insensitive search within the logs.
- Add support for regex search within the logs.


Deprecations and Removals
-------------------------

- ``buildbot.process.factory.Distutils`` factory has been deprecated.
- ``HashiCorpVaultSecretProvider`` has been removed.
- ``GerritStatusPush`` no longer accepts deprecated arguments: ``reviewCB``, ``startCB``, ``reviewArg``, ``startArg``, ``summaryCB``, ``summaryArg``, ``builders``, ``wantSteps``, ``wantLogs``.
- Deprecated module-level attributes have been deleted.
- ``GerritStatusPush`` callback functions now can only return dictionary type.
- AngularJS web frontend has been removed.
- Deprecated ``LineBoundaryFinder callback`` argument has been removed.
- Removed Python 2.7 support on the worker. This does not affect compatibility of connecting workers running old versions of Buildbot to masters running new versions of Buildbot.