Scriptworker

--------------------

.. _added-68:

Added
~~~~~

- ``scriptworker.artifacts`` is a new submodule that defines artifact
behavior
- we now support ``pushapk`` scriptworker instance types in
``cot.verify``

.. _changed-63:

Changed
~~~~~~~

- ``freeze_values`` is now ``get_frozen_copy``, and now returns a
frozen copy instead of modifying the object in place.
- ``unfreeze_values`` is now ``get_unfrozen_copy``
- ``check_config`` now calls ``get_frozen_copy`` on the ``config``
before comparing against ``DEFAULT_CONFIG``
- ``create_config`` calls ``get_unfrozen_copy``, resulting in a
recursively frozen config
- ``DEFAULT_CONFIG`` now uses ``frozendict``\ s and ``tuple``\ s in
nested config items.
- ``.asc`` files are now forced to ``text/plain``
- all ``text/plain`` artifacts are now gzipped, including .log, .asc,
.json, .html, .xml
- we no longer have ``task_output.log`` and ``task_error.log``.
Instead, we have ``live_backing.log``, for more
treeherder-friendliness

.. _removed-27:

Removed
~~~~~~~

- stop testing for task environment variables. This is fragile and
provides little benefit; let’s push on `bug
1328719 <https://bugzilla.mozilla.org/show_bug.cgi?id=1328719>`__
instead.

----------------------

.. _added-69:

Added
~~~~~

- ``unfreeze_values``, to unfreeze a ``freeze_values`` frozendict.

.. _changed-64:

Changed
~~~~~~~

- ``freeze_values`` now recurses.

.. _fixed-56:

Fixed
~~~~~

- delete azure queue entries on status code 409 (already claimed or
cancelled). This allows us to clean up cancelled tasks from the
queue, speeding up future polling.
- more retries and catches in ``find_task``, making it more robust.

----------------------

.. _fixed-57:

Fixed
~~~~~

- balrog tasks are now verifiable in chain of trust.

----------------------

.. _added-70:

Added
~~~~~

- ``verify_signed_tag``, which verifies the tag’s signature and makes
sure we’re updated to it.

.. _changed-65:

Changed
~~~~~~~

- ``rebuild_gpg_homedirs`` now uses git tags instead of checking for
signed commits.
- ``get_git_revision`` now takes a ``ref`` kwarg; it finds the revision
for that ref (e.g., tag, branch).
- ``update_signed_git_repo`` ``revision`` kwarg is now named ``ref``.
It also verifies and updates to the signed git tag instead of
``ref``.
- ``update_signed_git_repo`` now returns a tuple (revision, tag)
- ``build_gpg_homedirs_from_repo`` now uses ``verify_signed_tag``
instead of ``verify_signed_git_commit``, and takes a new ``tag`` arg.

.. _fixed-58:

Fixed
~~~~~

- the curl command in ``Dockerfile.gnupg`` now retries on failure.

.. _removed-28:

Removed
~~~~~~~

- ``verify_signed_git_commit_output``
- ``verify_signed_git_commit``

----------------------

.. _added-71:

Added
~~~~~

- beetmover and balrog scriptworker support in chain of trust
verification
- ``cot_restricted_trees`` config, which maps branch-nick to branches

.. _changed-66:

Changed
~~~~~~~

- Changed ``cot_restricted_scopes`` to be a scope to branch-nick dict,
indexed by ``cot_product``

.. _fixed-59:

Fixed
~~~~~

- nuke then move the tmp gpg homedir, rather than trying to [wrongly]
use ``overwrite_gpg_home`` on a parent dir

----------------------

.. _added-72:

Added
~~~~~

- Dockerfiles: one for general testing and one for gpg homedir testing,
with readme updates
- ``flake8_docstrings`` in tox.ini
- log chain of trust verification more verbosely, since we no longer
have real artifacts uploaded alongside

.. _changed-67:

Changed
~~~~~~~

- download cot artifacts into ``work_dir/cot`` instead of
``artifact_dir/public/cot``, to avoid massive storage dups
- ``download_artifacts`` now returns a list of full paths instead of
relative paths. Since ``upstreamArtifacts`` contains the relative
paths, this should be more helpful.
- ``contextual_log_handler`` now takes a ``logging.Formatter`` kwarg
rather than a log format string.

.. _changed-68:

Changed
~~~~~~~

- check for a new gpg homedir before ``run_loop``, because puppet will
now use ``rebuild_gpg_homedirs``

.. _fixed-60:

Fixed
~~~~~

- updated all docstrings to pass ``flake8_docstrings``
- switched to a three-phase lockfile for gpg homedir creation to avoid
race conditions (locked, ready, unlocked)
- catch ``aiohttp.errors.DisconnectedError`` and
``aiohttp.errors.ClientError`` in ``run_loop`` during
``upload_artifacts``
- compare the built docker-image tarball hash against
``imageArtifactHash``

.. _removed-29:

Removed
~~~~~~~

- the ``create_initial_gpg_homedirs`` entry point has been removed in
favor of ``rebuild_gpg_homedirs``.