Scriptworker

----------------------

.. _added-70:

Added
~~~~~

- ``verify_signed_tag``, which verifies the tag’s signature and makes
sure we’re updated to it.

.. _changed-65:

Changed
~~~~~~~

- ``rebuild_gpg_homedirs`` now uses git tags instead of checking for
signed commits.
- ``get_git_revision`` now takes a ``ref`` kwarg; it finds the revision
for that ref (e.g., tag, branch).
- ``update_signed_git_repo`` ``revision`` kwarg is now named ``ref``.
It also verifies and updates to the signed git tag instead of
``ref``.
- ``update_signed_git_repo`` now returns a tuple (revision, tag)
- ``build_gpg_homedirs_from_repo`` now uses ``verify_signed_tag``
instead of ``verify_signed_git_commit``, and takes a new ``tag`` arg.

.. _fixed-58:

Fixed
~~~~~

- the curl command in ``Dockerfile.gnupg`` now retries on failure.

.. _removed-28:

Removed
~~~~~~~

- ``verify_signed_git_commit_output``
- ``verify_signed_git_commit``

----------------------

.. _added-71:

Added
~~~~~

- beetmover and balrog scriptworker support in chain of trust
verification
- ``cot_restricted_trees`` config, which maps branch-nick to branches

.. _changed-66:

Changed
~~~~~~~

- Changed ``cot_restricted_scopes`` to be a scope to branch-nick dict,
indexed by ``cot_product``

.. _fixed-59:

Fixed
~~~~~

- nuke then move the tmp gpg homedir, rather than trying to [wrongly]
use ``overwrite_gpg_home`` on a parent dir

----------------------

.. _added-72:

Added
~~~~~

- Dockerfiles: one for general testing and one for gpg homedir testing,
with readme updates
- ``flake8_docstrings`` in tox.ini
- log chain of trust verification more verbosely, since we no longer
have real artifacts uploaded alongside

.. _changed-67:

Changed
~~~~~~~

- download cot artifacts into ``work_dir/cot`` instead of
``artifact_dir/public/cot``, to avoid massive storage dups
- ``download_artifacts`` now returns a list of full paths instead of
relative paths. Since ``upstreamArtifacts`` contains the relative
paths, this should be more helpful.
- ``contextual_log_handler`` now takes a ``logging.Formatter`` kwarg
rather than a log format string.

.. _changed-68:

Changed
~~~~~~~

- check for a new gpg homedir before ``run_loop``, because puppet will
now use ``rebuild_gpg_homedirs``

.. _fixed-60:

Fixed
~~~~~

- updated all docstrings to pass ``flake8_docstrings``
- switched to a three-phase lockfile for gpg homedir creation to avoid
race conditions (locked, ready, unlocked)
- catch ``aiohttp.errors.DisconnectedError`` and
``aiohttp.errors.ClientError`` in ``run_loop`` during
``upload_artifacts``
- compare the built docker-image tarball hash against
``imageArtifactHash``

.. _removed-29:

Removed
~~~~~~~

- the ``create_initial_gpg_homedirs`` entry point has been removed in
favor of ``rebuild_gpg_homedirs``.

----------------------

.. _changed-69:

Changed
~~~~~~~

- ``scriptworker.cot.verify.raise_on_errors`` now takes a kwarg of
``level``, which defaults to ``logging.CRITICAL``. This is to support
fuzzy task matching, where not matching a task is non-critical.
- ``scriptworker.cot.verify.verify_link_in_task_graph`` now supports
fuzzy task matching. If the Link’s ``task_id`` isn’t in the task
graph, try to match the task definition against the task graph
definitions, and throw ``CoTError`` on failure. This is to support
Taskcluster retriggers.
- ``verify_cot`` is now an entry point, rather than a helper script in
``scriptworker/test/data/``.

.. _fixed-61:

Fixed
~~~~~

- allowed for ``USE_SCCACHE`` as a build env var

----------------------

.. _added-73:

Added
~~~~~

- ``scriptworker.cot.verify`` now verifies the chain of trust for the
graph.
- ``scriptworker.exceptions.CoTError`` now marks chain of trust
validation errors.
- ``scriptworker.task.get_task_id``, ``scriptworker.task.get_run_id``,
``scriptworker.task.get_decision_task_id``,
``scriptworker.task.get_worker_type``
- ``scriptworker.log.contextual_log_handler`` for short-term logs
- added framework for new docs

.. _changed-70:

Changed
~~~~~~~

- config files are now yaml, to enable comments.
``config_example.json`` and ``cot_config_example.json`` have been
consolidated into ``scriptworker.yaml.tmpl``. ``context.cot_config``
items now live in ``context.config``.
- ``validate_artifact_url`` now takes a list of dictionaries as rules,
leading to more configurable url checking.
- ``scriptworker.cot`` is now ``scriptworker.cot.generate``. The
``get_environment`` function has been renamed to
``get_cot_environment``.
- ``scriptworker.gpg.get_body`` now takes a ``verify_sig`` kwarg.
- ``download_artifacts`` now takes ``valid_artifact_task_ids`` as a
kwarg.
- ``max_connections`` is now ``aiohttp_max_connections``
- scriptworker task definitions now expect an ``upstreamArtifacts``
list of dictionaries

.. _fixed-62:

Fixed
~~~~~

- docstring single backticks are now double backticks
- catch aiohttp exceptions on upload

.. _removed-30:

Removed
~~~~~~~

- removed all references to ``cot_config``
- removed the credential update, since puppet restarts scriptworker on
config change.

.. _section-120:

--------------------

.. _added-74:

Added
~~~~~

- ``gpg_lockfile`` and ``last_good_git_revision_file`` in config
- ``get_last_good_git_revision`` and ``write_last_good_git_revision``
now return the last good git revision, and write it to
``last_good_git_revision_file``, respectively.
- ``get_tmp_base_gpg_home_dir`` is a helper function to avoid
duplication in logic.
- ``rebuild_gpg_homedirs`` is a new entry point script that allows us
to recreate the gpg homedirs in a tmpdir, in a separate process
- ``is_lockfile_present``, ``create_lockfile``, and ``rm_lockfile`` as
helper functions for the two gpg homedir entry points.

.. _changed-71:

Changed
~~~~~~~

- ``sign_key``, ``rebuild_gpg_home_flat``, ``rebuild_gpg_home_signed``,
``build_gpg_homedirs_from_repo`` are no longer async.
- ``overwrite_gpg_home`` only keeps one backup.
- ``update_signed_git_repo`` now returns the latest git revision,
instead of a boolean marking whether the revision is new or not. This
will help avoid the scenario where we update, fail to generate the
gpg homedirs, and then stay on an old revision until the next push.
- ``update_logging_config`` now takes a ``file_name`` kwarg, which
allows us to create new log files for the ``rebuild_gpg_homedirs``
and ``create_initial_gpg_homedirs`` entry points.

.. _fixed-63:

Fixed
~~~~~

- ``build_gpg_homedirs_from_repo`` now waits to verify the contents of
the updated git repo before nuking the previous base gpg homedir.
- ``create_initial_gpg_homedirs`` now creates a logfile

.. _removed-31:

Removed
~~~~~~~

- ``rebuild_gpg_homedirs_loop`` is no longer needed, and is removed.

.. _section-121: