Scriptworker

---------------------

.. _changed-29:

Changed
~~~~~~~

- require py37 to be green
- support and require taskcluster>=4.0.0 (``taskcluster.aio`` rather
than ``taskcluster.async``, because ``async`` is a py37 keyword)

.. _section-63:

---------------------

.. _changed-30:

Changed
~~~~~~~

- tests that need an event loop are now all ``pytest.mark.asyncio``
and/or using the pytest-asyncio ``event_loop`` fixture, rather than
using the now-removed local ``event_loop`` fixture. This addresses
our intermittent test failures, though we need additional work (e.g.,
PR 244)
- added more test cases around
``get_upstream_artifacts_full_paths_per_task_id``, to allow for
multiple ``upstreamArtifacts`` entries for a single ``taskId``

.. _fixed-26:

Fixed
~~~~~

- fixed the hang in ``run_task`` – we were waiting for the
``max_timeout`` future to exit, which it did after sleeping for
``task_max_timeout`` seconds, so every task took the full timeout to
complete. Now we use ``asyncio.wait(timeout=...)``.
- fixed the unclosed session warnings in tests

.. _removed-12:

Removed
~~~~~~~

- removed ``get_future_exception`` after removing its last caller
- removed ``max_timeout`` after moving timeout handling into
``run_task`` via ``asyncio.wait``
- removed the ``event_loop`` test fixture; this may have conflicted
with the ``pytest-asyncio`` ``event_loop`` fixture

.. _section-64:

---------------------

.. _added-35:

Added
~~~~~

- added ``task_max_timeout_status``, ``reversed_statuses``, and
``invalid_reclaim_status`` to ``DEFAULT_CONFIG``
- added ``get_reversed_statuses`` for config-driven reversed statuses
- added ``task.kill_pid`` to kill a process tree
- added ``task.kill_proc`` to kill a subprocess proc
- added unit and integration tests for user cancel
- added ``utils.get_future_exception`` to get the status of a single
future

.. _changed-31:

Changed
~~~~~~~

- integration tests now require the
``queue:cancel-task:test-dummy-scheduler/*`` scope
- unit tests now run in random order
- ``max_timeout`` is now an async function with sleeps rather than a
synchronous function using ``call_later``
- split ``run_tasks`` into several helper functions
- all negative exit statuses now log ``Automation Error``

.. _fixed-27:

Fixed
~~~~~

- task timeouts should result in an ``intermittent-task``, rather than
a crashed scriptworker
- we now kill the task on a ``reclaim_task`` result of 409, allowing
for user cancellation
- added logging for uncaught exceptions in ``run_tasks``
- cancelled the ``reclaim_task`` future on task completion
- pointed docs at the new ``mdc1`` puppet server
- cot verification now renders the entire template rather than the
first task

.. _removed-13:

Removed
~~~~~~~

- ``REVERSED_STATUSES`` is removed, in favor of
``get_reversed_statuses``
- ``task.kill`` has been removed in favor of ``kill_pid`` and
``kill_proc``.
- quieted cot verification a bit by removing some ``log.debug`` lines

.. _section-65:

---------------------

.. _changed-32:

Changed
~~~~~~~

- added ``loop_function`` kwarg to ``sync_main`` for testing

.. _fixed-28:

Fixed
~~~~~

- fixed tests against aiohttp 3.3.0
- fixed concurrent test intermittent errors

.. _section-66:

---------------------

.. _fixed-29:

Fixed
~~~~~

- fixed ``mobile`` ``prebuilt_docker_image_task_types``
- we now log exceptions rather than printing a traceback to stderr

.. _section-67:

---------------------

.. _added-36:

Added
~~~~~

- added a restriction on a.m.o. production scopes.
- added ``prebuilt_docker_image_task_types``. These are the task types
that allow non-artifact docker images; if ``None``, all task types
are allowed.
- added ``get_in_tree_template``, ``get_action_context_and_template``,
``get_jsone_context_and_template`` to help support new action hooks.
- added ``verify_repo_matches_url`` to stop using ``.startswith()`` to
compare urls
- added ``REPO_SCOPE_REGEX`` to allow us to find the ``repo_scope`` in
a task’s scopes.
- added ``get_repo_scope`` to return the ``repo_scope`` in a task’s
scopes (or ``None``)
- added a ``test/data/cotv3`` dir for action hook test data.

.. _changed-33:

Changed
~~~~~~~

- set ``cot_version`` to 3.
- set ``min_cot_version`` to 2.
- we now require cot artifacts in ``verify_docker_image_sha``.
- we no longer check docker image shas against an allowlist; they
either match chain of trust artifact shas, or they’re a task type
that allows prebuilt docker images. If these are defined in-tree, we
trace the request to the tree, so these should be as trustable as the
tree in question.
- we no longer allow for ignoring decision tasks’ ``taskGroupId``\ s.
If they differ from the ``taskId``, we follow the chain back.
- we no longer skip ``verify_docker_worker_task`` for ``mobile``
``cot_product``; but we do allow for prebuilt docker images on all
task types.
- ``get_source_url`` now throws a ``CoTError`` if both the source url
and repo are defined, and the source url doesn’t match the repo.
- quieted the test output significantly.
- default test verbosity is toggled on by the
``SCRIPTWORKER_VERBOSE_TESTS`` env var.
- by default, tests now run concurrently for faster results. To allow
this, we no longer close the event loop anywhere.

.. _fixed-30:

Fixed
~~~~~

- we now log the exception at bad git tag signature verification.

.. _removed-14:

Removed
~~~~~~~

- removed cotv1 support
- removed ``docker_image_allowlists``
- removed ``gecko-decision`` from the decision ``workerType``\ s
- removed ``ACTION_MACH_COMMANDS`` and ``DECISION_MACH_COMMANDS``
- removed “fuzzy matching” task definitions in ``task-graph.json``.
With json-e enabled actions, we should be able to match the
``taskId`` exactly.
- removed ``verify_decision_command``; rebuilding the task definition
via json-e is more precise.
- removed ``get_jsone_template`` in favor of the other, more specific
template functions.

.. _fixed-31:

Fixed
~~~~~

- added ``.pytest_cache`` to ``.gitignore``

.. _section-68: