Synapse

====================

Features and Enhancements
-------------------------
- Add ``emit`` and ``stop`` keywords to Storm. The ``emit`` keyword is used
in functions to make them behave as generators, which can yield arbitrary
values. The ``stop`` keyword can be used to prematurely end a function which
is ``emit``'ing values.
(`2475 <https://github.com/vertexproject/synapse/pull/2475>`_)
- Add Storm Module Interfaces. This allows Storm Package authors to define
common module interfaces, so that multiple modules can implement the API
convention to provide a consistent set of data across multiple Storm
modules. A ``search`` convention is added to the Cortex, which will be used
in ``lookup`` mode when the ``storm:interface:search`` configuration option
is set.
(`2475 <https://github.com/vertexproject/synapse/pull/2475>`_)
- Storm queries in ``lookup`` mode now fire ``look:miss`` events into the
Storm message stream when the lookup value contains a valid node value,
but the node is not present in the current View.
(`2475 <https://github.com/vertexproject/synapse/pull/2475>`_)
- Add a ``:host`` secondary property to ``risk:hasvuln`` form to record
``it:host`` instances which have a vulnerability.
(`2512 <https://github.com/vertexproject/synapse/pull/2512>`_)
- Add ``synapse.lib.scrape`` support for identifying ``it:sec:cve`` values.
(`2509 <https://github.com/vertexproject/synapse/pull/2509>`_)

Bugfixes
--------
- Fix an ``IndexError`` that can occur during ``Layer.verify()`` routines.
These routines are in a beta status and are subject to change.
(`2507 <https://github.com/vertexproject/synapse/pull/2507>`_)
- Ensure that parameter and header arguments passed to Storm
``$lib.inet.http`` functions are cast into strings values.
(`2510 <https://github.com/vertexproject/synapse/pull/2510>`_)

====================

This release contains an automatic data migration that may cause additional
startup time on the first boot. This is done to unique array properties which
previously were not uniqued. Deployments with startup or liveliness probes
should have those disabled while this upgrade is performed to prevent
accidental termination of the Cortex process. Please ensure you have a tested
backup available before applying this update.

Features and Enhancements
-------------------------

- Update the following array properties to be unique sets, and add a data
model migration to update the data at rest:
(`2469 <https://github.com/vertexproject/synapse/pull/2469>`_)

- ``biz:rfp:requirements``
- ``crypto:x509:cert:ext:sans``
- ``crypto:x509:cert:ext:crls``
- ``crypto:x509:cert:identities:fqdns``
- ``crypto:x509:cert:identities:emails``
- ``crypto:x509:cert:identities:ipv4s``
- ``crypto:x509:cert:identities:ipv6s``
- ``crypto:x509:cert:identities:urls``
- ``crypto:x509:cert:crl:urls``
- ``inet:whois:iprec:contacts``
- ``inet:whois:iprec:links``
- ``inet:whois:ipcontact:roles``
- ``inet:whois:ipcontact:links``
- ``inet:whois:ipcontact:contacts``
- ``it:account:groups``
- ``it:group:groups``
- ``it:reveng:function:impcalls``
- ``it:reveng:filefunc:funccalls``
- ``it:sec:cve:references``
- ``risk:vuln:cwes``
- ``tel:txtmesg:recipients``

- Add Layer index verification routines, to compare the Layer indices against
the stored data for Nodes. This is exposed via the ``.verify()`` API on the
Stormtypes ``layer`` object.
These routines are in a beta status and are subject to change.
(`2488 <https://github.com/vertexproject/synapse/pull/2488>`_)
- The ``.json()`` API on ``inet:http:resp`` now raises a
``s_exc.BadJsonText`` exception, which can be caught with the Storm
``try ... catch`` syntax.
(`2500 <https://github.com/vertexproject/synapse/pull/2500>`_)
- Add ``$lib.inet.ipv6.expand()`` to expand an IPv6 address to its long form.
(`2502 <https://github.com/vertexproject/synapse/pull/2502>`_)
- Add ``hasPathObj()``, ``copyPathObj()`` and ``copyPathObjs()`` APIs to the
``JsonStor``.
(`2438 <https://github.com/vertexproject/synapse/pull/2438>`_)
- Allow setting a custom title when making documentation for Cell
``confdefs`` with the ``synapse.tools.autodoc`` tool.
(`2504 <https://github.com/vertexproject/synapse/pull/2504>`_)
- Update the minimum version of the ``aiohttp`` library to ``v3.8.1``.
(`2495 <https://github.com/vertexproject/synapse/pull/2495>`_)

Improved Documentation
----------------------
- Add content previously hosted at ``commercial.docs.vertex.link`` to the
mainline Synapse documentation. This includes some devops information
related to orchestration, information about Advanced and Rapid Power-Ups,
information about the Synapse User Interface, as well as some support
information.
(`2498 <https://github.com/vertexproject/synapse/pull/2498>`_)
(`2499 <https://github.com/vertexproject/synapse/pull/2499>`_)
(`2501 <https://github.com/vertexproject/synapse/pull/2501>`_)
- Add ``Synapse-Malshare`` and ``Synapse-TeamCymru`` Rapid Power-Ups to the
list of available Rapid Power-Ups.
(`2506 <https://github.com/vertexproject/synapse/pull/2506>`_)
- Document the ``jsonlines`` option for the ``api/v1/storm`` and
``api/v1/storm/nodes`` HTTP APIs.
(`2505 <https://github.com/vertexproject/synapse/pull/2505>`_)

====================

Features and Enhancements
-------------------------
- Add ``.onion`` and ``.bit`` to the TLD list used for scraping text. Update
the TLD list from the latest IANA TLD list.
(`2483 <https://github.com/vertexproject/synapse/pull/2483>`_)
(`2497 <https://github.com/vertexproject/synapse/pull/2497>`_)
- Add support for writeback mirroring of layers.
(`2463 <https://github.com/vertexproject/synapse/pull/2463>`_)
(`2489 <https://github.com/vertexproject/synapse/pull/2489>`_)
- Add ``$lib.scrape()`` Stormtypes API. This can be used to do programmatic
scraping of text using the same regular expressions used by the Storm
``scrape`` command and the ``synapse.lib.scrape`` APIs.
(`2486 <https://github.com/vertexproject/synapse/pull/2486>`_)
- Add a ``jsonlines`` output mode to Cortex streaming HTTP endpoints.
(`2493 <https://github.com/vertexproject/synapse/pull/2493>`_)
- Add a ``--raw`` argument to the Storm ``pkg.load`` command. This loads the
raw JSON response as a Storm package.
(`2491 <https://github.com/vertexproject/synapse/pull/2491>`_)
- Add a ``blocked`` enum to the ``proj:ticket:status`` property to represent a
blocked ticket.
(`2490 <https://github.com/vertexproject/synapse/pull/2490>`_)

Bugfixes
--------
- Fix a behavior with ``$path`` losing variables in pure Storm command
execution.
(`2492 <https://github.com/vertexproject/synapse/pull/2492>`_)

Improved Documentation
----------------------
- Update the description of the Storm ``scrape`` command.
(`2494 <https://github.com/vertexproject/synapse/pull/2494>`_)

====================

Features and Enhancements
-------------------------
- Add a Storm ``runas`` command. This allows admin users to execute Storm
commands as other users.
(`2473 <https://github.com/vertexproject/synapse/pull/2473>`_)
- Add a Storm ``intersect`` command. This command produces the intersection
of nodes emitted by running a Storm query over all inbound nodes to the
``intersect`` command.
(`2480 <https://github.com/vertexproject/synapse/pull/2480>`_)
- Add ``wait`` and ``timeout`` parameters to the ``Axon.hashes()`` and
``$lib.axon.list()`` APIs.
(`2481 <https://github.com/vertexproject/synapse/pull/2481>`_)
- Add a ``readonly`` flag to ``synapse.tools.genpkg.loadPkgProto()`` and
``synapse.tools.genpkg.tryLoadPkgProto()`` APIs. If set to ``True`` this
will open files in read only mode.
(`2485 <https://github.com/vertexproject/synapse/pull/2485>`_)
- Allow Storm Prim objects to be capable of directly yielding nodes when used
in ``yield`` statements.
(`2479 <https://github.com/vertexproject/synapse/pull/2479>`_)
- Update the StormDmon subsystem to add debug log information about state
changes, as well as additional data for structured logging output.
(`2455 <https://github.com/vertexproject/synapse/pull/2455>`_)

Bugfixes
--------
- Catch a fatal application error that can occur in the Cortex if the forked
process pool becomes unusable. Previously this would cause the Cortex to
appear unresponsive for executing Storm queries; now this causes the Cortex
to shut down gracefully.
(`2472 <https://github.com/vertexproject/synapse/pull/2472>`_)
- Fix a Storm path variable scoping issue where variables were improperly
scoped when nodes were passed into pure Storm commands.
(`2459 <https://github.com/vertexproject/synapse/pull/2459>`_)

====================

Features and Enhancements
-------------------------
- Update the cron subsystem logs to include the cron name, as well as adding
additional data for structured logging output.
(`2477 <https://github.com/vertexproject/synapse/pull/2477>`_)
- Add a ``sort_keys`` argument to the ``$lib.yaml.save()`` Stormtype API.
(`2474 <https://github.com/vertexproject/synapse/pull/2474>`_)

Bugfixes
--------
- Update the ``asyncio-socks`` version to a version which has a pinned version
range for the ``python-socks`` dependency.
(`2478 <https://github.com/vertexproject/synapse/pull/2478>`_)

====================

Bugfixes
--------
- Update the ``PyOpenSSL`` version to ``21.0.0`` and pin a range of modern
versions of the ``cryptography`` which have stronger API compatibility.
This resolves an API compatibility issue with the two libraries which
affected SSL certificate generation.
(`2476 <https://github.com/vertexproject/synapse/pull/2476>`_)