Synapse

====================

Automatic Migrations
--------------------
- Migrate Cortex nodes which may have been skipped in an earlier migration due
to missing tagprop indexes. See :ref:`datamigration` for more
information about automatic migrations.

Features and Enhancements
-------------------------
- Expand Synapse requirements to include updated versions of the ``base58``,
``cbor2``, ``lmdb``, ``pycryptodome``, ``PyYAML``, ``xxhash``.
(`2520 <https://github.com/vertexproject/synapse/pull/2520>`_)

Bugfixes
--------
- Fix an issue with the Tagprop migration from ``v2.42.0`` where a missing
index could have resulted in Layer storage nodes not being updated.
(`2522 <https://github.com/vertexproject/synapse/pull/2522>`_)
(`2523 <https://github.com/vertexproject/synapse/pull/2523>`_)
- Fix an issue with ``synapse.lib.platforms.linux.getTotalMemory()`` when
using a process segregated with the Linux cgroups2 API.
(`2517 <https://github.com/vertexproject/synapse/pull/2517>`_)

Improved Documentation
----------------------
- Add devops instructions related to automatic data migrations for Synapse
components.
(`2523 <https://github.com/vertexproject/synapse/pull/2523>`_)
- Update the model deprecation documentation for the ``it:host:model`` and
``it:host:make`` properties.
(`2521 <https://github.com/vertexproject/synapse/pull/2521>`_)

====================

Features and Enhancements
-------------------------
- Add Mach-O metadata support the file model. This includes the following
new forms: ``file:mime:macho:loadcmd``, ``file:mime:macho:version``,
``file:mime:macho:uuid``, ``file:mime:macho:segment``, and
``file:mime:macho:section``.
(`2503 <https://github.com/vertexproject/synapse/pull/2503>`_)
- Add ``it:screenshot``, ``it:prod:hardware``, ``it:prod:component``,
``it:prod:hardwaretype``, and ``risk:mitigation`` forms to the model. Add
``:hardware`` property to ``risk:hasvuln`` form. Add ``:hardware`` property
to ``it:host`` form. The ``:manu`` and ``:model`` secondary properties on
``it:host`` have been deprecated.
(`2514 <https://github.com/vertexproject/synapse/pull/2514>`_)
- The ``guid`` type now strips hyphen (``-``) characters when doing norm. This
allows users to provide external UUID / GUID strings for use.
(`2514 <https://github.com/vertexproject/synapse/pull/2514>`_)
- Add a ``Axon.postfiles()`` to allow POSTing files as multi-part form encoded
files over HTTP. This is also exposed through the ``fields`` argument on the
Storm ``$lib.inet.http.post()`` and ``$lib.inet:http:request`` APIs.
(`2516 <https://github.com/vertexproject/synapse/pull/2516>`_)
- Add ``.yu`` ccTLD to the list of TLDs identified by the Synapse scrape
functionality.
(`2518 <https://github.com/vertexproject/synapse/pull/2518>`_)
- Add ``mesg`` arguments to all instances of ``NoSuchProp`` exceptions.
(`2519 <https://github.com/vertexproject/synapse/pull/2519>`_)

====================

Features and Enhancements
-------------------------
- Add ``emit`` and ``stop`` keywords to Storm. The ``emit`` keyword is used
in functions to make them behave as generators, which can yield arbitrary
values. The ``stop`` keyword can be used to prematurely end a function which
is ``emit``'ing values.
(`2475 <https://github.com/vertexproject/synapse/pull/2475>`_)
- Add Storm Module Interfaces. This allows Storm Package authors to define
common module interfaces, so that multiple modules can implement the API
convention to provide a consistent set of data across multiple Storm
modules. A ``search`` convention is added to the Cortex, which will be used
in ``lookup`` mode when the ``storm:interface:search`` configuration option
is set.
(`2475 <https://github.com/vertexproject/synapse/pull/2475>`_)
- Storm queries in ``lookup`` mode now fire ``look:miss`` events into the
Storm message stream when the lookup value contains a valid node value,
but the node is not present in the current View.
(`2475 <https://github.com/vertexproject/synapse/pull/2475>`_)
- Add a ``:host`` secondary property to ``risk:hasvuln`` form to record
``it:host`` instances which have a vulnerability.
(`2512 <https://github.com/vertexproject/synapse/pull/2512>`_)
- Add ``synapse.lib.scrape`` support for identifying ``it:sec:cve`` values.
(`2509 <https://github.com/vertexproject/synapse/pull/2509>`_)

Bugfixes
--------
- Fix an ``IndexError`` that can occur during ``Layer.verify()`` routines.
These routines are in a beta status and are subject to change.
(`2507 <https://github.com/vertexproject/synapse/pull/2507>`_)
- Ensure that parameter and header arguments passed to Storm
``$lib.inet.http`` functions are cast into strings values.
(`2510 <https://github.com/vertexproject/synapse/pull/2510>`_)

====================

This release contains an automatic data migration that may cause additional
startup time on the first boot. This is done to unique array properties which
previously were not uniqued. Deployments with startup or liveliness probes
should have those disabled while this upgrade is performed to prevent
accidental termination of the Cortex process. Please ensure you have a tested
backup available before applying this update.

Features and Enhancements
-------------------------

- Update the following array properties to be unique sets, and add a data
model migration to update the data at rest:
(`2469 <https://github.com/vertexproject/synapse/pull/2469>`_)

- ``biz:rfp:requirements``
- ``crypto:x509:cert:ext:sans``
- ``crypto:x509:cert:ext:crls``
- ``crypto:x509:cert:identities:fqdns``
- ``crypto:x509:cert:identities:emails``
- ``crypto:x509:cert:identities:ipv4s``
- ``crypto:x509:cert:identities:ipv6s``
- ``crypto:x509:cert:identities:urls``
- ``crypto:x509:cert:crl:urls``
- ``inet:whois:iprec:contacts``
- ``inet:whois:iprec:links``
- ``inet:whois:ipcontact:roles``
- ``inet:whois:ipcontact:links``
- ``inet:whois:ipcontact:contacts``
- ``it:account:groups``
- ``it:group:groups``
- ``it:reveng:function:impcalls``
- ``it:reveng:filefunc:funccalls``
- ``it:sec:cve:references``
- ``risk:vuln:cwes``
- ``tel:txtmesg:recipients``

- Add Layer index verification routines, to compare the Layer indices against
the stored data for Nodes. This is exposed via the ``.verify()`` API on the
Stormtypes ``layer`` object.
These routines are in a beta status and are subject to change.
(`2488 <https://github.com/vertexproject/synapse/pull/2488>`_)
- The ``.json()`` API on ``inet:http:resp`` now raises a
``s_exc.BadJsonText`` exception, which can be caught with the Storm
``try ... catch`` syntax.
(`2500 <https://github.com/vertexproject/synapse/pull/2500>`_)
- Add ``$lib.inet.ipv6.expand()`` to expand an IPv6 address to its long form.
(`2502 <https://github.com/vertexproject/synapse/pull/2502>`_)
- Add ``hasPathObj()``, ``copyPathObj()`` and ``copyPathObjs()`` APIs to the
``JsonStor``.
(`2438 <https://github.com/vertexproject/synapse/pull/2438>`_)
- Allow setting a custom title when making documentation for Cell
``confdefs`` with the ``synapse.tools.autodoc`` tool.
(`2504 <https://github.com/vertexproject/synapse/pull/2504>`_)
- Update the minimum version of the ``aiohttp`` library to ``v3.8.1``.
(`2495 <https://github.com/vertexproject/synapse/pull/2495>`_)

Improved Documentation
----------------------
- Add content previously hosted at ``commercial.docs.vertex.link`` to the
mainline Synapse documentation. This includes some devops information
related to orchestration, information about Advanced and Rapid Power-Ups,
information about the Synapse User Interface, as well as some support
information.
(`2498 <https://github.com/vertexproject/synapse/pull/2498>`_)
(`2499 <https://github.com/vertexproject/synapse/pull/2499>`_)
(`2501 <https://github.com/vertexproject/synapse/pull/2501>`_)
- Add ``Synapse-Malshare`` and ``Synapse-TeamCymru`` Rapid Power-Ups to the
list of available Rapid Power-Ups.
(`2506 <https://github.com/vertexproject/synapse/pull/2506>`_)
- Document the ``jsonlines`` option for the ``api/v1/storm`` and
``api/v1/storm/nodes`` HTTP APIs.
(`2505 <https://github.com/vertexproject/synapse/pull/2505>`_)

====================

Features and Enhancements
-------------------------
- Add ``.onion`` and ``.bit`` to the TLD list used for scraping text. Update
the TLD list from the latest IANA TLD list.
(`2483 <https://github.com/vertexproject/synapse/pull/2483>`_)
(`2497 <https://github.com/vertexproject/synapse/pull/2497>`_)
- Add support for writeback mirroring of layers.
(`2463 <https://github.com/vertexproject/synapse/pull/2463>`_)
(`2489 <https://github.com/vertexproject/synapse/pull/2489>`_)
- Add ``$lib.scrape()`` Stormtypes API. This can be used to do programmatic
scraping of text using the same regular expressions used by the Storm
``scrape`` command and the ``synapse.lib.scrape`` APIs.
(`2486 <https://github.com/vertexproject/synapse/pull/2486>`_)
- Add a ``jsonlines`` output mode to Cortex streaming HTTP endpoints.
(`2493 <https://github.com/vertexproject/synapse/pull/2493>`_)
- Add a ``--raw`` argument to the Storm ``pkg.load`` command. This loads the
raw JSON response as a Storm package.
(`2491 <https://github.com/vertexproject/synapse/pull/2491>`_)
- Add a ``blocked`` enum to the ``proj:ticket:status`` property to represent a
blocked ticket.
(`2490 <https://github.com/vertexproject/synapse/pull/2490>`_)

Bugfixes
--------
- Fix a behavior with ``$path`` losing variables in pure Storm command
execution.
(`2492 <https://github.com/vertexproject/synapse/pull/2492>`_)

Improved Documentation
----------------------
- Update the description of the Storm ``scrape`` command.
(`2494 <https://github.com/vertexproject/synapse/pull/2494>`_)

====================

Features and Enhancements
-------------------------
- Add a Storm ``runas`` command. This allows admin users to execute Storm
commands as other users.
(`2473 <https://github.com/vertexproject/synapse/pull/2473>`_)
- Add a Storm ``intersect`` command. This command produces the intersection
of nodes emitted by running a Storm query over all inbound nodes to the
``intersect`` command.
(`2480 <https://github.com/vertexproject/synapse/pull/2480>`_)
- Add ``wait`` and ``timeout`` parameters to the ``Axon.hashes()`` and
``$lib.axon.list()`` APIs.
(`2481 <https://github.com/vertexproject/synapse/pull/2481>`_)
- Add a ``readonly`` flag to ``synapse.tools.genpkg.loadPkgProto()`` and
``synapse.tools.genpkg.tryLoadPkgProto()`` APIs. If set to ``True`` this
will open files in read only mode.
(`2485 <https://github.com/vertexproject/synapse/pull/2485>`_)
- Allow Storm Prim objects to be capable of directly yielding nodes when used
in ``yield`` statements.
(`2479 <https://github.com/vertexproject/synapse/pull/2479>`_)
- Update the StormDmon subsystem to add debug log information about state
changes, as well as additional data for structured logging output.
(`2455 <https://github.com/vertexproject/synapse/pull/2455>`_)

Bugfixes
--------
- Catch a fatal application error that can occur in the Cortex if the forked
process pool becomes unusable. Previously this would cause the Cortex to
appear unresponsive for executing Storm queries; now this causes the Cortex
to shut down gracefully.
(`2472 <https://github.com/vertexproject/synapse/pull/2472>`_)
- Fix a Storm path variable scoping issue where variables were improperly
scoped when nodes were passed into pure Storm commands.
(`2459 <https://github.com/vertexproject/synapse/pull/2459>`_)