Synapse

=====================

Features and Enhancements
-------------------------
- Update the Telepath TLS connections to require a minimum TLS version of 1.2.
(`2833 <https://github.com/vertexproject/synapse/pull/2833>`_)
- Update the Axon implementation to use the ``initServiceStorage()`` and
``initServiceRuntime()`` methods, instead of overriding ``__anit__``.
(`2837 <https://github.com/vertexproject/synapse/pull/2837>`_)
- Update the minimum allowed versions of the ``aiosmtplib`` and ``regex``
libraries.
(`2832 <https://github.com/vertexproject/synapse/pull/2832>`_)
(`2841 <https://github.com/vertexproject/synapse/pull/2841>`_)

Bugfixes
--------
- Catch ``LarkError`` exceptions in all Storm query parsing modes.
(`2840 <https://github.com/vertexproject/synapse/pull/2840>`_)
- Catch ``FileNotFound`` errors in ``synapse.tools.healthcheck``. This could
be caused by the tool running during container startup, and prior to a
service making its Unix listening socket available.
(`2836 <https://github.com/vertexproject/synapse/pull/2836>`_)
- Fix an issue in ``Axon.csvrows()`` where invalid data would cause
processing of a file to stop.
(`2835 <https://github.com/vertexproject/synapse/pull/2835>`_)
- Address a deprecation warning in the Synapse codebase.
(`2842 <https://github.com/vertexproject/synapse/pull/2842>`_)
- Correct the type of ``syn:splice:splice`` to be ``data``. Previously it
was ``str``.
(`2839 <https://github.com/vertexproject/synapse/pull/2839>`_)

Improved Documentation
----------------------
- Replace ``livenessProbe`` references with ``readinessProbe`` in the
Kubernetes documentation and examples. The ``startupProbe.failureThreshold``
value was increased to its maximum value.
(`2838 <https://github.com/vertexproject/synapse/pull/2838>`_)
- Fix a typo in the Rapid Power-Up documentation.
(`2831 <https://github.com/vertexproject/synapse/pull/2831>`_)

=====================

Automatic Migrations
--------------------
- Migrate the ``risk:alert:type`` property to a ``taxonomy`` type
and create new nodes as needed.
(`2828 <https://github.com/vertexproject/synapse/pull/2828>`_)
- Migrate the ``pol:country:name`` property to a ``geo:name`` type
and create new nodes as needed.
(`2828 <https://github.com/vertexproject/synapse/pull/2828>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Features and Enhancements
-------------------------
- Updates to the ``geo``, ``inet``, ``media``, ``pol``, ``proj``, and
``risk`` models.
(`2828 <https://github.com/vertexproject/synapse/pull/2828>`_)
(`2829 <https://github.com/vertexproject/synapse/pull/2829>`_)

``geo:area``
Add a new type to record the size of a geographic area.

``geo:place:taxonomy``
Add a form to record an analyst defined taxonomy of different places.

``geo:place``
Add a ``type`` property to record the taxonomy of a place.

``inet:web:memb``
This form has been deprecated.

``inet:web:member``
Add a guid form that represents a web account's membership in a channel or group.

``media:news:taxonomy``
Add a form to record an analyst defined taxonomy of different types or sources of news.

``media:news``
Add a ``type`` property to record the taxonomy of the news.
Add an ``ext:id`` property to record an external identifier provided by a publisher.

``pol:vitals``
Add a guid form to record the vitals for a country.

``pol:country``
Add ``names``, ``place``, ``dissolved`` and ``vitals`` secondary properties.
The ``name`` is changed from a ``str`` to a ``geo:name`` type.
Deprecate the ``pop`` secondary property.

``pol:candidate``
Add an ``incumbent`` property to note if the candidate was an incumbent
in a race.

``proj``
Add missing docstrings to the ``proj`` model forms.

``risk:alert:taxonomy``
Add a form to record an analyst defined taxonomy of alert types.

``risk:alert``
The ``type`` property is changed from a ``str`` to the
``risk:alert:taxonomy`` type.

- Add ``**`` as a power operator for Storm expression syntax.
(`2827 <https://github.com/vertexproject/synapse/pull/2827>`_)
- Add a new test helper, ``synapse.test.utils.StormPkgTest`` to assist with
testing Rapid Power-Ups.
(`2819 <https://github.com/vertexproject/synapse/pull/2819>`_)
- Add ``$lib.axon.metrics()`` to get the metrics from the Axon that the
Cortex is connected to.
(`2818 <https://github.com/vertexproject/synapse/pull/2818>`_)
- Add ``pack()`` methods to the ``auth:user`` and ``auth:role``
objects. This API returns the definitions of the User and Role objects.
(`2823 <https://github.com/vertexproject/synapse/pull/2823>`_)
- Change the Storm Package ``require`` values to log debug messages instead
of raising exceptions if the requirements are not met. Add a
``$lib.pkg.deps()`` API that allows inspecting if a package has its
dependencies met or has conflicts.
(`2820 <https://github.com/vertexproject/synapse/pull/2820>`_)

Bugfixes
--------
- Prevent ``None`` objects from being normalized as tag parts from variables
in Storm.
(`2822 <https://github.com/vertexproject/synapse/pull/2822>`_)
- Avoid intermediate conversion to floats during storage operations related to
Synapse Number objects in Storm.
(`2825 <https://github.com/vertexproject/synapse/pull/2825>`_)

Improved Documentation
----------------------
- Add Developer documentation for writing Rapid Power-Ups.
(`2803 <https://github.com/vertexproject/synapse/pull/2803>`_)
- Add the ``synapse.tests.utils`` package to the Synapse API autodocs.
(`2819 <https://github.com/vertexproject/synapse/pull/2819>`_)
- Update Devops documentation to note the storage requirements for taking
backups of Synapse services.
(`2824 <https://github.com/vertexproject/synapse/pull/2824>`_)
- Update the Storm ``min`` and ``max`` command help to clarify their usage.
(`2826 <https://github.com/vertexproject/synapse/pull/2826>`_)

=====================

Features and Enhancements
-------------------------
- Add a new tool, ``synapse.tools.axon2axon``, for copying the data from one
Axon to another Axon.
(`2813 <https://github.com/vertexproject/synapse/pull/2813>`_)
(`2816 <https://github.com/vertexproject/synapse/pull/2816>`_)

Bugfixes
--------
- Subquery filters did not update runtime variables in the outer scope. This
behavior has been updated to make subquery filter behavior consistent with
regular subqueries.
(`2815 <https://github.com/vertexproject/synapse/pull/2815>`_)
- Fix an issue with converting the Number Storm primitive into its Python
primitive.
(`2811 <https://github.com/vertexproject/synapse/pull/2811>`_)

=====================

Features and Enhancements
-------------------------
- Add a Number primitive to Storm to facilitate fixed point math
operations. Values in expressions which are parsed as floating
point values will now be Numbers by default. Values can also
be cast to Numbers with ``$lib.math.number()``.
(`2762 <https://github.com/vertexproject/synapse/pull/2762>`_)
- Add ``$lib.basex.encode()`` and ``$lib.basex.decode()`` for
encoding and decoding strings using arbitrary charsets.
(`2807 <https://github.com/vertexproject/synapse/pull/2807>`_)
- The tag removal operator (``-``) now accepts lists of tags
to remove.
(`2808 <https://github.com/vertexproject/synapse/pull/2808>`_)
- Add a ``$node.difftags()`` API to calculate and optionally apply
the difference between a list of tags and those present on a node.
(`2808 <https://github.com/vertexproject/synapse/pull/2808>`_)
- Scraped Ethereum addresses are now returned in their EIP55
checksummed form. This change also applies to lookup mode.
(`2809 <https://github.com/vertexproject/synapse/pull/2809>`_)
- Updates to the ``mat``, ``ps``, and ``risk`` models.
(`2804 <https://github.com/vertexproject/synapse/pull/2804>`_)

``mass``
Add a type for storing mass with grams as a base unit.

``ps:vitals``
Add a form to record statistics and demographic data about a person
or contact.

``ps:person``
Add a ``vitals`` secondary property to record the most recent known
vitals for the person.

``ps:contact``
Add a ``vitals`` secondary property to record the most recent known
vitals for the contact.

``risk:tool:taxonomy``
Add a form to record an analyst defined taxonomy of different tools.

``risk:tool:software``
Add a form to record software tools used in threat activity.

``risk:threat``
Add ``reporter``, ``reporter:name``, ``org:loc``, ``org:names``,
and ``goals`` secondary properties.

- Annotate the following light edges.
(`2804 <https://github.com/vertexproject/synapse/pull/2804>`_)

``uses``
When used with ``risk:threat`` nodes, the edge indicates the target
node is used by the source node.

Bugfixes
--------
- Fix language used in the ``model.deprecated.check`` command.
(`2806 <https://github.com/vertexproject/synapse/pull/2806>`_)
- Remove the ``-y`` switch in the ``count`` command.
(`2806 <https://github.com/vertexproject/synapse/pull/2806>`_)

=====================

Automatic Migrations
--------------------
- Migrate `crypto:x509:cert:serial` from `str` to `hex` type. Existing values
which cannot be converted as integers or hex values will be moved into
nodedata under the key ``migration:0_2_10`` as ``{'serial': value}``
(`2789 <https://github.com/vertexproject/synapse/pull/2789>`_)
- Migrate ``ps:contact:title`` to the ``ou:jobtitle`` type and create
``ou:jobtitle`` nodes.
(`2789 <https://github.com/vertexproject/synapse/pull/2789>`_)
- Correct hugenum property index values for values with more than
28 digits of precision.
(`2766 <https://github.com/vertexproject/synapse/pull/2766>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Features and Enhancements
-------------------------
- Updates to the ``crypto`` and ``ps`` models.
(`2789 <https://github.com/vertexproject/synapse/pull/2789>`_)

``crypto:x509:cert``
The ``serial`` secondary property has been changed from a ``str`` to a
``hex`` type.

``ps:contact``
The type of the ``title`` secondary property has been changed from a
``str`` to an ``ou:jobtitle``.

- Add ``$lib.hex.toint()``, ``$lib.hex.fromint()``, ``$lib.hex.trimext()``
and ``$lib.hex.signext()`` Storm APIs for handling hex encoded integers.
(`2789 <https://github.com/vertexproject/synapse/pull/2789>`_)
- Add ``set()`` and ``setdefault()`` APIs on the SynErr exception class.
Improve support for unpickling SynErr exceptions.
(`2797 <https://github.com/vertexproject/synapse/pull/2797>`_)
- Add logging configuration to methods which are called in spawned processes,
and log exceptions occurring in the processes before tearing them down.
(`2795 <https://github.com/vertexproject/synapse/pull/2795>`_)

Bugfixes
--------
- BadTypeValu errors raised when normalizing a tag timestamp now include
the name of the tag being set.
(`2797 <https://github.com/vertexproject/synapse/pull/2797>`_)
- Correct a CI issue that prevented the v2.103.0 Docker images from
being published.
(`2798 <https://github.com/vertexproject/synapse/pull/2798>`_)

Improved Documentation
----------------------
- Update data model documentation.
(`2796 <https://github.com/vertexproject/synapse/pull/2796>`_)

=====================

Features and Enhancements
-------------------------
- Updates to the ``it``, ``ou``, and ``risk`` models.
(`2778 <https://github.com/vertexproject/synapse/pull/2778>`_)

``it:prod:soft``
Add a ``techniques`` secondary property to record techniques employed by
the author of the software.

``ou:campaign``
Add a ``techniques`` secondary property to record techniques employed by
the campaign.

``ou:org``
Add a ``techniques`` secondary property to record techniques employed by
the org.

``ou:technique``
Add a form to record specific techniques used to achieve a goal.

``ou:technique:taxonomy``
Add a form to record an analyst defined taxonomy of different techniques.

``risk:attack``
Add a ``techniques`` secondary property to record techniques employed
during the attack.
Deprecate the following secondary properties, in favor of using light
edges.

- ``target``
- ``target:host``
- ``target:org``
- ``target:person``
- ``target:place``
- ``used:email``
- ``used:file``
- ``used:host``
- ``used:server``
- ``used:software``
- ``used:url``
- ``used:vuln``
- ``via:email``
- ``via:ipv4``
- ``via:ipv6``
- ``via:phone``

``risk:compromise``
Add a ``techniques`` secondary property to record techniques employed
during the compromise.

``risk:threat``
Add a form to record a threat cluster or subgraph of threat activity
attributable to one group.

- Annotate the following light edges.
(`2778 <https://github.com/vertexproject/synapse/pull/2778>`_)

``targets``
When used with ``ou:org``, ``ou:campaign``, ``risk:threat``, or
``risk:attack`` nodes, the edge indicates the target node was targeted
by the source node.

``uses``
When used with an ``ou:campaign`` or ``risk:attack`` node, the edge
indicates the target node is used by the source node.

- Change the behavior of the Storm ``count`` command to consume nodes.
If the previous behavior is desired, use the ``--yield`` option when
invoking the ``count`` command.
(`2779 <https://github.com/vertexproject/synapse/pull/2779>`_)
- Add ``$lib.random.int()`` API to Storm for generating random integers.
(`2783 <https://github.com/vertexproject/synapse/pull/2783>`_)
- Add a new tool, ``synapse.tools.livebackup`` for taking a live backup of
a service.
(`2788 <https://github.com/vertexproject/synapse/pull/2788>`_)
- The Storm ``$lib.jsonstor.cacheset()`` API now returns a dict containing the
path and time. The ``$lib.jsonstor.cacheget()`` API now has an argument to
retrieve the entire set of enveloped data.
(`2790 <https://github.com/vertexproject/synapse/pull/2790>`_)
- Add a HTTP 404 handler for the Axon ``v1/by/sha256/<sha256>`` endpoint which
catches invalid ``<sha256>`` values.
(`2780 <https://github.com/vertexproject/synapse/pull/2780>`_)
- Add helper scripts for doing bulk Synapse Docker image builds and testing.
(`2716 <https://github.com/vertexproject/synapse/pull/2716>`_)
- Add ``aha:\\`` support to ``synapse.tools.csvtool``.
(`2791 <https://github.com/vertexproject/synapse/pull/2791>`_)

Bugfixes
--------
- Ensure that errors that occur when backing up a service are logged prior
to tearing down the subprocess performing the backup.
(`2781 <https://github.com/vertexproject/synapse/pull/2781>`_)
- Add missing docstring for ``$lib.stix.import``.
(`2786 <https://github.com/vertexproject/synapse/pull/2786>`_)
- Allow setting tags on a Node from a Storm ``List`` object.
(`2782 <https://github.com/vertexproject/synapse/pull/2782>`_)

Improved Documentation
----------------------
- Remove ``synapse-google-ct`` from the list of Rapid Power-Ups.
(`2779 <https://github.com/vertexproject/synapse/pull/2779>`_)
- Add developer documentation for building Synapse Docker containers.
(`2716 <https://github.com/vertexproject/synapse/pull/2716>`_)
- Fix spelling errors in model documentation.
(`2782 <https://github.com/vertexproject/synapse/pull/2782>`_)

Deprecations
------------
- The ``vertexproject/synapse:master-py37`` and
``vertexproject/synapse:v2.x.x-py37`` Docker containers are no longer being
built.
(`2716 <https://github.com/vertexproject/synapse/pull/2716>`_)