Synapse

=====================

Features and Enhancements
-------------------------
- Update ``synapse.test.utils.SynTest`` helpers to disable sysctl checks
for test services by default.
(`3741 <https://github.com/vertexproject/synapse/pull/3741>`_)

Bugfixes
--------
- Fix a key positioning error in the LMDBSlab when scanning backwards
by prefix.
(`3739 <https://github.com/vertexproject/synapse/pull/3739>`_)
- Fix a bug in the ``str`` type normalization routine for handling floating
point values. The floating point values are now also run through the
string norming logic.
(`3742 <https://github.com/vertexproject/synapse/pull/3742>`_)
- Add missing beholder messages for view layer modifications.
(`3743 <https://github.com/vertexproject/synapse/pull/3743>`_)

Improved Documentation
----------------------
- Update Devops documentation to add additional information about low downtime
service updates, Rapid Power-Up updates, and release cadence information.
Update references from ``docker-compose`` to use ``docker compose``.
(`3722 <https://github.com/vertexproject/synapse/pull/3722>`_)

=====================

Automatic Migrations
--------------------
- Populate an additional index of buids by form in Layers.
(`3729 <https://github.com/vertexproject/synapse/pull/3729>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Model Changes
-------------
- Updates to the ``infotech`` and ``file`` models.
(`3702 <https://github.com/vertexproject/synapse/pull/3702>`_)
(`3725 <https://github.com/vertexproject/synapse/pull/3725>`_)
(`3732 <https://github.com/vertexproject/synapse/pull/3732>`_)
- See :ref:`userguide_model_v2_170_0` for more detailed model changes.

Features and Enhancements
-------------------------
- Added ``$lib.model.migration.s.riskHasVulnToVulnerable`` migration helper
to create ``risk:vulnerable`` nodes from ``risk:hasvuln`` nodes.
(`3734 <https://github.com/vertexproject/synapse/pull/3734>`_)
- Added ``$lib.model.migration.s.itSecCpe_2_170_0()`` migration helper to update
``it:sec:cpe`` nodes created before this release. Details about the migration
helper can be found in the help (``help -v $lib.model.migration.s.itSecCpe_2_170_0``)
(`3515 <https://github.com/vertexproject/synapse/pull/3515>`_)
- Update Storm lift optimization for tag filters to also allow hinting
based on runtsafe variable values.
(`3733 <https://github.com/vertexproject/synapse/pull/3733>`_)
- Log an info message with the current Cell and Synapse version on startup.
(`3723 <https://github.com/vertexproject/synapse/pull/3723>`_)
- Add per-Cell version checks to prevent accidental downgrades of services.
(`3728 <https://github.com/vertexproject/synapse/pull/3728>`_)
- Add a check to Cells that will warn when performance related sysctl values
are not configured correctly on the host. This warning can be disabled with
the ``health:sysctl:checks`` configuration option.
(`3712 <https://github.com/vertexproject/synapse/pull/3712>`_)
- Add ``forms`` and ``interfaces`` type options to the ``ndef`` type, which
require the value to be one of the specified forms, or inherit one of the
specified interfaces.
(`3724 <https://github.com/vertexproject/synapse/pull/3724>`_)
- Add support for pivoting from an ``ndef`` secondary prop to specific form.
(`3715 <https://github.com/vertexproject/synapse/pull/3715>`_)
- Add support for pivoting to or from ``ndef`` array properties.
(`3720 <https://github.com/vertexproject/synapse/pull/3720>`_)
- Add an index of buids by form to Layers. A ``getStorNodesByForm()`` API has
been added to Storm Layer objects to retrieve storage nodes using this index.
(`3729 <https://github.com/vertexproject/synapse/pull/3729>`_)
- Storm Dmon APIs called on a Cortex mirror now call up to the leader to
retrieve their result.
(`3735 <https://github.com/vertexproject/synapse/pull/3735>`_)
- Add a ``insertParentFork()`` API on Storm View objects to insert a new
View between an existing fork and its parent View.
(`3731 <https://github.com/vertexproject/synapse/pull/3731>`_)
- Quorum merge requests are now allowed on Views which have forks.
(`3738 <https://github.com/vertexproject/synapse/pull/3738>`_)

Bugfixes
--------
- Fix a formatting issue in an error message that could be raised during
JSON decoding in a Storm ``http:api:request`` object.
(`3730 <https://github.com/vertexproject/synapse/pull/3730>`_)
- Fix an issue where ``inet:url`` norming did not handle IPv6 addresses
in the host portion of the URL correctly.
(`3727 <https://github.com/vertexproject/synapse/pull/3727>`_)
- Fix an issue where executing the ``view.exec`` command from within a
privileged Storm runtime still checked user permissions for the specified
view.
(`3726 <https://github.com/vertexproject/synapse/pull/3726>`_)
- Update logic for parsing CPE 2.2 and CPE 2.3 strings to be more compliant with
the specification. This resulted in better conversions from CPE 2.2 to CPE 2.3
and CPE 2.3 to CPE 2.2.
(`3515 <https://github.com/vertexproject/synapse/pull/3515>`_)

=====================

Features and Enhancements
-------------------------
- Add a data migration helper library, ``$lib.model.migration``. This
contains functions to help with migrating data via Storm.
(`3714 <https://github.com/vertexproject/synapse/pull/3714>`_)
- Add Extended HTTP API iden values to structured Storm query logs.
(`3710 <https://github.com/vertexproject/synapse/pull/3710>`_)
- Add ``node.data.set`` and ``node.data.pop`` to the list of declared
Cortex permissions.
(`3716 <https://github.com/vertexproject/synapse/pull/3716>`_)

Bugfixes
--------
- Restore cron iden values in structured Storm query logs.
(`3710 <https://github.com/vertexproject/synapse/pull/3710>`_)
- The Storm APIs ``$lib.min()`` and ``$lib.max()`` now handle a single
input. The Storm APIs ``$lib.min()`` and ``$lib.max()`` now raise a
``StormRuntimeError`` when there is no input provided to them. Previously
these conditions caused a Python exception in the Storm runtime.
(`3711 <https://github.com/vertexproject/synapse/pull/3711>`_)
- The ``onboot:optimize`` configuration now skips optimizing any LMDB files
found in the Cell local backup storage.
(`3713 <https://github.com/vertexproject/synapse/pull/3713>`_)

Deprecations
------------
- Removed the Telepath APIs ``CoreApi.enableMigrationMode`` and
``CoreApi.disableMigrationMode``. Remove support for the Cell
``hiveboot.yaml`` file. These had a removal date of 2025-05-05.
(`3717 <https://github.com/vertexproject/synapse/pull/3717>`_)

=====================

Model Changes
-------------
- Add a new model, ``plan``, for modeling elements of plannings systems.
(`3697 <https://github.com/vertexproject/synapse/pull/3697>`_)
- See :ref:`userguide_model_v2_168_0` for more detailed model changes.

Features and Enhancements
-------------------------
- Cortex data model migrations will now be checked and executed when the
service is promoted to being a leader. This allows for Cortex updates
which use mirrors to have minimal downtime. Cortex model migrations which
are executed using Storm will always run directly on the Cortex leader.
(`3694 <https://github.com/vertexproject/synapse/pull/3694>`_)
(`3695 <https://github.com/vertexproject/synapse/pull/3695>`_)
- The Storm ``aha:pool.del()`` method now returns the full name of the
service that was removed.
(`3704 <https://github.com/vertexproject/synapse/pull/3704>`_)

Bugfixes
--------
- The Storm command ``aha.pool.svc.del`` now prints out the name of the
service that was removed from the pool or notes that there were no
services removed.
(`3704 <https://github.com/vertexproject/synapse/pull/3704>`_)
- When setting a service "down" with AHA, conditionally clear the ``ready``
flag as well. Previously this flag was not cleared, and offline services
could still report as ``ready``.
(`3705 <https://github.com/vertexproject/synapse/pull/3705>`_)
- Add missing sleep statements to callers of ``Layer.syncNodeEdits2()``.
(`3700 <https://github.com/vertexproject/synapse/pull/3700>`_)

Improved Documentation
----------------------
- Update Storm command reference documentation to add additional examples
for the ``uniq`` command. Update Storm command reference documentation to
add ``gen.geo.place`` and ``gen.it.av.scan.result`` commands.
(`3699 <https://github.com/vertexproject/synapse/pull/3699>`_)
- Update type specific documentation. Add additional information about ``loc``
and ``syn:tag`` behavior with prefixes and wlidcards. Add a section on the
``duration`` and ``taxonomy`` types.
(`3703 <https://github.com/vertexproject/synapse/pull/3703>`_)
- Add documentation for ``$lib.auth.easyperm.level`` constants and the
``$lib.dict.has()`` function.
(`3706 <https://github.com/vertexproject/synapse/pull/3706>`_)

=====================

Automatic Migrations
--------------------
- Set the ``protected`` flag on all Views in the Cortex, using the existing
value of the ``nomerge`` flag.
(`3681 <https://github.com/vertexproject/synapse/pull/3681>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Model Changes
-------------
- Updates to the ``base`` and ``file`` models.
(`3674 <https://github.com/vertexproject/synapse/pull/3674>`_)
(`3688 <https://github.com/vertexproject/synapse/pull/3688>`_)
- See :ref:`userguide_model_v2_167_0` for more detailed model changes.

Features and Enhancements
-------------------------
- Add ``aha.svc.list`` and ``aha.svc.stat`` commands to enumerate the AHA
services. Add ``$lib.aha`` Storm APIs to delete, get, and list the AHA
services.
(`3685 <https://github.com/vertexproject/synapse/pull/3685>`_)
(`3692 <https://github.com/vertexproject/synapse/pull/3692>`_)
(`3693 <https://github.com/vertexproject/synapse/pull/3693>`_)
- Add a ``protected`` option that can be set on Views to prevent
merging and deletion. This replaces the ``nomerge`` option.
(`3679 <https://github.com/vertexproject/synapse/pull/3679>`_)
- Add Beholder events for creating, deleting, and updating Macros.
(`3681 <https://github.com/vertexproject/synapse/pull/3681>`_)
- Update the ``StormPkgTest.getTestCore()`` API to add a ``prepkghook``
callback option. This can be used to execute code prior to loading Storm
packages. The ``getTestCore()`` API now waits for ``onload`` handlers to
complete for each package it loads.
(`3687 <https://github.com/vertexproject/synapse/pull/3687>`_)
- Ensure that the ``Cell.ahaclient`` is fully owned and managed by the
``Cell``. It will no longer use a global client that may exist.
(`3677 <https://github.com/vertexproject/synapse/pull/3677>`_)
- Update the ``stix2-validator`` library constraints to ``>=3.2.0,<4.0.0``.
Update the allowed range of the ``idna`` library to ``>=3.6,<3.8``.
(`3672 <https://github.com/vertexproject/synapse/pull/3672>`_)
(`3684 <https://github.com/vertexproject/synapse/pull/3684>`_)

Bugfixes
--------
- Asyncio Tasks created by signal handlers on the Base object are now held
onto, to ensure that they cannot be garbage collected before or during
their task execution.
(`3686 <https://github.com/vertexproject/synapse/pull/3686>`_)
- Update the ``Axon.postfiles`` and ``Axon.wput`` APIs to check for the
existence of files before attempting to send them over an HTTP connection.
(`3682 <https://github.com/vertexproject/synapse/pull/3682>`_)
- Fix an issue where pruning a non-existent tag mistakenly pruned related
tags.
(`3673 <https://github.com/vertexproject/synapse/pull/3673>`_)
- Ensure that macro names are at least 1 character in length.
(`3679 <https://github.com/vertexproject/synapse/pull/3679>`_)
- Fix a bug where ``$lib.telepath.open()`` could leak Python exceptions into
the Storm runtime.
(`3685 <https://github.com/vertexproject/synapse/pull/3685>`_)

Improved Documentation
----------------------
- Add documentation for ``$lib.aha``, ``$lib.aha.pool``, and the ``aha:pool``
type.
(`3685 <https://github.com/vertexproject/synapse/pull/3685>`_)

Deprecations
------------
- Deprecate the use of ``hiveboot.yaml`` to configure a Cell hive. This will be
removed on 2024-05-05.
(`3678 <https://github.com/vertexproject/synapse/pull/3678>`_)
- The ``nomerge`` option on views has been deprecated. It is automatically
redirected to the ``protected`` option. This redirection will be removed in
``v3.0.0``.
(`3681 <https://github.com/vertexproject/synapse/pull/3681>`_)
- The Telepath APIs for interacting with a Cell Hive, ``listHiveKey``,
``getHiveKeys``, ``getHiveKey``, ``setHiveKey``, ``popHiveKey``, and
``saveHiveTree`` have been deprecated. The tools ``synapse.tools.hive.load``
and ``synapse.tools.hive.save`` have been deprecated. These will be removed
in ``v3.0.0``.
(`3683 <https://github.com/vertexproject/synapse/pull/3683>`_)
- The ``Telepath.Pipeline`` class has been marked as deprecated and will be
removed in ``v3.0.0``.
(`3691 <https://github.com/vertexproject/synapse/pull/3691>`_)

=====================

Model Changes
-------------
- Updates to the ``inet``, ``ou``, ``person`` and ``risk`` models.
(`3649 <https://github.com/vertexproject/synapse/pull/3649>`_)
(`3653 <https://github.com/vertexproject/synapse/pull/3653>`_)
(`3657 <https://github.com/vertexproject/synapse/pull/3657>`_)
- See :ref:`userguide_model_v2_166_0` for more detailed model changes.

Features and Enhancements
-------------------------
- When setting a tag on a node, the tag value is now redirected based on
parent tags having ``:isnow`` properties set.
(`3650 <https://github.com/vertexproject/synapse/pull/3650>`_)
- Add a ``$lib.spooled.set()`` Storm API. This can be used to get a
``spooled:set`` object. This set will offload the storage of its members
to a temporary location on disk when it grows above a certain size.
(`3632 <https://github.com/vertexproject/synapse/pull/3632>`_)
- Add a ``$lib.cache.fixed()`` Storm API. This can be used to get a
``cache:fixed`` object. This cache will execute user provided callbacks
written in Storm upon a cache miss.
(`3661 <https://github.com/vertexproject/synapse/pull/3661>`_)
- Add a ``pool`` option to Cron jobs. This can be set to True to enable a
Cron job storm query to be executed on a Storm pool member.
(`3652 <https://github.com/vertexproject/synapse/pull/3652>`_)
- Add a ``pool`` option to Extended HTTP API handlers. This can be set to
True to enable an HTTP request handler to be executed on a Storm pool member.
(`3663 <https://github.com/vertexproject/synapse/pull/3663>`_)
(`3667 <https://github.com/vertexproject/synapse/pull/3667>`_)
- Add a new Storm API, ``$lib.cortex.httpapi.getByPath()``, that can be
used to get an ``http:api`` object by its path. The ``path`` value is
evaluated in the same order that the HTTP endpoint resolves the handlers.
(`3663 <https://github.com/vertexproject/synapse/pull/3663>`_)
- Add ``--list`` and ``--gate`` options to ``synapse.tools.modrole`` and
``synapse.tools.moduser``.
(`3632 <https://github.com/vertexproject/synapse/pull/3632>`_)
- Add a ``view.getMergingViews()`` Storm API. This returns a list of view
idens that have open merge requests on a view.
(`3666 <https://github.com/vertexproject/synapse/pull/3666>`_)
- The Storm API ``show:storage`` option now includes storage information for
any embedded properties.
(`3656 <https://github.com/vertexproject/synapse/pull/3656>`_)
- Update the ``LinkShutDown`` exception that a Telepath client may raise to
indicate that the connection has been disconnected.
(`3640 <https://github.com/vertexproject/synapse/pull/3640>`_)
- Add repr functions for printing the ``aha:pool`` and ``http:api`` objects
in Storm.
(`3663 <https://github.com/vertexproject/synapse/pull/3663>`_)
(`3665 <https://github.com/vertexproject/synapse/pull/3665>`_)
- The Telepath ``Pool`` object has been replaced with a new object,
``ClientV2``. This is now the only object returned by the
``synapse.telepath.open()`` API. This is an AHA pool aware Client which
can be used to connect to an AHA pool.
(`3662 <https://github.com/vertexproject/synapse/pull/3662>`_)
- Remove the unused Provenance subsystem from the Cortex.
(`3655 <https://github.com/vertexproject/synapse/pull/3655>`_)
- Constrain the ``stix2-validator`` library to ``3.0.0,<3.2.0`` due to
an API change. This constraint is expected be changed in the next
release.
(`3669 <https://github.com/vertexproject/synapse/pull/3669>`_)

Bugfixes
--------
- Fix a bug where a Cortex ``promote()`` call could hang when tearing down
any running Cron jobs. Cron jobs cancelled during a promotion event will
be logged but their cancelled status will not be recorded in the Nexus.
(`3658 <https://github.com/vertexproject/synapse/pull/3658>`_)
- Fix a bug where the Storm pool configuration could cause a Cortex to fail
to start up. The Storm pool is now configured upon startup but its use is
blocked until the Storm pool is ready to service requests.
(`3662 <https://github.com/vertexproject/synapse/pull/3662>`_)
- Ensure that the URL argument provided to ``cortex.storm.pool.set`` can be
parsed as a Telepath URL. Previously any string input was accepted.
(`3665 <https://github.com/vertexproject/synapse/pull/3665>`_)

Improved Documentation
----------------------
- Update the list of Cortex permissions in the Admin Guide to include
``service.add``, ``service.del``, ``service.get``, and ``service.list``.
(`3647 <https://github.com/vertexproject/synapse/pull/3647>`_)
- Update the docstring for the Storm ``cortex.storm.pool.del`` command to note
the effects of removing a pool and the interruption of running queries.
(`3665 <https://github.com/vertexproject/synapse/pull/3665>`_)
- Update the documentation for the Storm ``http:api`` object to include the
``methods`` attribute.
(`3663 <https://github.com/vertexproject/synapse/pull/3663>`_)

Deprecations
------------
- The Telepath ``task:init`` message format has been marked as deprecated and
will be removed in ``v3.0.0``. This should not affect any users using Synapse
``v2.x.x`` in their client code.
(`3640 <https://github.com/vertexproject/synapse/pull/3640>`_)
- The authgate with the name ``cortex`` is not used for permission checking and
will be removed in ``v3.0.0``. At startup, the Cortex will now check for any
use of this authgate and log warning messages. Attempts to set permissions
with this gateiden via Storm will produce ``warn`` messages.
(`3648 <https://github.com/vertexproject/synapse/pull/3648>`_)