Synapse

=====================

Features and Enhancements
-------------------------
- Add a ``$lib.jsonstor.cachedel()`` API to allow for the removal of data
created by ``$lib.jsonstor.cacheget()``.
(`3322 <https://github.com/vertexproject/synapse/pull/3322>`_)

Bugfixes
--------
- Ensure the base Cell ``fini()``'s the Aha client that it creates. This fixes
a unit test performance issue.
(`3324 <https://github.com/vertexproject/synapse/pull/3324>`_)

Deprecations
------------
- Mark the following Cryotank related API arguments and functions as
deprecated. These APIs are related to server-side offset tracking for
callers. Code which relies on these should be updated to do local offset
tracking. These APIs and arguments will be removed in v2.150.0.
(`3326 <https://github.com/vertexproject/synapse/pull/3326>`_)

- ``CryoApi.puts(seqn=...)`` argument.
- ``CryoApi.rows(seqn=...)`` argument.
- ``CryoApi.slice(iden=...)`` argument.
- ``CryoApi.offset()`` function.
- ``CryoTank.getOffset()`` function.
- ``CryoTank.setOffset()`` function.
- ``CryoTank.puts(seqn=...)`` argument.
- ``CryoTank.rows(seqn=...)`` argument.
- ``CryoTank.slice(iden=...)`` argument.
- ``TankAPI.offset()`` function.
- ``TankApi.puts(seqn=...)`` argument.
- ``TankAPI.slice(iden=...)`` argument.

=====================

Features and Enhancements
-------------------------
- Add ``wait`` and ``timeout`` arguments to Cryotank ``slice()`` APIs.
(`3320 <https://github.com/vertexproject/synapse/pull/3320>`_)
- Add a ``charset`` parameter to the Storm ``inet:imap:server.search()`` API.
This can be used to specify the ``CHARSET`` value when crafting a search
query.
(`3318 <https://github.com/vertexproject/synapse/pull/3318>`_)

Bugfixes
--------
- Vendor the ``asyncio.timeouts.Timeout`` class from Python 3.11.3 to ensure
correct task cancellation behavior is available for
``synapse.common.wait_for()``.
(`3321 <https://github.com/vertexproject/synapse/pull/3321>`_)

=====================

Features and Enhancements
-------------------------
- Update Storm ``graph`` projection to only include edges between nodes in the
result set and include a `"reverse": true` in the edge info when embedding
an edge on its target node once it is yielded.
(`3305 <https://github.com/vertexproject/synapse/pull/3305>`_)
- Map the Nexus LMDB slab with ``map_async=True`` by default.
(`3314 <https://github.com/vertexproject/synapse/pull/3314>`_)
- Mark the Storm ``macro.exec`` as a ``readonly`` safe command. Mark the
Storm APIs ``$lib.macro.list()`` and ``$lib.macro.get()`` as ``readonly``
safe. Mark the ``str`` APIs as ``readonly`` safe.
(`3316 <https://github.com/vertexproject/synapse/pull/3316>`_)

Bugfixes
--------
- Fix an issue where Layer data migrations failed when a layer was marked
as ``readonly``.
(`3313 <https://github.com/vertexproject/synapse/pull/3313>`_)
- Fix an issue where utility functions for packed nodes in
``synapse.lib.node`` did not handle nodes from HTTP API endpoints.
(`3315 <https://github.com/vertexproject/synapse/pull/3315>`_)

=====================

Automatic Migrations
--------------------
- Update indexing for light edges to index the N1 and N2 node identifiers
together.
(`3302 <https://github.com/vertexproject/synapse/pull/3302>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Model Changes
-------------
- Update to the ``inet``, ``it``, and ``meta`` models.
(`3285 <https://github.com/vertexproject/synapse/pull/3285>`_)
(`3298 <https://github.com/vertexproject/synapse/pull/3298>`_)
(`3301 <https://github.com/vertexproject/synapse/pull/3301>`_)
(`3310 <https://github.com/vertexproject/synapse/pull/3310>`_)

**New Types**

``it:sec:tlp``
The US CISA Traffic-Light-Protocol used to designate information sharing
boundaries.

``meta:priority``
A generic priority enumeration.

``meta:severity``
A generic severity enumeration.


**New Forms**

``it:sec:metrics``
A node used to track metrics of an organization's infosec program.

``it:sec:vuln:scan``
An instance of running a vulnerability scan.

``it:sec:vuln:scan:result``
A vulnerability scan result for an asset.``

**New Properties**

``it:dev:repo:issue``
The form had the following properties added to it:

``updated``
The time the issue was updated.

``id``
The ID of the issue in the repository system.

``it:dev:repo:issue:comment``
The form had the following properties added to it:

``created``
The time the comment was created.

``updated``
The time the comment was updated.

``it:dev:repo:diff:comment``
The form had the following properties added to it:

``created``
The time the comment was created.

``updated``
The time the comment was updated.

``meta:note``
The form had the following properties added to it:

``updated``
The time the note was updated.

**Deprecated Properties**

``it:exec:proc``
The ``it:exec:proc`` form had the following property marked as deprecated:

* ``src:exe``

``inet:whois:iprec``
The ``inet:whois:iprec`` form had the following property marked as deprecated:

* ``registrant``


Features and Enhancements
-------------------------
- Add a new Storm keyword, ``reverse( ... )``, which can be used to run a lift
operation in reverse order.
(`3266 <https://github.com/vertexproject/synapse/pull/3266>`_)
- Update indexing for light edges to index the N1 and N2 node identifiers
together.
(`3302 <https://github.com/vertexproject/synapse/pull/3302>`_)
- Update the Storm ``once`` command behavior and documentation to be more
intuitive when setting its timestamp and allowing nodes through it.
(`3282 <https://github.com/vertexproject/synapse/pull/3282>`_)
- Add a ``synapse_version`` key to the Storm Package schema. This can be used
to provide a string version indentifier with a minimum and maximum version,
such as ``>=2.145.0,<3.0.0``.
(`3304 <https://github.com/vertexproject/synapse/pull/3304>`_)
- Update the Storm runtime to respect permissions declared with a ``default``
value of ``true``. This allows Storm packages to define permissions which
are defaulted to ``true``.
(`3287 <https://github.com/vertexproject/synapse/pull/3287>`_)
- Add a ``SIGHUP`` handler to the base Cell which can be used to reload HTTPS
certificate files from disk. The ``synapse.tools.reload`` tool can also be
used to trigger this behavior.
(`3293 <https://github.com/vertexproject/synapse/pull/3293>`_)
- The optional ``max:users`` feature no longer counts ``locked`` or
``archived`` users when adding users.
(`3295 <https://github.com/vertexproject/synapse/pull/3295>`_)
- Update the YAML functions to use the ``yaml.CSafeLoader`` and
``yaml.CSafeDumper``.
(`3289 <https://github.com/vertexproject/synapse/pull/3289>`_)

Bugfixes
--------
- Replace ``asyncio.wait_for()`` use with a copy of the Python 3.12
implementation to avoid a race condition when cancelling tasks.
(`3299 <https://github.com/vertexproject/synapse/pull/3299>`_)
(`3307 <https://github.com/vertexproject/synapse/pull/3307>`_)
- Fix an issue with the Storm trigger ``set()`` method not properly checking
the values that it allows to be set.
(`3290 <https://github.com/vertexproject/synapse/pull/3290>`_)
- Fix an off-by-one bug in the ``SlabSeqn.aiter()`` method.
(`3300 <https://github.com/vertexproject/synapse/pull/3300>`_)
- Fix a performance issue with the IPv6 regular expression used in the scrape
APIs.
(`3311 <https://github.com/vertexproject/synapse/pull/3311>`_)

Improved Documentation
----------------------
- Revise the Storm User Guide to consolidate the background information
and data modeling sections. Add a user focused section on Views and Layers.
(`3303 <https://github.com/vertexproject/synapse/pull/3303>`_)
- Add ``int`` type specific information to the Storm documentation.
(`3288 <https://github.com/vertexproject/synapse/pull/3288>`_)
- The Storm ``movetag`` command now moves the ``doc:url`` property from the
old ``syn:tag`` node to the new ``syn:tag`` node.
(`3294 <https://github.com/vertexproject/synapse/pull/3294>`_)
- Storm Library and Type documentation no longer renders function signatures
with Python style defaults.
(`3296 <https://github.com/vertexproject/synapse/pull/3296>`_)

Deprecations
------------
- Many deprecated Cortex and splice related APIs have been marked for removal
after 2023-10-01. The full list of APIs which will be removed can be found
at :ref:`changelog-depr-20231001`.
(`3292 <https://github.com/vertexproject/synapse/pull/3292>`_)
- The use of ``synapse.common.aclosing()`` has been replaced with
``contextlib.aclosing()``. The vendored ``aclosing()`` implementation will
be removed in ``v2.250.0``.
(`3206 <https://github.com/vertexproject/synapse/pull/3206>`_)

=====================

Model Changes
-------------
- Updates to the ``inet:dns`` and ``it`` model.
(`3257 <https://github.com/vertexproject/synapse/pull/3257>`_)
(`3276 <https://github.com/vertexproject/synapse/pull/3276>`_)

**New Forms**

``it:dev:repo:type:taxonomy``
A version control system type taxonomy.

``it:dev:repo``
A version control system instance.

``it:dev:repo:remote``
A remote repo that is tracked for changes/branches/etc.

``it:dev:repo:branch``
A branch in a version control system instance.

``it:dev:repo:commit``
A commit to a repository.

``it:dev:repo:diff``
A diff of a file being applied in a single commit.

``it:dev:repo:issue``
An issue raised in a repository.

``it:dev:repo:issue:comment``
A comment on an issue in a repository.

``it:dev:repo:diff:comment``
A comment on a diff in a repository.

**New Properties**

``inet:dns:answer``
The form had the following properties added to it:

``time``
The time that the DNS response was transmitted.

Features and Enhancements
-------------------------
- The iden of the Cron job is now added to the Storm query log made with
the ``synapse.storm`` logger when using structured logging.
(`3235 <https://github.com/vertexproject/synapse/pull/3235>`_)
- Add a ``keepalive`` option to the Storm query ``opts``. This may be used
with long-running Storm queries when behind a network proxy or load balancer
which may terminate idle connections.
(`3272 <https://github.com/vertexproject/synapse/pull/3272>`_)
- Update the allowed versions of the ``cryptography`` library.
(`3281 <https://github.com/vertexproject/synapse/pull/3281>`_)

Bugfixes
--------
- Fix an issue where Storm Dmons could start prior to data model migrations.
(`3279 <https://github.com/vertexproject/synapse/pull/3279>`_)
- Adjust the storage convention for ``once`` state data to fix an edge case
and clarify documentation.
(`3282 <https://github.com/vertexproject/synapse/pull/3282>`_)
- Fix an issue with missing keys in storage nodes during migrations.
(`3284 <https://github.com/vertexproject/synapse/pull/3284>`_)

=====================

Model Changes
-------------
- Update to the ``crypto`` model.
(`3256 <https://github.com/vertexproject/synapse/pull/3256>`_)

**Updated Types**

``hex``
The ``zeropad`` option has been changed from a ``bool`` to an ``int``.
It may now be used to specify the zero extended length of the hex string.

**Updated Properties**

``crypto:x509:cert``
The form had the following properties updated on it:

``serial``
The ``size`` value has been changed to ``zeropad`` to zeropad values
with less than 40 octets, and to allow storing large serial numbers from
malformed certificates.

Features and Enhancements
-------------------------
- Add ``$lib.model.ext.getExtModel()`` and
``$lib.model.ext.addExtModel()`` Storm APIs to get all the extended model
definitions in a Cortex and to add extended model definitions to
a Cortex in bulk.
(`3252 <https://github.com/vertexproject/synapse/pull/3252>`_)
- Add ``inet:ipv6`` to the list of types identified with scrape APIs. The
``inet:server`` form identified by scrape APIs now also identifies IPv6
server addresses.
(`3259 <https://github.com/vertexproject/synapse/pull/3259>`_)
- Add a check to the Cortex startup to identify and log the presence of
deprecated model elements and direct users to check and lock them
at :ref:`storm-model-deprecated-check`.
(`3253 <https://github.com/vertexproject/synapse/pull/3253>`_)
(`3264 <https://github.com/vertexproject/synapse/pull/3264>`_)
- Add a new Storm function, ``$lib.vars.type()``, to get the type
value of an object.
(`3100 <https://github.com/vertexproject/synapse/pull/3100>`_)
- Add a Storm library, ``$lib.pack``, for packing and unpacking structured
byte values.
(`3261 <https://github.com/vertexproject/synapse/pull/3261>`_)
- The Storm ``$lib.gen()`` functions and associated commands now generate
stable guid values based on their inputs when making nodes.
(`3268 <https://github.com/vertexproject/synapse/pull/3268>`_)
- Add the ``.bazar`` TLD to the list of TLDs identified by the Synapse scrape
functionality.
(`3271 <https://github.com/vertexproject/synapse/pull/3271>`_)
- Add the View iden to the task identifier for running Storm tasks.
(`3247 <https://github.com/vertexproject/synapse/pull/3247>`_)
- Add performance related sysctl values to the output of the Storm
``Cell.getSystemInfo()`` and ``$lib.cell.getSystemInfo()`` APIs.
(`3236 <https://github.com/vertexproject/synapse/pull/3236>`_)
- Update the allowed versions of the ``vcrpy`` library. Thank you
``captainGeech42`` for the contribution.
(`3204 <https://github.com/vertexproject/synapse/pull/3204>`_)

Bugfixes
--------
- Ensure the input to the ``CoreAPI.storm()`` ( and related APIs ) is a
string.
(`3255 <https://github.com/vertexproject/synapse/pull/3255>`_)
(`3269 <https://github.com/vertexproject/synapse/pull/3269>`_)
- Fix a bug in ``synapse.tools.aha.enroll`` where a user with a
``telepath.yaml`` file containing an ``aha:servers`` key with a list of
lists failed to enroll a local user.
(`3260 <https://github.com/vertexproject/synapse/pull/3260>`_)
- Fix an issue where Storm functions using ``emit`` failed to cleanup their
sub-runtimes.
(`3250 <https://github.com/vertexproject/synapse/pull/3250>`_)
- Add verification that a Storm function call is being made on a callable
object and raise a ``StormRuntimeError`` if the object cannot be called.
Previously invalid calls could raise a ``TypeError``.
(`3243 <https://github.com/vertexproject/synapse/pull/3243>`_)
- Fix the order of the Beholder ``cron:stop`` message firing when a Cron job
is stopped.
(`3265 <https://github.com/vertexproject/synapse/pull/3265>`_)

Improved Documentation
----------------------
- Add a section to the Storm reference for user defined functions in Storm.
That can be found at :ref:`storm-adv-functions`.
(`3245 <https://github.com/vertexproject/synapse/pull/3245>`_)
- Update the devops documentation to add a note about the Telepath ``aha://``
protocol using a ``mirror=true`` parameter to connect to a service mirror
instead of a leader.
(`3267 <https://github.com/vertexproject/synapse/pull/3267>`_)
- Update the ``preboot.sh`` example script to account for Docker changes
introduced in ``v2.133.0``.