Synapse

=====================

Model Changes
-------------
- Update ``pe:langid`` to include all language IDs and tags from MS-LCID.
(`3851 <https://github.com/vertexproject/synapse/pull/3851>`_)
- Add additional fields to ``it:sec:stix:indicator``.
(`3858 <https://github.com/vertexproject/synapse/pull/3858>`_)
- Add ``geo:telem:node`` property to more directly track where a node has been.
(`3864 <https://github.com/vertexproject/synapse/pull/3864>`_)
- Add DNS reply code enumeration values to ``inet:dns:request:reply:code``.
(`3868 <https://github.com/vertexproject/synapse/pull/3868>`_)
- See :ref:`userguide_model_v2_179_0` for more detailed model changes.

Features and Enhancements
-------------------------
- Add support for a ``ca_cert`` key to ``$ssl_opts`` on Storm APIs. This can be
used to provide a CA chain for a specific HTTP API call.
(`3849 <https://github.com/vertexproject/synapse/pull/3849>`_)
- Optimize pivot behavior in Storm to avoid unnecessarily re-normalizing
values.
(`3853 <https://github.com/vertexproject/synapse/pull/3853>`_)
- Added ``force`` option to extended property delete APIs to automatically
remove data.
(`3863 <https://github.com/vertexproject/synapse/pull/3863>`_)

Bugfixes
--------
- Fix a bug where trigger name and doc updates set via ``syn:trigger`` nodes
did not persist.
(`3848 <https://github.com/vertexproject/synapse/pull/3848>`_)
- Fix an issue that prevented removing permissions from vaults.
(`3865 <https://github.com/vertexproject/synapse/pull/3865>`_)
- Fix an issue that prevented the old name reference from being removed when a
vault is renamed.
(`3865 <https://github.com/vertexproject/synapse/pull/3865>`_)
- When generating the AHA provisioning URL, the AHA service now binds to
0.0.0.0 instead of the ``dns:name`` configuration value.
(`3866 <https://github.com/vertexproject/synapse/pull/3866>`_)
- Catch additional Python exceptions which could be raised by malformed input
to ``$lib.stix.import.ingest()`` and raise ``BadArg`` instead.
(`3867 <https://github.com/vertexproject/synapse/pull/3867>`_)
- Catch Python ``TypeError`` exceptions in ``$lib.math.number()`` and raise
``BadCast`` exceptions.
(`3871 <https://github.com/vertexproject/synapse/pull/3871>`_)

Deprecations
------------
- Deprecate the ``$tag`` variable in triggers in favor of ``$auto.opts.tag``
(`3854 <https://github.com/vertexproject/synapse/pull/3854>`_)

=====================

Features and Enhancements
-------------------------
- Setting the ``aha:network`` value on the AHA service, as demonstrated in the
deployment guide, is now mandatory.
(`3783 <https://github.com/vertexproject/synapse/pull/3783>`_)
- Added ``synapse.tools.aha.clone`` command to make it easy to bootstrap AHA
mirrors.
(`3783 <https://github.com/vertexproject/synapse/pull/3783>`_)
- Added support for dynamically registered AHA mirrors.
(`3783 <https://github.com/vertexproject/synapse/pull/3783>`_)
- Updated service base class to retrieve updated AHA servers on startup.
(`3783 <https://github.com/vertexproject/synapse/pull/3783>`_)
- Update ``$lib.inet.imap`` and ``$lib.inet.smtp`` APIs to use certificates
present in the Cortex ``tls:ca:dir`` directory. Add ``ssl_verify`` options to
the ``$lib.inet.imap.connect()`` and ``inet:smtp:message.send()`` APIs to
disable TLS verification.
(`3842 <https://github.com/vertexproject/synapse/pull/3842>`_)
- Update the ``aioimaplib`` library constraints to ``>=1.1.0,<1.2.0``.
(`3842 <https://github.com/vertexproject/synapse/pull/3842>`_)
- Log the path of the LMDB file that was backed up in
``synapse.tools.backup.backup_lmdb``.
(`3843 <https://github.com/vertexproject/synapse/pull/3843>`_)

Bugfixes
--------
- Remove a potential race condition in onfini handler registration.
(`3840 <https://github.com/vertexproject/synapse/pull/3840>`_)
- Cause service startup to fail with a clear error message when attempting to
bootstrap a service with a ``mirror`` configuration and the ``aha:provision``
configuration option is missing, or the service storage has been manipulated
into a invalid state.
(`3844 <https://github.com/vertexproject/synapse/pull/3844>`_)

Improved documentation
----------------------
- Update deployment guide to include optional steps to deploy AHA mirrors.
(`3783 <https://github.com/vertexproject/synapse/pull/3783>`_)
- Update deployment guide to clarify ``aha:network`` selection vs ``dns:name``
selection.
(`3783 <https://github.com/vertexproject/synapse/pull/3783>`_)
- Move data model update information for the ``v2.133.0`` release and above
from the changelog and into their own section of the User Guide.
(`3839 <https://github.com/vertexproject/synapse/pull/3839>`_)
- Update Synapse tool examples to use ``aha://`` URLs.
(`3839 <https://github.com/vertexproject/synapse/pull/3839>`_)

Deprecations
------------
- Deprecate the ``Cell.conf.reqConfValu()`` API. This has been replaced with
``Cell.conf.req()``.
(`3783 <https://github.com/vertexproject/synapse/pull/3783>`_)

=====================

Automatic Migrations
--------------------
- Migrate Axon metrics from hive to hotcounts. Migrate Cryotank names storage
from hive to SafeKeyVal storage. Migrate Cortex configuration data from hive
to SafeKeyVal storage. Migrate Cell info and auth configuration from hive to
SafeKeyVal storage.
(`3698 <https://github.com/vertexproject/synapse/pull/3698>`_)
(`3825 <https://github.com/vertexproject/synapse/pull/3825>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Model Changes
-------------
- Add model elements to represent the DriveSerialNumber and MachineID
properties of an LNK file.
(`3817 <https://github.com/vertexproject/synapse/pull/3817>`_)
- Add ``biz:deal:id`` property to track deal identifiers.
(`3832 <https://github.com/vertexproject/synapse/pull/3832>`_)
- Add ``inet:service:message:type`` property to capture message types.
(`3832 <https://github.com/vertexproject/synapse/pull/3832>`_)
- Added ``meta:rule:type`` taxonomy.
(`3834 <https://github.com/vertexproject/synapse/pull/3834>`_)
- See :ref:`userguide_model_v2_177_0` for more detailed model changes.

Features and Enhancements
-------------------------
- Add a new Cell configuration option, ``auth:password:policy``. This can be
used to configure password policy options for authentication.
(`3698 <https://github.com/vertexproject/synapse/pull/3698>`_)
- Add ``$lib.gen.cryptoX509CertBySha256()`` helper function to create
``crypto:x509:cert`` nodes from a SHA256.
(`3801 <https://github.com/vertexproject/synapse/pull/3801>`_)
- Add ``$lib.gen.fileBytesBySha256()`` helper function to create ``file:bytes``
nodes from a SHA256.
(`3801 <https://github.com/vertexproject/synapse/pull/3801>`_)
- Add ``$lib.model.migration.s.inetSslCertToTlsServercert()`` migration helper
to migrate ``inet:ssl:cert`` nodes to ``inet:tls:servercert`` nodes.
(`3801 <https://github.com/vertexproject/synapse/pull/3801>`_)
- Add ``$lib.gen.inetTlsServerCertByServerAndSha256()`` helper function to
create ``inet:tls:servercert`` nodes from a server (or URI) and SHA256.
(`3801 <https://github.com/vertexproject/synapse/pull/3801>`_)
- Added Storm library for creating printable tables: ``$lib.tabular``.
(`3818 <https://github.com/vertexproject/synapse/pull/3818>`_)
- Add ``$lib.model.ext.addEdge()`` and ``$lib.model.ext.delEdge()`` APIs for
managing extended model edge definitions.
(`3824 <https://github.com/vertexproject/synapse/pull/3824>`_)
- Added ``--wipe`` option to the ``merge`` command which replaces the top layer
of the view once the merge is complete. Using ``--wipe`` makes incremental
merges more performant.
(`3828 <https://github.com/vertexproject/synapse/pull/3828>`_)
- Updated ``view.merge`` command to use ``$view.swapLayer()`` for improved
performance.
(`3828 <https://github.com/vertexproject/synapse/pull/3828>`_)
- Added ``$view.swapLayer()`` API to allow users to start fresh with an
existing view.
(`3828 <https://github.com/vertexproject/synapse/pull/3828>`_)
- Update the ``aiohttp`` library constraints to ``>=3.10.0,<4.0``. Update the
``aiohttp-socks`` library constraints to ``>=0.10.0,<0.11.0``.
(`3830 <https://github.com/vertexproject/synapse/pull/3830>`_)
- Tightened up ``aha.svc.list`` Storm command output when using ``--nexus``.
(`3835 <https://github.com/vertexproject/synapse/pull/3835>`_)

Bugfixes
--------
- Prevent the root user for a Synapse service from being locked, archived, or
having its admin status removed.
(`3698 <https://github.com/vertexproject/synapse/pull/3698>`_)
- Catch Python ``TypeError`` exceptions that could be raised by
``$lib.base64.decode()`` and now raise ``StormRuntimeError`` detailing the
problem.
(`3827 <https://github.com/vertexproject/synapse/pull/3827>`_)
- Fix ``Bad file descriptor`` errors that could happen during link teardown.
(`3831 <https://github.com/vertexproject/synapse/pull/3831>`_)

=====================

Model Changes
-------------
- Updates to the ``inet`` model.
(`3811 <https://github.com/vertexproject/synapse/pull/3811>`_)
(`3814 <https://github.com/vertexproject/synapse/pull/3814>`_)
- See :ref:`userguide_model_v2_176_0` for more detailed model changes.

Features and Enhancements
-------------------------
- Add ``storm.exec`` command for executing arbitrary text as Storm.
(`3807 <https://github.com/vertexproject/synapse/pull/3807>`_)
(`3812 <https://github.com/vertexproject/synapse/pull/3812>`_)
- Ensure the ``synapse.storm`` structured log messages contain the view iden.
(`3812 <https://github.com/vertexproject/synapse/pull/3812>`_)
- Added ``$lib.storm.run()`` to programmatically invoke Storm.
(`3813 <https://github.com/vertexproject/synapse/pull/3813>`_)
- Remove the per-node pivot errors from the Cortex log output.
(`3819 <https://github.com/vertexproject/synapse/pull/3819>`_)

=====================

Automatic Migrations
--------------------
- Migrate existing ndef secondary properties to use the new ndef property
indexing.
(`3794 <https://github.com/vertexproject/synapse/pull/3794>`_)
(`3809 <https://github.com/vertexproject/synapse/pull/3809>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Features and Enhancements
-------------------------
- Update Cell with ``_getCellHttpOpts()`` method to allow for overriding default
HTTP options.
(`3770 <https://github.com/vertexproject/synapse/pull/3770>`_)
- Add additional indexing for ndef based secondary properties.
(`3794 <https://github.com/vertexproject/synapse/pull/3794>`_)
(`3809 <https://github.com/vertexproject/synapse/pull/3809>`_)
- Implement ``--prs-from-git`` in ``synapse.tools.changelog``.
(`3800 <https://github.com/vertexproject/synapse/pull/3800>`_)
- Update the ``getCellInfo()`` API to include HTTPS listener addresses and
ports.
(`3802 <https://github.com/vertexproject/synapse/pull/3802>`_)
- Improve permissions checking performance in the Storm ``merge`` command.
(`3804 <https://github.com/vertexproject/synapse/pull/3804>`_)
- Support multiple tags in the diff command, which also allows for more
efficient deduplication (e.g. ``diff --tag foo bar``
versus ``diff --tag foo | diff --tag bar | uniq``).
(`3806 <https://github.com/vertexproject/synapse/pull/3806>`_)
- Add information about the remote link when logging common server side
Telepath errors.
(`3808 <https://github.com/vertexproject/synapse/pull/3808>`_)

Bugfixes
--------
- Fix an AttributeError in ``synapse.tools.changelog``.
(`3798 <https://github.com/vertexproject/synapse/pull/3798>`_)
- Fix for large array props causing system lag.
(`3799 <https://github.com/vertexproject/synapse/pull/3799>`_)

Improved documentation
----------------------
- Remaining docs have been converted from Jupyter notebook format to RST.
(`3803 <https://github.com/vertexproject/synapse/pull/3803>`_)

Deprecations
------------
- Deprecate the use of the ``synapse.lib.jupyter`` library. This will be
removed on 2024-08-26.
(`3803 <https://github.com/vertexproject/synapse/pull/3803>`_)

=====================

Automatic Migrations
--------------------
- Renormalize ``ou:position:title``, ``ou:conference:name``, and
``ou:conference:names`` secondary properties.
(`3701 <https://github.com/vertexproject/synapse/pull/3701>`_)
- Populate new ``econ:currency`` nodes from existing secondary properties.
(`3790 <https://github.com/vertexproject/synapse/pull/3790>`_)
- Add a Cortex storage migration to set the correct View iden value on all
Trigger definitions.
(`3760 <https://github.com/vertexproject/synapse/pull/3760>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Model Changes
-------------
- Add a new model, ``entity``, for modeling elements related to entity
resolution.
(`3781 <https://github.com/vertexproject/synapse/pull/3781>`_)
- Updates to the ``crypto``, ``econ``, ``files``, ``ou``, and ``pol`` models.
(`3790 <https://github.com/vertexproject/synapse/pull/3790>`_)
(`3781 <https://github.com/vertexproject/synapse/pull/3781>`_)
- See :ref:`userguide_model_v2_174_0` for more detailed model changes.

Features and Enhancements
-------------------------
- Add additional context to structured log information when a long LMDB commit
is detected.
(`3747 <https://github.com/vertexproject/synapse/pull/3747>`_)
- Add support to ``synapse.lib.msgpack`` functions for handling integers
requiring more than 64 bits to store them.
(`3767 <https://github.com/vertexproject/synapse/pull/3767>`_)
(`3780 <https://github.com/vertexproject/synapse/pull/3780>`_)
- Add support for Storm variables in array filters.
(`3775 <https://github.com/vertexproject/synapse/pull/3775>`_)
- Add a ``kill()`` API to the Storm ``cron`` objects.
(`3787 <https://github.com/vertexproject/synapse/pull/3787>`_)
(`3796 <https://github.com/vertexproject/synapse/pull/3796>`_)
- Add log messages when a cron job is enabled or disabled.
(`3793 <https://github.com/vertexproject/synapse/pull/3793>`_)

Bugfixes
--------
- Trigger definitions now always have the View iden that they belong to set
upon View creation. The Storm ``$lib.trigger.set()`` API now uses the trigger
view instead of the current view when checking permissions.
(`3760 <https://github.com/vertexproject/synapse/pull/3760>`_)
- Add missing item information when an error occurs while replaying a nexus
change entry upon startup
(`3778 <https://github.com/vertexproject/synapse/pull/3778>`_)
- Fix the startup order for the Cortex embedded JSONStor to avoid an issue with
the nexus replay on startup.
(`3779 <https://github.com/vertexproject/synapse/pull/3779>`_)
- Wrap the Nexus mirror loop setup code in a try/except block to handle
unexpected errors.
(`3781 <https://github.com/vertexproject/synapse/pull/3781>`_)
- Only fire the beholder ``pkg:add`` events when the contents of a Storm
package change.
(`3785 <https://github.com/vertexproject/synapse/pull/3785>`_)