Synapse

====================

Features and Enhancements
-------------------------
- Add a ``search`` mode to Storm. The ``search`` mode utilizes the Storm
search interface to lift nodes. The ``lookup`` mode no longer uses the
search interface.
(`2689 <https://github.com/vertexproject/synapse/pull/2689>`_)
- Add a ``?mirror=true`` flag to ``aha://`` Telepath URLs which will cause
the Aha service lookups to prefer using a mirror of the service rather than
the leader.
(`2681 <https://github.com/vertexproject/synapse/pull/2681>`_)
- Add ``$lib.inet.http.urlencode()`` and ``$lib.inet.http.urldecode()`` Storm
APIs for handling URL encoding.
(`2688 <https://github.com/vertexproject/synapse/pull/2688>`_)
- Add type validation for all Cell configuration options throughout the
lifetime of the Cell and all operations which modify its configuration
values. This prevents invalid values from being persisted on disk.
(`2687 <https://github.com/vertexproject/synapse/pull/2687>`_)
(`2691 <https://github.com/vertexproject/synapse/pull/2691>`_)

Bugfixes
--------
- Fix an issue where the ``=`` sign in the Storm grammar was assigned an
anonymous terminal name by the grammar parser. This caused an issue with
interpreting various syntax errors.
(`2690 <https://github.com/vertexproject/synapse/pull/2690>`_)

====================

Automatic Migrations
--------------------
- Re-normalize the migrated properties noted in the data model updates listed
below. See :ref:`datamigration` for more information about automatic
migrations.

Features and Enhancements
-------------------------
- Updates to the ``crypto``, ``infotech``, ``ou``, and ``person`` models.
(`2620 <https://github.com/vertexproject/synapse/pull/2620>`_)
(`2684 <https://github.com/vertexproject/synapse/pull/2684>`_)

``crypto:algorithm``
Add a form to represent a named cryptography algorithm.

``crypto:key``
Add a form to represent a cryptographic key and algorithm.

``crypto:smart:effect:transfertoken``
Add a form to represent the effect of transferring ownership of a
non-fungible token.

``crypto:smart:effect:transfertokens``
Add a form to represent the effect of transferring multiple fungible
tokens.

``crypto:smart:effect:edittokensupply``
Add a form to represent the increase or decrease in the supply of
fungible tokens.

``it:prod:softname``
Add a form to represent a software name.

``it:host``
Add a ``:os:name`` secondary property.

``it:mitre:attack:software``
Migrate the ``:name`` and ``:names`` properties to ``it:prod:softname``
type.

``it:prod:soft``
Migrate the ``:name`` and ``:names`` properties to ``it:prod:softname``
type.

``it:prod:softver``
Deprecate the ``:software:name`` property.
Migrate the ``:name`` and ``:names`` properties to ``it:prod:softname``
type.

``it:app:yara:rule``
Add a ``:family`` property to represent the software family the rule is
designed to detect.

``it:sec:c2:config``
Add a form to represent C2 configuration data.

``ou:campaign``
Add a ``:org:name`` property to represent the name of the organization
responsible the campaign.
Add a ``:org:fqdn`` property to represent the fqdn of the organization
responsible the campaign.
Add a ``:team`` property to represent the team responsible for the
campaign.

``ou:team``
Add a form to represent a team within an organization.

``ou:industry``
Migrate the ``:name`` property to ``ou:industryname`` type.
Add a ``:names`` property for alternative names.

``ou:industryname``
Add a form to represent the name of an industry.

``ou:position``
Add a ``:team`` property to represent the team associated with a given
position.

``ps:contact``
Add a ``:crypto:address`` property to represent the crypto currency
address associated with the contact.

- Add ``$lib.copy()`` to Storm. This allows making copies of objects which
are compatible with being serialized with msgpack.
(`2678 <https://github.com/vertexproject/synapse/pull/2678>`_)
- Remove `print` events from the Storm `limit` command.
(`2674 <https://github.com/vertexproject/synapse/pull/2674>`_)

Bugfixes
--------
- Fix an issue where client certificates presented in Telepath ``ssl``
connections could fallback to resolving users by a prefix. This was not
intended to be allowed when client certificates are used with Telepath.
(`2675 <https://github.com/vertexproject/synapse/pull/2675>`_)
- Fix an issue where ``node:del`` triggers could fail to fire when adding
nodeedits directly to a view or snap.
(`2654 <https://github.com/vertexproject/synapse/pull/2654>`_)
- Fix header escaping when generating autodoc content for Synapse Cells.
(`2677 <https://github.com/vertexproject/synapse/pull/2677>`_)
- Assorted unit tests fixes to make tests more stable.
(`2680 <https://github.com/vertexproject/synapse/pull/2680>`_)
- Fix an issue with Storm function argument parsing.
(`2685 <https://github.com/vertexproject/synapse/pull/2685>`_)

Improved Documentation
----------------------
- Add an introduction to Storm libraries and types.
(`2670 <https://github.com/vertexproject/synapse/pull/2670>`_)
(`2683 <https://github.com/vertexproject/synapse/pull/2683>`_)
- Fix small typos and corrections in the devops documentation.
(`2673 <https://github.com/vertexproject/synapse/pull/2673>`_)

====================

Features and Enhancements
-------------------------
- Updates to the ``inet`` and ``infotech`` models.
(`2666 <https://github.com/vertexproject/synapse/pull/2666>`_)

``:sandbox:file``
Add a ``sandbox:file`` property to record an initial sample from a
sandbox environment to the following forms:

``it:exec:proc``
``it:exec:thread``
``it:exec:loadlib``
``it:exec:mmap``
``it:exec:mutex``
``it:exec:pipe``
``it:exec:url``
``it:exec:bind``
``it:exec:file:add``
``it:exec:file:del``
``it:exec:file:read``
``it:exec:file:write``
``it:exec:reg:del``
``it:exec:reg:get``
``it:exec:reg:set``


``it:host:activity``
Update the interface to add a ``sandbox:file`` property to record an
initial sample from a sandbox environment.

- Changed primary Storm parser to a LALR compatible syntax to gain 80x speed
up in parsing Storm queries
(`2649 <https://github.com/vertexproject/synapse/pull/2649>`_)
- Added service provisioning API to AHA service and associated tool
``synapse.tools.aha.provision.service`` and documentation to make
it easy to bootstrap Synapse services using service discovery and
SSL client-side certificates to identify service accounts.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added user provisioning API to AHA service and associated tools
``synapse.tools.aha.provision.user`` and ``synapse.tools.aha.enroll``
to make it easy to bootstrap new users with SSL client-side certificates
and AHA service discovery configuration.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added automatic mirror initialization logic to Synapse services to
enable new mirrors to be initilized dynamically via AHA provisioning
rather than from a pre-existing backup.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added ``handoff()`` API to Synapse services to allow mirrors to be
gracefully promoted to leader.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added ``synapse.tools.promote`` to allow easy promotion of mirror to
leader using the new ``handoff()`` API.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added ``aha:provision`` configuration to Synapse services to allow
them to automatically provision and self-configure using AHA.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Adjusted Synapse service configuration preference to allow runtime settings
to be stored in ``cell.yaml``.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added optional ``certhash`` parameter to telepath ``ssl://`` URLs to
allow cert-pinning behavior and automatic trust of provisioning URLs.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added ``synapse.tools.moduser`` and ``synapse.tools.modrole`` commands
to modernize and ease user/role management from within Synapse service
docker containers.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Add ``$lib.jsonstor.cacheget()`` and ``lib.jsonstor.cacheset()`` functions
in Storm to easily implement data caching in the JSONStor.
(`2662 <https://github.com/vertexproject/synapse/pull/2662>`_)
- Add a ``params`` option to ``$lib.inet.http.connect()`` to pass parameters
when creating Websocket connections in Storm.
(`2664 <https://github.com/vertexproject/synapse/pull/2664>`_)

Bugfixes
--------
- Added ``getCellRunId()`` API to Synapse services to allow them to detect
incorrect mirror configurations where they refer to themselves.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Ensure that CLI history files can be read and written upon
starting interactive CLI tools.
(`2660 <https://github.com/vertexproject/synapse/pull/2660>`_)
- Assorted unit tests fixes to make tests more stable.
(`2656 <https://github.com/vertexproject/synapse/pull/2656>`_)
(`2665 <https://github.com/vertexproject/synapse/pull/2665>`_)
- Fix several uses of Python features which are formally deprecated
and may be removed in future Python versions.
(`2668 <https://github.com/vertexproject/synapse/pull/2668>`_)

Improved Documentation
----------------------
- Added new Deployment Guide with step-by-step production ready deployment
instructions
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Refactored Devops Guide to give task-oriented instructions on performing
common devops tasks.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added new minimal Admin Guide as a place for documenting Cortex admin tasks.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Updated Getting Started to direct users to synapse-quickstart instructions.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Added ``easycert`` tool documentation.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Removed ``cmdr`` tool documentation to emphasize newer tools such as
``storm``.
(`2641 <https://github.com/vertexproject/synapse/pull/2641>`_)
- Update the list of available Advanced and Rapid Power-Ups.
(`2667 <https://github.com/vertexproject/synapse/pull/2667>`_)

====================

Features and Enhancements
-------------------------
- Update the allowed versions of the ``pyopenssl`` and ``pytz`` libraries.
(`2657 <https://github.com/vertexproject/synapse/pull/2657>`_)
(`2658 <https://github.com/vertexproject/synapse/pull/2658>`_)

Bugfixes
--------
- When setting ival properties, they are now properly merged with existing
values. This only affected multi-layer views.
(`2655 <https://github.com/vertexproject/synapse/pull/2655>`_)

====================

Bugfixes
--------
- Fix a parsing regression in inet:url nodes related to unencoded "" symbols
in URLs.
(`2653 <https://github.com/vertexproject/synapse/pull/2653>`_)

====================

Features and Enhancements
-------------------------
- Updates to the ``inet`` and ``infotech`` models.
(`2634 <https://github.com/vertexproject/synapse/pull/2634>`_)
(`2644 <https://github.com/vertexproject/synapse/pull/2644>`_)
(`2652 <https://github.com/vertexproject/synapse/pull/2652>`_)

``inet:url``
The ``inet:url`` type now recognizes various ``file:///`` values from
RFC 8089.

``it:sec:cve``
The ``it:sec:cve`` type now replaces various Unicode dashes with hyphen
characters when norming. This allows a wider range of inputs to be
accepted for the type. Scrape related APIs have also been updated to
match on this wider range of inputs.

- The Cell now uses ``./backup`` as a default path for storing backups in, if
the ``backup:dir`` path is not set.
(`2648 <https://github.com/vertexproject/synapse/pull/2648>`_)
- Add POSIX advisory locking around the Cell ``cell.guid`` file, to prevent
multiple processes from attempting to start a Cell from the same directory.
(`2642 <https://github.com/vertexproject/synapse/pull/2642>`_)
- Change the default ``SLAB_COMMIT_WARN`` time from 5 seconds to 1 second, in
order to quickly identify slow storage performance.
(`2630 <https://github.com/vertexproject/synapse/pull/2630>`_)
- Change the Cell ``iterBackupArchive`` and ``iterNewBackupArchive`` routines
to always log exceptions they encounter, and report the final log message
at the appropriate log level for success and failure.
(`2629 <https://github.com/vertexproject/synapse/pull/2629>`_)
- When normalizing the ``str`` types, when ``onespace`` is specified, we skip
the ``strip`` behavior since it is redundant.
(`2635 <https://github.com/vertexproject/synapse/pull/2635>`_)
- Log exceptions raised by Cell creation in ``initFromArgv``. Catch
``lmdb.LockError`` when opening a LMDB database and re-raise an exception
with a clear error message.
(`2638 <https://github.com/vertexproject/synapse/pull/2638>`_)
- Update schema validation for Storm packages to ensure that cmd arguments do
not have excess fields in them.
(`2650 <https://github.com/vertexproject/synapse/pull/2650>`_)

Bugfixes
--------
- Adjust comma requirements for the JSON style list and dictionary expressions
in Storm.
(`2636 <https://github.com/vertexproject/synapse/pull/2636>`_)
- Add Storm query logging in a code execution path where it was missing.
(`2647 <https://github.com/vertexproject/synapse/pull/2647>`_)
- Tuplify the output of ``synapse.tools.genpkg.loadPkgProto`` to ensure that
Python list constructs ``[...]`` do not make it into Power-Up documentation.
(`2646 <https://github.com/vertexproject/synapse/pull/2646>`_)
- Fix an issue with heavy Stormtypes objects where caching was preventing
some objects from behaving in a dynamic fashion as they were intended to.
(`2640 <https://github.com/vertexproject/synapse/pull/2640>`_)
- In norming ``int`` values, when something is outside of the minimum or
maximum size of the type, we now include the string representation of the
valu instead of the raw value.
(`2643 <https://github.com/vertexproject/synapse/pull/2643>`_)
- Raise a ``NotReady`` exception when a client attempts to resolve an
``aha://`` URL and there have not been any ``aha`` servers registered.
(`2645 <https://github.com/vertexproject/synapse/pull/2645>`_)

Improved Documentation
----------------------
- Update Storm command reference to add additional commands.
(`2633 <https://github.com/vertexproject/synapse/pull/2633>`_)
- Expand Stormtypes API documentation.
(`2637 <https://github.com/vertexproject/synapse/pull/2637>`_)
(`2639 <https://github.com/vertexproject/synapse/pull/2639>`_)