Synapse

=====================

Features and Enhancements
-------------------------
- Set ``Link`` high water mark to one byte in preparation for Python 3.11
support.
(`3064 <https://github.com/vertexproject/synapse/pull/3064>`_)
- Allow specifying dictionary keys in Storm with expressions and backtick
format strings.
(`3065 <https://github.com/vertexproject/synapse/pull/3065>`_)
- Allow using deref syntax (``*$form``) when lifting by form with tag
(``*$formtag``) and form with tagprop (``*$formtag:tagprop``).
(`3065 <https://github.com/vertexproject/synapse/pull/3065>`_)
- Add ``cron:start`` and ``cron:stop`` messages to the events emitted by the
``behold()`` API on the Cortex. These events are only emitted by the leader.
(`3062 <https://github.com/vertexproject/synapse/pull/3062>`_)

Bugfixes
--------
- Fix an issue where an Aha service running on a non-default port would
not have that port included in the default Aha URLs.
(`3049 <https://github.com/vertexproject/synapse/pull/3049>`_)
- Restore the ``view.addNode()`` Storm API behavior where making a node on
a View object that corresponds to the currently executing view re-used the
current Snap object. This allows nodeedits to be emitted from the Storm
message stream.
(`3066 <https://github.com/vertexproject/synapse/pull/3066>`_)

=====================

Features and Enhancements
-------------------------
- Add additional Storm commands to assist with managing Users and Roles in
the Cortex.
(`2923 <https://github.com/vertexproject/synapse/pull/2923>`_)
(`3054 <https://github.com/vertexproject/synapse/pull/3054>`_)

``auth.gate.show``
Shows the definition for an AuthGate.

``auth.role.delrule``
Used to delete a rule from a Role.

``auth.role.mod``
Used to modify properties of a Role.

``auth.role.del``
Used to delete a Role.

``auth.role.show``
Shows the definition for a Role.

``auth.role.list``
List all Roles.

``auth.user.delrule``
Used to delete a rule from a User.

``auth.user.grant``
Used to grant a Role to a User.

``auth.user.revoke``
Used to revoke a Role from a User.

``auth.role.mod``
Used to modify properties of a User.

``auth.user.show``
Shows the definition of a User.

``auth.user.list``
List all Users.

- Update some of the auth related objects in Storm:
(`2923 <https://github.com/vertexproject/synapse/pull/2923>`_)

``auth:role``
Add ``popRule()`` and ``getRules()`` functions. Add a ``.gates``
accessor to get all of the AuthGates associated with a role.

``auth:user``
Add ``popRule()`` and ``getRules()`` functions. Add a ``.gates``
accessor to get all of the AuthGates associated with a user.

- Add ``$lib.auth.textFromRule()``, ``$lib.auth.getPermDefs()`` and
``$lib.auth.getPermDef()`` Storm library APIs to assist with working
with permissions.
(`2923 <https://github.com/vertexproject/synapse/pull/2923>`_)
- Add a new Storm library function, ``$lib.iters.enum()``, to assist with
enumerating an iterable object in Storm.
(`2923 <https://github.com/vertexproject/synapse/pull/2923>`_)
- Update the ``NoSuchName`` exceptions which can be raised by Aha during
service provisioning to clarify they are likely caused by re-using the
one-time use URL.
(`3047 <https://github.com/vertexproject/synapse/pull/3047>`_)
- Update ``gen.ou.org.hq`` command to set ``ps:contact:org`` if unset.
(`3052 <https://github.com/vertexproject/synapse/pull/3052>`_)
- Add an ``optional`` flag for Storm package dependencies.
(`3058 <https://github.com/vertexproject/synapse/pull/3058>`_)
- Add ``.]``, ``[.``, ``http[:``, ``https[:``, ``hxxp[:`` and ``hxxps[:``
to the list of known defanging strategies which are identified and
replaced during text scraping.
(`3057 <https://github.com/vertexproject/synapse/pull/3057>`_)

Bugfixes
--------
- Fix an issue where passing a non-string value to ``$lib.time.parse``
with ``errok=$lib.true`` would still raise an exception.
(`3046 <https://github.com/vertexproject/synapse/pull/3046>`_)
- Fix an issue where context managers could potentially not release
resources after exiting.
(`3055 <https://github.com/vertexproject/synapse/pull/3055>`_)
- Fix an issue where variables with non-string names could be passed
into Storm runtimes.
(`3059 <https://github.com/vertexproject/synapse/pull/3059>`_)
- Fix an issue with the Cardano regex used for scraping addresses.
(`3057 <https://github.com/vertexproject/synapse/pull/3057>`_)
- Fix an issue where scraping a partial Cardano address could raise
an error.
(`3057 <https://github.com/vertexproject/synapse/pull/3057>`_)
- Fix an issue where the Storm API ``view.addNode()`` checked permissions
against the incorrect authgate. This API now only returns a node if the
View object is the same as the View the Storm query is executing in.
(`3060 <https://github.com/vertexproject/synapse/pull/3060>`_)

Improved Documentation
----------------------
- Fix link to Storm tool in Synapse Power-Ups section.
(`3053 <https://github.com/vertexproject/synapse/pull/3053>`_)
- Add Kubernetes deployment examples, which show deploying Synapse services
with Aha based provisioning. Add an example showing one mechanism to set
``sysctl``'s in a managed Kubernetes deployment.
(`3047 <https://github.com/vertexproject/synapse/pull/3047>`_)

=====================

Features and Enhancements
-------------------------
- Add a ``size()`` method on the STIX bundle object.
(`3043 <https://github.com/vertexproject/synapse/pull/3043>`_)
- Update the minimum version of the ``aio-socks`` library to ``0.8.0``.
Update some unittests related to SOCKS proxy support to account for
multiple versions of the ``python-socks`` library.
(`3044 <https://github.com/vertexproject/synapse/pull/3044>`_)

Improved Documentation
----------------------
- Update the Synapse documentation to add PDF and HTMLZip formats.

=====================

Features and Enhancements
-------------------------
- Added ``--try`` option to ``gen.risk.vuln``, ``gen.pol.country``,
``gen.pol.country.government``, and ``gen.ps.contact.email`` commands
and their associated Storm functions.
(`3030 <https://github.com/vertexproject/synapse/pull/3030>`_)
- Added ``$lib.gen.orgHqByName`` and ``$lib.gen.langByName``.
(`3030 <https://github.com/vertexproject/synapse/pull/3030>`_)
- Added the configuration option ``onboot:optimize`` to all services
to allow devops to delay service startup and allow LMDB to optimize
storage for both size and performance. May also be set by environment
variable ``SYN_<SERVICE>_ONBOOT_OPTIMIZE=1``
(`3001 <https://github.com/vertexproject/synapse/pull/3001>`_)
- Ensure that ``AuthDeny`` exceptions include the user iden in the ``user``
key, and the name in the ``username`` field. Previously the ``AuthDeny``
exceptions had multiple identifiers for these fields.
(`3035 <https://github.com/vertexproject/synapse/pull/3035>`_)
- Add an optional ``--view`` argument to the ``synapse.tools.storm`` CLI
tool. This allows a user to specify their working View for the Storm CLI.
This was contributed by captainGeech42.
(`2937 <https://github.com/vertexproject/synapse/pull/2937>`_)
- Updates to ``synapse.lib.scope`` and the ``Scope`` class. A ``Scope.copy()``
method has been added to create a shallow copy of a ``Scope``. A module
level ``clone(task)`` function has been added which clones the current task
scope to the target ``task``. Async Tasks created with ``Base.schedCoro()``
calls now get a shallow copy of the parent task scope.
(`3021 <https://github.com/vertexproject/synapse/pull/3021>`_)
- Add a new Storm command, ``batch``, to assist in processing nodes in batched
sets.
(`3034 <https://github.com/vertexproject/synapse/pull/3034>`_)
- Add global permissions, storm.macro.admin`` and ``storm.macro.edit``, to
allow users to administer or edit macros.
(`3037 <https://github.com/vertexproject/synapse/pull/3037>`_)
- Mark the following Storm APIs as safe to execute in read-only queries:
``$lib.auth.users.get()``, ``$lib.auth.users.list()``,
``$lib.auth.users.byname()``, ``$lib.auth.roles.get()``,
``$lib.auth.roles.list()``, ``$lib.auth.roles.byname()``,
``$lib.auth.gates.get()`` and ``$lib.auth.gates.list()``.
(`3038 <https://github.com/vertexproject/synapse/pull/3038>`_)
- Added ``uplink`` key to ``getCellInfo()``, which indicates whether
the Cell is currently connected to an upstream mirror.
(`3041 <https://github.com/vertexproject/synapse/pull/3041>`_)

Bugfixes
--------
- Fix an issue in the Storm grammar where part of a query could potentially
be incorrectly parsed as an unquoted case statement.
(`3032 <https://github.com/vertexproject/synapse/pull/3032>`_)
- Fix an issue where exceptions could be raised which contained data that was
not JSON serializable. ``$lib.raise`` arguments must now also be JSON safe.
(`3029 <https://github.com/vertexproject/synapse/pull/3029>`_)
- Fix an issue where a spawned process returning a non-pickleable exception
would not be handled properly.
(`3036 <https://github.com/vertexproject/synapse/pull/3036>`_)
- Fix an issue where a locked user could login to a Synapse service on a TLS
Telepath connection if the connection presented a trusted client certificate
for the locked user.
(`3035 <https://github.com/vertexproject/synapse/pull/3035>`_)
- Fix a bug in ``Scope.enter()`` where the added scope frame was not removed
when the context manager was exited.
(`3021 <https://github.com/vertexproject/synapse/pull/3021>`_)
- Restoring a service via the ``SYN_RESTORE_HTTPS_URL`` environment variable
could timeout when downloading the file. The total timeout for this process
has been disabled.
(`3042 <https://github.com/vertexproject/synapse/pull/3042>`_)

Improved Documentation
----------------------
- Update the Synapse glossary to add terms related to the permissions system.
(`3031 <https://github.com/vertexproject/synapse/pull/3031>`_)
- Update the model docstrings for the ``risk`` model.
(`3027 <https://github.com/vertexproject/synapse/pull/3027>`_)

Deprecations
------------
- The ``ctor`` support in ``Scope`` has been removed. The population of the
global default scope with environment variables has been removed.
(`3021 <https://github.com/vertexproject/synapse/pull/3021>`_)

=====================

Automatic Migrations
--------------------
- If the ``risk:vuln:cvss:av`` property equals ``V`` it is migrated to ``P``.
(`3013 <https://github.com/vertexproject/synapse/pull/3013>`_)
- Parse ``inet:http:cookie`` nodes to populate the newly added
``:name`` and ``:value`` properties.
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)
- See :ref:`datamigration` for more information about automatic migrations.

Features and Enhancements
-------------------------
- Added the ``belief`` model which includes the following new forms:
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

``belief:system``
A belief system such as an ideology, philosophy, or religion.

``belief:tenet``
A concrete tenet potentially shared by multiple belief systems.

``belief:subscriber``
A contact which subscribes to a belief system.

``belief:system:type:taxonomy``
A hierarchical taxonomy of belief system types.

- Added declaration for ``risk:compromise -(uses)> ou:technique``
light-weight edges.
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

- Updated ``inet:http:session`` and ``inet:http:request`` forms to
include the following property:
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

``:cookies``
An array of ``inet:http:cookie`` values associated with the node.

- Updated the ``inet:http:cookie`` form to include the following properties:
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

``name``
The name of the cookie preceding the equal sign.

``value``
The value of the cookie after the equal sign if present.

- Added logic to allow constructing multiple ``inet:http:cookie``
nodes by automatically splitting on ``;`` such as ``foo=bar; baz=faz``
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

- Updated ``it:log:event`` to add the following properties:
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

``type``
An ``it:log:event:type:taxonomy`` type for the log entry.

``ext:id``
An external ID that uniquely identifies this log entry.

``product``
An ``it:prod:softver`` of the product which produced the log entry.

- Updated the ``risk:compromise`` form to include the following properties:
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

``goal``
An ``ou:goal`` node representing the assessed primary goal of the
compromise.

``goals``
An array of ``ou:goal`` nodes representing additional goals of the
compromise.

- Updated ``risk:attack`` and ``risk:compromise`` forms to deprecate the
``techniques`` property in favor of using ``-(uses)> ou:technique``
light-weight edges.
(`3015 <https://github.com/vertexproject/synapse/pull/3015>`_)

- Updates to the ``inet:dns``, and ``media`` models.
(`3005 <https://github.com/vertexproject/synapse/pull/3005>`_)
(`3017 <https://github.com/vertexproject/synapse/pull/3017>`_)

``inet:dns:answer``
Remove all read-only flags present on the secondary properties for this
form.

``media:news``
Add an ``updated`` property to record last time the news item was updated.

- Updated ``inet:flow`` to include the following properties:
(`3017 <https://github.com/vertexproject/synapse/pull/3017>`_)

``src:ssh:key``
The key sent by the client as part of an SSH session setup.

``dst:ssh:key``
The key sent by the server as part of an SSH session setup.

``src:ssl:cert``
The x509 certificate sent by the client as part of an SSL/TLS negotiation.

``dst:ssl:cert``
The x509 certificate sent by the server as part of an SSL/TLS negotiation.

``src:rdp:hostname``
The hostname sent by the client as part of an RDP session setup.

``src:rdp:keyboard:layout``
The keyboard layout sent by the client as part of an RDP session setup.

- Add ``synapse.utils.stormcov``, a Coverage.py plugin for measuring code
coverage of Storm files.
(`2961 <https://github.com/vertexproject/synapse/pull/2961>`_)
- Clean up several references to the ``cell.auth`` object in HTTP API
handlers. Move the logic in ``/api/v1/auth/onepass/issue`` API handler to
the base Cell.
(`2998 <https://github.com/vertexproject/synapse/pull/2998>`_)
(`3004 <https://github.com/vertexproject/synapse/pull/3004>`_)
- Clarify the error message encountered by a Synapse mirrored service if
the mirror gets desynchronized from its upstream service.
(`3006 <https://github.com/vertexproject/synapse/pull/3006>`_)
- Update how read-only properties are handled during merges. The ``.created``
property will always be set when merging a node down. If two nodes have
other conflicting read-only property values, those will now emit a warning
in the Storm runtime.
(`2989 <https://github.com/vertexproject/synapse/pull/2989>`_)
- The ``Axon.wget()`` API response now includes HTTP request history, which is
added when the API request encounters redirects. The ``$lib.axon.wget()``
Storm API now includes information about the original request URL. This data
is now used to create ``inet:urlredir`` nodes, such as when the Storm
``wget`` command is used to retrieve a file.
(`3011 <https://github.com/vertexproject/synapse/pull/3011>`_)
- Ensure that ``BadTypeValu`` exceptions raised when normalizing invalid
data with the ``time`` type includes the value in the exception message.
(`3009 <https://github.com/vertexproject/synapse/pull/3009>`_)
- Add a callback on Slab size expansion to trigger a free disk space check
on the related cell.
(`3016 <https://github.com/vertexproject/synapse/pull/3016>`_)
- Add support for choices in Storm command arguments.
(`3019 <https://github.com/vertexproject/synapse/pull/3019>`_)
- Add an optional parameter to the Storm ``uniq`` command to allow specifying
a relative property or variable to operate on rather than node iden.
(`3018 <https://github.com/vertexproject/synapse/pull/3018>`_)
- Synapse HTTP API logs now include the user iden and username when that
information is available. For deployments with structured logging enabled,
the HTTP path, HTTP status code, user iden, and username are added to
that log message.
(`3007 <https://github.com/vertexproject/synapse/pull/3007>`_)
- Add ``web_useriden`` and ``web_username`` attributes to the Synapse HTTP
Handler class. These are used for HTTP request logging to populate
the user iden and username data. These are automatically set when a user
authenticates using a session token or via basic authentication.
The HTTP Session tracking now tracks the username at the time the session
was created. The ``_web_user`` value, which previously pointed to a heavy
HiveUser object, is no longer populated by default.
(`3007 <https://github.com/vertexproject/synapse/pull/3007>`_)
- Add ``$lib.inet.http.codereason`` Storm API for translating HTTP status
codes to reason phrases. ``inet:http:resp`` objects now also have a
``reason`` value populated.
(`3023 <https://github.com/vertexproject/synapse/pull/3023>`_)
- Update the minimum version of the ``cryptography`` library to ``39.0.1`` and
the minimum version of the ``pyopenssl`` library to ``23.0.0``.
(`3022 <https://github.com/vertexproject/synapse/pull/3022>`_)

Bugfixes
--------
- The Storm ``wget`` command created ``inet:urlfile`` nodes with the ``url``
property of the resolved URL from ``aiohttp``. This made it so that a user
could not pivot from an ``inet:url`` node which had a URL encoded parameter
string to the resulting ``inet:urlfile`` node. The ``inet:urlfile`` nodes
are now made with the original request URL to allow that pivoting to occur.
(`3011 <https://github.com/vertexproject/synapse/pull/3011>`_)
- The ``Axon.wget()`` and ``$lib.axon.wget()`` APIs returned URLs in the
``url`` field of their responses which did not contain fragment identifiers.
These API responses now include the fragment identifier if it was present in
the resolved URL.
(`3011 <https://github.com/vertexproject/synapse/pull/3011>`_)
- The Storm ``tree`` command did not properly handle Storm query arguments
which were declared as ``storm:query`` types.
(`3012 <https://github.com/vertexproject/synapse/pull/3012>`_)
- Remove an unnecessary permission check in the Storm ``movenodes`` command
which could cause the command to fail.
(`3002 <https://github.com/vertexproject/synapse/pull/3002>`_)
- When a user email address was provided to the HTTP API
``/api/v1/auth/adduser``, the handler did not properly set the email using
change controlled APIs, so that information would not be sent to mirrored
cells. The email is now being set properly.
(`2998 <https://github.com/vertexproject/synapse/pull/2998>`_)
- The ``risk:vuln:cvss:av`` enum incorrectly included ``V`` instead of ``P``.
(`3013 <https://github.com/vertexproject/synapse/pull/3013>`_)
- Fix an issue where the ``ismax`` specification on time types did not merge
time values correctly.
(`3017 <https://github.com/vertexproject/synapse/pull/3017>`_)
- Fix an issue where using a function call to specify the tag in a tagprop
operation would not be correctly parsed.
(`3020 <https://github.com/vertexproject/synapse/pull/3020>`_)

Improved Documentation
----------------------
- Update copyright notice to always include the current year.
(`3010 <https://github.com/vertexproject/synapse/pull/3010>`_)

Deprecations
------------
- The ``synapse.lib.httpapi.Handler.user()`` and
``synapse.lib.httpapi.Handler.getUserBody()`` methods are marked as
deprecated. These methods will be removed in Synapse ``v2.130.0``.
(`3007 <https://github.com/vertexproject/synapse/pull/3007>`_)

=====================

Features and Enhancements
-------------------------

- Updates to the ``biz``, ``file``, ``lang``, ``meta``, ``pol``, and
``risk`` models.
(`2984 <https://github.com/vertexproject/synapse/pull/2984>`_)

``biz:service``
Add a ``launched`` property to record when the operator first made the
service available.

``file:bytes``
Add ``exe:compiler`` and ``exe:packer`` properties to track the software
used to compile and encode the file.

``lang:language``
Add a new guid form to represent a written or spoken language.

``lang:name``
Add a new form to record the name of a language.

``meta:node``
Add a ``type`` property to record the note type.

``meta:note:type:taxonomy``
Add a form to record an analyst defined taxonomy of note types.

``pol:country``
Correct the ``vitals`` property type from ``ps:vitals`` to ``pol:vitals``.

``ps:contact``
Add a ``lang`` property to record the language specified for the contact.

Add a ``langs`` property to record the alternative languages specified for
the contact.

``ps:skill``
Add a form to record a specific skill which a person or organization may
have.

``ps:skill:type:taxonomy``
Add a form to record a taxonomy of skill types.

``ps:proficiency``
Add a form to record the assessment that a given contact possesses a
specific skill.

``risk:alert``
Add a ``priority`` property that can be used to rank alerts by priority.

``risk:compromise``
Add a ``severity`` property that can be used as a relative severity score
for the compromise.

``risk:threat``
Add a ``type`` property to record the type of the threat cluster.

``risk:threat:type:taxonomy``
Add a form to record a taxonomy of threat types.

- Add support for Python 3.10 to Synapse.
(`2962 <https://github.com/vertexproject/synapse/pull/2962>`_)
- Update the Synapse docker containers to be built from a Debian based image,
instead of an Ubuntu based image. These images now use Python 3.10 as the
Python runtime.
(`2962 <https://github.com/vertexproject/synapse/pull/2962>`_)
- Add an optional ``--type`` argument to the Storm ``note.add`` command.
(`2984 <https://github.com/vertexproject/synapse/pull/2984>`_)
- Add a Storm command, ``gen.lang.language``, to lift or generate a
``lang:language`` node by name.
(`2984 <https://github.com/vertexproject/synapse/pull/2984>`_)
- Update the allowed versions of the ``cbor2`` library; and upgrade the
versions of ``aiostmplib`` and ``aiohttp-socks`` to their latest versions.
(`2986 <https://github.com/vertexproject/synapse/pull/2986>`_)
- The ``X-XSS-Protection`` header was removed from the default HTTP API
handlers. This header is non-standard and only supported by Safari browsers.
Service deployments which rely on this header should use the
``https:headers`` configuration option to inject that header into their
HTTP responses.
(`2997 <https://github.com/vertexproject/synapse/pull/2997>`_)

Bugfixes
--------
- Malformed hash values normalized as ``file:bytes`` raised exceptions which
were not properly caught, causing Storm ``?=`` syntax to fail. Malformed
values are now properly handled in ``file:bytes``.
(`3000 <https://github.com/vertexproject/synapse/pull/3000>`_)

Improved Documentation
----------------------
- Update the Storm filters user guide to include expression filters
(`2997 <https://github.com/vertexproject/synapse/pull/2997>`_)
- Update Storm type-specific behavior user guide to clarify ``guid``
deconfliction use cases and some associated best practices.
(`2997 <https://github.com/vertexproject/synapse/pull/2997>`_)
- Update Storm command reference user guide to document ``gen.*`` commands.
(`2997 <https://github.com/vertexproject/synapse/pull/2997>`_)

Deprecations
------------
- The Cortex APIs ``provStacks()`` and ``getProvStack(iden)`` have been
removed.
(`2995 <https://github.com/vertexproject/synapse/pull/2995>`_)