Synapse

Latest version: v2.171.0

Safety actively analyzes 638430 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 49 of 53

0.0.49

Not secure
--------------------
V0.0.49 is primarily a bugfix release to address regressions and issues from v0.0.47.

Enhancements
- 706 - The ``Cortex.formNodeByProp()`` API will now call ``setTufoProps()`` on a node if it already exists and properties are passed into ``formNodeByProp``. This allows properties to be added in a “last caller wins” scenario.
- 706 - The ``node:add`` splice now accepts a ``tags`` value which contains a list of tags to add to the node created by the splice.

Bugs
- 705 - Fix an issue in ``Configable``\’s ``confdef`` decorator helper for identifying config options in classes. This method was previously skipping Telepath proxy objects during introspection; and a regression was introduced which didn’t properly skip non-callable object methods. This caused an issue when doing introspection on objects which had Telepath ``Proxy`` objects as the value of class attributes. This regression has been fixed by properly skipping non-callable class members.
- 706 - Splice changes introduced in v0.0.47 were improperly wrapping messages. This meant that the ``Cortex.splice()`` and ``Cortex.splices()`` APIs required splice messages as though they came off of the ``EventBus``. This required sending duplicate, redundant data over the network during remote calls. This splice reactors now expect unpacked splices. This does not affect splices written to splice files or splices converted to the new format, since those splices were already unwrapped.
- 707 - The Telepath socket multiplexor (``synapse.lib.socket.Plex``) had an issue added where xlist sockets may not have been properly culled from the list of Sockets maintained by the plex. The xlist sockets were the same as the rlist sockets, so this has been simplified. In addition, an identity check for fileno was changed to a equality check.

Documentation
- 706 - Examples of all different``splice`` message forms have been added in a ``Cortex`` test ``test_cortex_splices``. This can be used as an immediate reference to people doing manual splice creation.
- 708 - Added ``synapse.cryotank.CryoCell`` to the automatically generated ``Configable`` docs.

0.0.48

Not secure
--------------------

New Features
- 698 - Add ``source``, ``seen``, ``record`` and ``recref`` forms to allow for tracking data sources and references in a ``Cortex`` hypergraph.
- 698 - Add ``ou:meet``, ``ou:meet:attendee``, ``ou:conference``, and ``ou:conference:attendee`` forms to allow for tracking meetings and conferences, as well as their attendees.

Enhancements
- 702 - ``synapse.lib.datapath`` now has a ``DataPath()`` class which is used to represent the steps in a given path. This allows pre-computation of a given datapath to reduce overhead when the same path may need to be parsed multiple times.

Bugs
- 703 - Ensure that the ``AxonCell`` was sending the correct marshalled exceptions.
- 704 - Change the ``auth`` value of the ``Cortex`` configuration for a ``cell:pool`` to be a Base64 encoded version of the msgpack encoded auth dictionary. This allows for configuring the ``cell:pool`` via a JSON ``Daemon`` configuration file.

0.0.47

Not secure
--------------------

Summary

There are a sizeable amount of changes included in v0.0.47. A few items are highlighted here; please see the sections after the summary for a full list of new features and enhancements.

Notable New features

- A new data storage object, the ``CryoTank``, has been introduced. This utilizes LMDB to perform fast storage and retrieval of structured records. This can be used to quickly store and retrieve large feeds of data.
- A new service architecture has been implemented in Synapse. This uses an asynchronous, generator based approach to networking and remote calling. This has been utilized to reimplement the ``Axon`` storage (now ``AxonCell``), as well as implementing the networking protocol for the ``CryoTank``, the ``CryoCell``. Detailed documentation for this service architecture will be included in future Synapse release.
- A new LMDB key-value store helper has been added, the ``KvStor``. This is used to implement persist dictionary and set objects, which are used as helpers to the ``Cell``. Previously, a service could use their own Cortex as a simple KV store, but that was potentially heavy thing to do. This alternative is considerably lighter for these simple use cases.
- Storm now supports the use of subquery filters. An example query is the following: ``inet:dns:a -{ :ipv4 -> inet:ipv4 +cc=us } `` which will lift all ``inet:dns:a`` nodes, then filter nodes whose IPv4 address has a ``us`` country code. Detailed documentation for this will be added in a future Synapse release.

Backwards Incompatible Changes

- A global socket multiplexor, implemented using EPOLL, has been added to Synapse. This change means that Synapse will only work on Linux at the moment. The Vertex Project does plan on restoring Windows and OSX support in a future release.
- The Cortex Splice format has changed, in order to support better filtration. This requires that splice producers and splice consumers update simultaneously to v0.0.47. If users have existing splice logs that need to be updated, a helper function has been written to to convert an existing splice log file to the new format. See the notes below for additional information.
- The following Synapse components have been removed or altered significantly:
* ``synapse.axon`` has been rewritten substantially. In addition, it no longer imports ``synapse.lib.hashset`` using a star import.
* ``synapse.lib.persist`` has been removed. It is no longer used in core Synapse code.
* ``synapse.lib.heap`` has been removed. It contained an implementation error and is no longer used in core Synapse code.
* ``synapse.lib.atomfile`` has been changed to currently only support Linux.

Known Issues

- 700 - The ``axon:upload`` handler on the ``AxonCell`` can allow bytes to be stored twice in the same ``BlobStor``. This may result in a ``BlobStor`` having bytes which the ``AxonCell`` does not have direct knowledge of. No data is lost here, but extra disk space may be consumed.

New Features
- 637, 650, 695 - Added ``synapse.lib.cell``. This contains the ``Cell`` class and related helper classes. The ``Cell`` is the base class for a microservices architecture. Similar to a ``CoreModule``, the ``Cell`` is designed to be subclassed and have some functions overridden. See ``Cell`` docstrings for additional notes.
- 637, 695 - Added ``synapse.neuron`` module. This contains the ``Neuron()`` class which is used as a service directory for Synapse ``Cell`` based services. The ``Neuron`` is responsible for doing service provisioning and service name resolution. Additional documentation related to Neuron, Cell architectures will be included in a future Synapse release.
- 650, 679 - The ``synapse.axon.Axon`` implementation was removed, and replaced by a new ``AxonCell`` and ``BlobCell`` implementations. This decouples the ``Axon`` idea from the ``Cortex`` implementation, and makes the ``AxonCell`` solely responsible for storing files and doing retrieval by SHA256 hash. The ``AxonCell`` acts as a indexing master, while multiple ``BlobCells`` need to be used in order to store files. The ``AxonClient`` helper is available to interact with a ``AxonCell``, while the ``BlobClient`` is available to interact with the ``BlocCell``. The use of this new storage implementation does require the deployment of a ``Neuron``, since these are all ``Cell`` based services. Additional documentation related to Axon architecture will be included in a future Synapse release.
- 637, 654, 655, 674 - Added a new module, ``synapse.cryotank``. It contains a ``CryoTank()`` class, which is used for storing structured data (anything that can be msgpacked) into a LMDB database. It also contains a ``CryoCell()`` class for creation, management, deletion of ``CryoTank`` objects, as well as for putting data into remote ``CryoTank``’s. It has a ``CryoClient()`` helper object which can be used to interact with the remote ``CryoCell``.
- 648, 649, 672, 674, 675 - Added ``synapse.tools.cryo.cat`` to look at data from remote ``CryoTank``s, dump that data to disk in different formats, or add data to a remote ``CryoTank`` when reading data from stdin.
- 681 - Add ``cellpool:conf`` config option to the ``Cortex``. This is a dictionary which takes three value, a ``auth`` blob for talking to a ``Neuron``, as well as the ``host`` and ``port`` used to connect to the ``Neuron``. The configuration of a ``cellpool:conf``, and subsequentially ``axon:name``, is required in order to allow a ``Cortex`` to save bytes to a ``AxonCell`` using the ``formNodeByBytes()`` and ``formNodeByFd()`` APIs.
- 637 - Added ``RetnWait()`` to the ``synapse.lib.thread``. This can (and should) be used as a context manager, which can emulate a synchronous callback occuring, while waiting for a local ``thread.event`` to be set.
- 650 - Added ``synapse.lib.lmdb``. This contains several LMDB related classes for future use with new Cortex storage layers. The APIs for these objects should not be considered stable for third party use at this time.
- 637, 642, 658 - Added a new module, ``synapse.lib.net``, to handle asynchronous networking code. This includes a new EPoll based multiplexer for Socket IO. An instance of this plex is used as part of a global plex (located in ``synapse.glob``). This change currently makes Synapse only compatible with Linux operating systems. Eventually, we’ll re-add Windows and OSX support in a future release.
- 687 - Added ``ps:persona`` and ``ps:persona:has`` forms to represent the idea of a persona.
- 637 - Added a new way to marshall exceptions, ``synapse.common.getexcfo()`` which will be the preferred way to marshall exceptions moving forward for Synapse code.
- 636, 638, 676, 678, 689 - Added the ``synapse.lib.crypto`` module. This contains several new modules. ``synapse.lib.crypto.tinfoil`` contains a simple AESGCM encryption/decryption class called ``TinFoilHat``. ``synapse.lib.crypto.ecc`` contains helpers for doing ECC key pair generation and doing ECDHE key exchange. ``synapse.lib.crypto.vault`` contains a ``Vault()`` class. The ``Vault()`` can be used to store key pairs and do key signing locally.
- 634 - The Storm CLI command ``jointags()`` was split into ``jointags()`` and ``pivottags()`` which behave similarly to the ``pivot()`` and ``refs()`` commands, respectively. The old default behavior for ``jointags()`` was moved to ``pivottags()``, and the current ``jointags()`` behavior now reflects the old ``jointags(keep_nodes=1)`` behavior.
- 628, 659 - Added ``synapse.lib.kv``. This module contains the ``KvStor()`` class; which implements a LMDB backed kv-store. An instance of the ``KvStor()`` object can be used to get ``KvLook()``, ``KvDict()`` and ``KvSet()`` objects. The ``KvLook`` can be used to do ``key=valu`` type storage, which is directly backed by the LMDB database. The ``KvDict`` can be used for ``key=valu`` storage, but it also keeps a in-memory dictionary of the contents to avoid database lookups. The ``KvSet`` can be treated similarly to a ``set()`` object, but addition and removal is backed by the LMDB database.
- 691 - Added a initial HTTP model for tracking to the ``inet`` model. Added ``inet:http:request``, ``inet:http:response``, ``inet:http:header``, ``inet:http:param``, ``inet:http:reqhead``, and ``inet:http:reqparam`` forms.
- 606 - Added ``synapse.lib.membrane`` and ``Membrane()`` class. The ``Membrane`` object can be used as a filter on the events fired by a ``EventBus`` to determine whether or not they are sent to a function provided to the ``Membrane()`` object.
- 606 - Added a ``membranes`` config option to the ``Cortex`` which applies rules to ``splice`` events and puts splices into a named ``Fifo`` for later consumption.

Enhancements
- 667 - Added subquery based filtration for Storm. Subqueries take all of the query nodes, apply either a positive or negative a filter to those nodes, and then add matching nodes back to the query results.
- 685 - Migrated ``ou:has``\* forms to a generic ``ou:org:has`` Xref form. Add ``seen:min`` and ``seen:max`` secondary properties.
- 650 - Added ``genraises()`` API to ``synapse.lib.iq.SynTest``. This mimics the ``raises()`` API, but expects the function to be a generator. This consumes the generator in a bound local and expects that to raise.
- 650 - Added ``getAxonCore()`` API to ``synapse.lib.iq.SynTest`` to get a ``TstEnv`` instance which has a Axon, Cortex, Blob, Neuron, Daemon and clients configured already.
- 650 - Added ``synapse.common.ehex()``/``synapse.common.uhex()`` wraps around ``binascii.hexlify()`` and ``binascii.unhexlify()`` APIs.
- 650 - Added ``synapse.common.buid()`` to generate a 32 byte hash. Unlike the ``guid()`` API, this returns bytes. It can take a msgpackable object to generate a stable buid.
- 650 - Added ``synapse.common.spin()`` to consume a generator and discard the results as fast as possible.
- 686 - Updated ``synapse.tools.autodoc`` to extract configable docs for a defined list of classes which use the ``initConfDefs()`` API to to define their config options.
- 650 - Added ``initConfDefs()`` API to ``synapse.lib.config.Configable()``. This function is executed during ``__init__``. It is intended to be overridden by implementers, and will replace use of the ``confdef`` decorator function over time. Care must be taken by implementers when class mixing or inheriting that a new class properly calls or overrides ``initConfDefs``.
- 650 - Updated ``synapse.lib.queue.Queue`` behavior to raise an exception (``IsFini`` or ``TimeOut``) when ``get()`` has no objects to return. Previously this returned ``None``; however it was impossible to distinguish if the queue legitimately returned ``None`` or there was a timeout or if the ``Queue`` had been fini’d.
- 673, 681 - Changed the ``Cortex`` config option of ``axon:url`` to be ``axon:name``. This is the name of the ``AxonCell`` resolve via the cores’ ``CellPool``.
- 670 - ``ou:hasalias`` form to note when a org has an alias or alternative name.
- 671, 687 - Migrated ``ps:has``\* forms to a single ``ps:person:has`` xref form. Add ``seen:min`` and ``seen:max`` secondary properties.
- 631 - The ``synapse.async`` module was renamed to ``synapse.synasync`` in order to allow for future compatibility with Python 3.7.
- 637, 650 - Added ``iterfile()``, ``loadfile()`` and ``dumpfile()`` APIs to ``synapse.lib.msgpack``. These iterate over a file, load the contents of a file, and save a value to a file, respectively.
- 637 - Removed ``synapse.lib.heapfile``. It had implementation errors which caused issues with data recovery and is no longer used for backing ``Axon`` storage. In addition, ``synapse.lib.persist`` was also removed since it is no longer being used.
- 637 - Added ``synapse.lib.const`` to store constants.
- 637 - The ``AtomFile`` implementations were simplified to use ``pread``/``pwrite`` and drop memory mapped file support.
- 637 - A new decorator API, ``inpool`` was added to fire functions in the synapse global thread pool. This is available from ``synapse.glob``. Assuming ``synapse.glob`` is imported as ``s_glob``, it can decorate functions as ``s_glob.inpool``.
- 637 - The ``DmonConf()`` class has been updated to parse a ``cells`` directive, which can be used to launch ``Cell`` based objects. These objects are run in their own processes.
- 640, 641 - The ``SynTest`` helper function ``getLoggerStream()`` now yields a ``io.StringIO`` object which is mixed in with a ``threading.Event`` object. In addition, the ``getLoggerStream()`` now takes a second, optional, argument which must be a string. When a log message containing that string is logged, the event will be set. This allows writing tests which wait for a log message to be logged prior to continuing the test.
- 635 - Added ``seen:min`` and ``seen:max`` secondary properties to ``inet:dns:req``.
- 632 - Added ``loadConfPath()`` API to ``syapse.lib.config.Configable`` to load configuration options from a JSON file at a given path.
- 629 - Changed the``digests()`` API in ``synapse.lib.hashset.HashSet()`` to return the digests in bytes instead hex.
- 627 - Added a ``waitall()`` API to ``synapse.lib.iq.TestSteps()`` to wait for all steps to complete.
- 626 - Added ``lockfile()`` API to ``synapse.common``. This is a helper for advisory locking around a file provided by the ``fcntl`` library.
- 620 - In Storm, lifts by form which are followed by a positive tag filter are now optimized to use a single lift-by-tag operation at the storage layer.
- 615 - The use of limits in the Storm query language has been normalized. For operators which consume the input set of nodes, a limit applies a maximum number of nodes to the output set. For operators which do not consume the input set of nodes, a limit applies to the maximum number of nodes added to the current working set.
- 619 - Add test showing that read-only properties can be set via ingest if they do not previously exist.
- 614 - Added a ``--importfile`` switch to the ``synapse.tools.easycert`` tool to assist with easily importing a certificate file to their certdir.
- 616 - Added ``isTufoProp()`` API to the ``DataModel`` to check to see if a property is a valid prop for the currently loaded data model.
- 600 - Splices format was changed to better encapsulate the splice to support filtering. Old splice logs will not be replayable as is, and can be converted with ``synapse.lib.splices.convertSpliceFd()``. This change is backwards incompatible with previous versions of synapse.
- 611 - Added ``seen:min`` and ``seen:max`` secondary properties to ``inet:urlredir``.
- 610 - Splice Pump producers have debug messages logged about splice consumption, errors, and remaining splices.

Bugs
- 690 - Defvals passed to configuration options for ``synapse.lib.config.Configable`` will now smash existing values for a given option when they are set.
- 493, 677 - Removed unused (and not working) ``ssh://`` link implementation stub.
- 669 - Removed unused ``parse_macro_join`` function from ``synapse.lib.syntax``.
- 668 - Fix an issue where the ``inet:fqdn`` normalization would allow an internationalized domain to be stored after doing IDNA encoding, but would fail to render in the Storm CLI due to it actually being invalid Punycode. The raw string is now returned if the string starts with ``xn--``. This is a harsh reminder the world is not tidy.
- 657 - Fix race conditions related to ``Fifo`` resync/ack/push operations.
- 652 - The ``synapse.lib.queue.Queue`` object would accept additional data after it had either ``done()`` or ``fini()`` called on it. This is no longer allowed.
- 651 - Fix a bug in ``synapse.lib.config.Configable`` where config options passed in during ``__init__`` were set prior to all confdefs being loaded.
- 637 - ``synapse.common.gendirs()`` would fail if the directories needed to be made by ``os.makedirs()`` already existed. This has been fixed.
- 633 - Removed ``py`` command from the base CLI commands. This could be used to perform authenticated remote code execution as the process running the Daemon object that a user connected to.
- 623 - ``node:del`` events fired by ``Cortex.delTufo()`` were incorrectly setting the primary property on the ``node:del`` events. This was preventing tag deletion from working properly.
- 617 - Clean up ``Cortex`` fifo APIs to not automatically incref ``Fifo`` objects for every object access.
- 618 - Refactored ``Cortex.__init__()`` to allow using the ``syn:cortex`` alias to start up a ``Cortex`` via a ``Daemon`` properly.
- 616 - Fix bug in ``_initCoreFifo`` which prevented Fifo’s from being initialized on startup properly. Changed the ``syn:fifo`` node to a comp type.

Documentation
- 630 - Updated docstrings in ``synapse.lib.sqlite``.

0.0.46

Not secure
--------------------

Bugs
- Pull in version bounds for third-party packages from the master branch. This addresses an immediate issue where Synapse fails to work with a pypi installation and Tornado 5.0.0 is installed.

0.0.45

Not secure
--------------------

New Features
- 661 - Added the ``inet:group`` form and type. This is analogous to the ``inet:user`` form for tracking a given group name.

Enhancements
- 666 - Add ``ps:person:name:en`` and various ``name:en`` subprops props to distinguish between localized and English versions of names.
- 661 - The ``inet:web:group`` form was made more robust. The second field of the sepr type was renamed to ``:id`` to represent a site specific identifier, freeing the ``:name`` property up to represent a descriptive name of the group. ``:name:en`` was added to provide for a English version of the name. ``:signup``, ``:signup:ipv4`` and ``signup:ipv6`` props were added to track when groups were created. ``:loc`` and ``:latlong`` props were added to allow for some location based tracking. ``:seen:min`` and ``:seen:max`` secondary properties were added to track when a group was active.
- 662 - The python ``msgpack`` library was version locked to 0.5.1.
- 663 - Added ``:name:en`` and ``:realname:en`` secondary props to ``inet:web:acct`` for tracking English specific versions of a name.

Bugs
- 624 - The ``ps:person`` was missing the ``:name:middle`` field. This has been added.
- 660 - The ``ou:member`` form was broken. This has been changed to be a Comp type and had the secondary property ``:end`` added to it.

0.0.44

Not secure
The minimum version of msgpack used by Synapse has been upgraded to be at least 0.5.0. This is complicated by the fact that the ``msgpack-python package`` was renamed to ``msgpack`` by its maintainers. Installation of Synapse from PyPi using the sdist and wheel packages should upgrade msgpack in a working state, and docker image users should not be affected by this change. Users which directly use a checkout of the Synapse Github repository should uninstall msgpack-python and then install msgpack. This can be done with the following commands (you may need to adjust them for your installation; accounting for aliases or the use of `python3`, etc):

- `python -m pip uninstall msgpack-python`
- `python -m pip install msgpack`

The ``synapse.lib.heap.Heapfile`` and ``synapse.axon.Axon`` structures are planned to undergo a significant rewrite, possibly landing as early as v0.0.45. These rewrites will not be backwards compatible, in order to account for design issues in these structures. If you are using these classes for doing data storage, please reach out to us via Slack as soon as possible (see https://github.com/vertexproject/synapse/blob/master/README.rst for a link to join our Synapse Slack chatroom).

New Features
- 592 - Added GUID type ``it:auth:passwdhash`` to store password hashes without requiring the plaintext password.
- 604 - Added ``synapse.lib.atomic.Counter()`` class as a thread safe counter object.
- 595 - Added the ``rcode`` secondary property to the ``inet:dns:look`` form to allow encoding the DNS response code for a given lookup.
- 595 - The ``it:dev:regkey`` type is now a lowercased string. Existing ``it:dev:regkey`` and ``it:dev:regval:key`` values will be automatically lowercased as a model migration.
- 607 - Added ``Axon.getAxonStatus()`` and ``AxonHost.getAxonHostStatus()`` APIs to get runtime information about Axons.

Enhancements
- 591 - Removed the Cortex CLI command ``nextseq`` in favor of using the Storm macro command ``nexttag()``.
- 591 - Add Storm test coverage.
- 593 - Added ``puts()`` API to the ``synapse.lib.fifo.Fifo`` object to the ``Fifo`` structure to allow for bulk object adding.
- 596 - Added Axon tests to show a clone sync operation restarting after a persist offset file is removed.
- 601 - Ensure the ``synapse.lib.heap.Heap`` file storage only grows upward and cannot be resized down by replaying a resize event.
- 602 - Set backoff / backoff-retry values for the Drone ci configuration for the git plugin.
- 603 - The ``synapse.lib.tags.tufoHasTag()`` and ``synapse.lib.tufo.tagged()`` APIs were equivalent functions. ``tufoHasTag`` has been removed.
- 598, 609 - Updated msgpack-python>=0.4.8 to ``msgpack=>0.5.1`` as a dependency.
- 598 - Updated ``synaspe.lib.msgpack.Unpk`` msgpack unpacker helper to use the new ``tell()`` API introduced in msgpack 0.5.0.
- 598 - Added a future-proofing msgpack test to ensure we break when the msgpack python API eventually changes to assuming a strict utf8 compliance and can then versionlock our use of msgpack.
- 605 - Added simple integrity checking to the ``synapse.lib.heap.HeapFile`` structure.

Bugs
- 591 - ``synapse.lib.cache.KeyCache`` behavior updated to no longer store cache valus if they are None. If None was cached, that value was unable to be updated later.
- 594, 597 - Fix Ingest user docs which were using outdated forms. Thanks tn3rt for the bug report!
- 599 - Fix ``AxonMixin.eatfd()`` and ``AxonMixin.eatbytes()`` to allow file descriptors and bytes which have zero bytes to be consumed.

Documentation
- 605 - Rewrote API docstrings for ``synapse.lib.heap.HeapFile``.

Page 49 of 53

Links

Releases

Has known vulnerabilities

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.