Synapse

--------------------

New Features
- 542 - The ``Daemon`` now automatically calls ``item.fini()`` for items which are made from a Daemon configuration which are EventBus objects when the Daemon is ``fini()``'d. This allows the Daemon to tear down all instances of eventbus objects which it makes without having to share the object and set the ``onfini`` option when configuring the share.

Enhancements
- 541 - Added ``exe``, ``proc`` and ``host`` secondary properties to the ``inet:dns:look`` format to allow capturing a DNS lookup which may have originated from a file, process or a host.
- 540 - When the socket multiplexer does an ``accept()`` call, the remote ip and port are logged at the debug (``logging.DEBUG``) log level.

Bugs
- 58, 537 - Fixed IPv6 type norm() operations for OSX by using the ``ipaddress`` library instead of the ``s_socket.inet_ntop()`` function.
- 543 - Recent changes to pytest (included in the base image used for doing CI testing) changed how logging is performed. This disables those pytest changes.

--------------------

New Features
- 529 - Synapse Docker container ``vertexproject/synapse`` is now built off of a base container, ``vertexproject/synaspe-base-image:py36``. This container is hosted from https://github.com/vertexproject/synapse-base-image and is also used for CI testing. The synapse-base-image contains all of the dependencies required for Synapse, as well as having software updated via `apt-get`. The base container also has ``:py35`` and ``:py34`` tags available as well.
- 523 - Added ``Cortex.extCoreFifo()`` to put of a list of items in a named Cortex FIFO.
- 523 - Added ``DataModel.addPropTypeHook()`` to allow a DataModel user to fire a function whenever a type is used to define a property. This can be used to define callbacks by a ``CoreModule``.
- 523, 538 - Added ``synapse.lib.db`` to handle pooled connections to databases. Added ``synapse.lib.sqlite`` to handle SQLite specific DB optimizations.
- 523 - Added ``synapse.lib.gis`` to handle geospatial computations.
- 523 - Added ``synaspe.lib.iq.TestSteps`` helper. This allows for for interlocking events for multithreaded tests.
- 523 - Multiple improvements to ``CoreModule``’s. They may now get a unique ``_mod_iden`` property by an implementor. This value can be retrieved with the ``CoreModule.getModIden()`` API. Added ``getModProp()`` and ``setModProp()`` APIs so that the CoreModule can store data in the attached ``Cortex`` object. Added a ``finiCoreModule()`` API which is automatically registered as a fini function for the CoreModule. CoreModule implementors can override this API in order to have resources torn down automatically.
- 523 - Added ``synapse.lib.revision`` module to provide helpers for doing revision path enforcement.
- 523 - Added ``syn:alias`` node types to allow for a global GUID alias for a given iden.
- 523 - Added ``synapse.models.geospace.LatLongType`` and ``synapse.models.geospace.DistType`` for normalizing Latitude/longitude data and distances.
- 523 - Added ``geo:nloc`` form to allow tracking the physical location of a given node over time.
- 539 - Added ``inet:wifi:ap`` node type to allow the intersection of a SSID and a BSSID value.
- 539 - Added ``tel:mob:imid`` form to represent the knowledge of an IMEI and IMSI together.
- 539 - Added ``tel:mob:imsiphone`` node type to represent the knowledge of an IMEI and a telephone together.

Enhancements
- 528 - When a property value is included in the Storm ``stats()`` operator, that value is now normed using ``getPropNorm()``.
- 529, 532 - The ``SvcProxy`` now refires ``syn:svc:init`` and ``syn:svc:fini`` events, so users of the SvcProxy may now react to those events to know that a service has been added or removed from the ``SvcBus``.
- 534 - Log messages for exceptions which occur on the ``EventBus`` which cause an exception in ``dist()`` function now include the repr of the ``EventBus`` object so it is clear what type of object had the error, and the mesg itself.
- 534 - ``traceback.print_exc()`` calls have been replaced by `logger.exception()`` calls so traceback information is directed through logging mechanisms, instead of being printed to stdout.
- 534 - The ``Axon.has()`` api now validates the ``hvalu`` parameter is not None before querying the Axon db via getTufosByProp, to ensure that we are not returning an arbitrary ``axon:blob`` tufo.
- 523 - ``Cortex.formTufoByProp()`` will now fire ``node:set:prop`` events for each property in a newly created node. This does not affect splice generation.
- 523 - ``Cortex.delTufo()`` will now fire ``node:set:prop`` events for each secondary property in the deleted node to indicate the new-valu is None.
- 523 - Added ``SynTest.getDirCore()`` and ``SynTest.getTestSteps()`` helpers for getting directory backed Cortexes and TestStep objects, respectively.
- 523 - ``CoreModule.getModPath()`` Now returns None if the Cortex the module is loaded in is not a directory backed Cortex.
- 523 - ``synapse.lib.queue.Queue`` now has a ``size()`` API and a ``__len__`` implementation which allows inspection of how many items are in the internal ``collections.deque`` object.
- 523 - Added ``synapse.lib.scope.pop()`` and ``synapse.lib.scope.Scope.pop()`` methods, which allow either a thread or object local scope to have a named object pop’ed out of it, similar to ``dict.pop()``.
- 523 - Storm syntax integer parser now supports parsing negative values (starting with a ``-`` sign), parsing values which start with ``0x`` as hex values, parsing values which start with ``0b`` as binary strings, and parsing floats properly.
- 539 - ``CompType`` forms now accept dictionary of values as input. They keys which map to fields and optfields are used to form the node.
- 539 - Added ``ipv4``, ``tcp4``, and ``udp4`` secondary properties to ``inet:dns:look``. These represent the IP address which requested the look, and the servers which may have responded to the look.

Bugs
- 529 - The ``SvcProxy`` object was incorrectly registering services by name, as well as tags, in its' ``ByTag`` helper. This was causing the ``SvcProxy`` to think additional services were still available after they were no longer available to the ``SvcBus``. This has been corrected, and the ``SvcProxy`` no longer misuses the ``ByTag`` helper.
- 531 - The atexit handler for the ``EventBus`` had a bad reference which could have triggered a NameError on shutdown. This has been fixed.
- 533 - Change the ``Axon`` test test_axon_host_spinbackup to use waiters on ``syn:svc:init`` events to address a race condition.
- 534 - Additional proxy objects are fini'd during Axon tests.
- 523 - ``Socket.send()`` now catches ``OSError`` and ``ConnectionError`` exceptions and fini’s the socket if they occur.

Documentation
- 527 - Added Storm documentation for the ``stats()`` operator.
- 534 - Docstrings for ``synapse.lib.persist.Dir.items()``, ``synapse.lib.service.SvcProxy.callByTag``, ``synapse.lib.service.runSynSvc`` have been rewritten.

--------------------

New Features
- 524 - The Cortex class has a new API ``getCoreMods``, which returns a list of the currently loaded CoreModules in the Cortex.

Enhancements
- 522 - Exceptions raised during the thread Pool ``_run_work`` function are logged with additional information about what failed to run.

Bugs
- 522 - The ``synapse.lib.msgpack.en()`` function's use of the global msgpack.Packer object was wrapped in a try/except block; so that in the event of an exception during packing, we call the ``reset`` method to clear internal buffers of the object. It was possible that a serialization failure leaves data in the object, which would then be passed along to a subsequent caller. See https://github.com/msgpack/msgpack-python/issues/258 for example code showing this issue.
- 522 - Ensure that the axonbus Proxy objects made by Axon and Axonhost objects are fini'd.
- 522 - Fini more objects during Axon and Telepath tests which were not properly fini'd.
- 525 - The Axon ``_fireAxonClones`` function did not wait for its existing clones to come online (since they are handled by threads) befor entering the ``_findAxonClones`` routine. This could have caused the Axon to attempt to make additional clones for itself until the number of clones the Axon had loaded met the ``axon:clones`` option. The ``_fireAxonClones`` clones routine now waits 60 seconds for each previously known clone to come online before attempting to bring new clones online for itself.
- 526 - Pypi package had included a `scripts` package. This included development related scripts and was not intended for redistribution; and it collides with an existing `scripts` package on Pypi.

Documentation
- 522 - Update docstrings for ``telepath.openurl`` and ``telepath.openlink`` APIs.
dditional clones for itself until the number of clones the Axon had loaded me

--------------------

New Features
- 504 - Universal node properties, ``tufo:form`` and ``node:created``, are now model properties. Those properties do not have a form associated with them. In addition, the universal node property ``node:ndef`` was added. This is the guid derived from the primary property and primary property together, giving a way to universally represent a node value in a anonymous form. Universal properties are now added to the the datamodel documentation generated by autodoc. The associated migration for adding ``node:ndef`` values to nodes migrates all forms loaded into the Cortex datamodel at the time of startup. Depending on the size of a Cortex, this migration may take a long time to complete and it is encouraged that large (10 million+ node) Cortexes have a test migration done on a backup of the Cortex.
- 515 - Add a ``inet:addr`` type, which normalizes both IPV4 and IPV6 values to a single IPV6 value, which will produce a IPV4 sub if the address is part of the v6 -> v4 mapped space.
- 515 - Add a ``inet:dns:req`` form to record a DNS request which was made by an IP at a given time.
- 515 - Add a ``inet:dns:type`` type to enumerate different types of DNS requests.

Enhancements
- 516 - The ``task:<taskname>`` events fired by the Storm task() operator includes all the nodes in the current query set under the ``nodes`` value, instead of firing a single event per node under the ``node`` key.
- 504 - The msgpack helpers, ``synapse.common.msgenpack``, ``synapse.common.msgunpack`` and ``synapse.common.msgpackfd`` were removed. They are duplicates of functionality present in ``synapse.lib.msgpack`` content. They are replaced by ``synpase.lib.msgpack.en``, ``synapse.lib.msgpack.un`` and ``synapse.lib.fd respectively``.

Bugs
- 517 - The ``Cortex.delTufoTag`` API did not return the tufo to the caller. It now returns the modified tufo to the caller.
- 518 - Ensure Axon resources are fini'd during Axon related tests.
- 519 - The tests for normalizing the string ``'now'`` as a ``time`` type are more forgiving of system load.

Documentation
- 512 - Added style guide notes to prefer returning None over raising exceptions.
- 513 - Added filter documentation for storm
- 520 - Added a link to the Synapse slack chat to the readme.rst file. Invite your friends, they're welcome here!

--------------------

New Features
- 502 - Added the ``dir:///`` handler for opening a Cortex (currently SQLite backed) by file path.
- 502 - Added a Telepath reminder API to facillitate server side statefullness on Proxy reconnect.
- 502 - Added a Cortex metadata directory configable option and helpers for CoreModules to access that directory.
- 507 - Added ``inet:dns:rev6`` form for recording IPV56 PTR lookups.

Enhancements
- 502 - Added a reqPerm() API helper to require a user have a given permission.
- 502 - Removed old/broken session management code.

Bugs
- 170, 501 - Replaced the Python ``re`` module with ``regex``. This addresses a unicode parsing error in the ``re`` module which prevented the correct identification of some punycode encoded FQDN values.
- 508 - Add a signal handler for ``SIGTERM`` to the ``Eventbus.main()`` function. This allows for gracefully shutting down a dmon which was started in a Docker contain. Previously, ``SIGTERM`` was not caught and caused the Python process to close ungracefully.
- 509 - Removed Python 3.7 from test matrix until ``synapse.async`` library is removed.
- 509 - Fixed a bug in formTufoByProp which allowed the formation of nodes which were valid props, but not actually forms.
- 509 - Fixed a bug in storm that prevented setting read-only properties on nodes which may not have had the read-only property present.

Documentation
- 503 - Added docstrings to inet.py, dns.py and files.py models.
- 505 - Added link to docker for the ``vertexproject/synapse`` images.
- 510 - Added docs for running PostgreSQL cortex tests manually with Docker.
- 511 - Added code style guidelines to indicate the preference of the ``regex`` module over the use of ``re``.

--------------------

New Features
- 480 - Added a fully asynchronous push FIFO structure in order to support future Synapse built services.
- 490 - Added a ``make:json`` typecast which can be used to cast an object into a JSON string.
- 492 - Added a JSONL to messagepack tool. ``synapse.tools.json2mpk`` can be used to convert a JSONL file to a stream of messagepack's blobs.
- 496 - Added a ingest helper ``setGestFunc`` to the IngestApi mixin. This allows a function to be registered which performs data ingest without relying on a full ingest definition being created.
- 480 - Configable objects have a new method, `reqConfOpts()`. This method checks all configabl options; if an option has the property 'req' which evaluates to True and the value is not set on the object, a ReqConfOpt exception is thrown. This can be used to enforce an object to have specific configuration options set.

Enhancements
- 490 - Ensured Synapse was generating pretty JSON strings in places where a human may end up reading the JSON directly.
- 497 - Made axon exception logging more verbose.
- 489 - Docker images are now built in DockerCloud using a Dockerfile contained in the Synapse repository. The ``vertexproject/synapse`` image will use the ``python:3.6.3-slim`` base image moving forward, as to keep container size smaller.
- 480 - During a graceful shutdown, an atexit handler will now attempt to ``fini()`` all EventBus objects which have not been fini()'d and have had the ``self._fini_atexit`` flag set to True on them.
- 480 - AtomFile objects may now be truncated to reduce their size.


Bugs
- 487 - Removed Python 2.7 from the list of suppported Python versions in setup.py trove classifiers.
- 491 - Fixed a race condition in splicepump tests for ``node:created`` values.
- 494 - Added a minimum and maximum value for the ``IntType`` integer value to ensure it is bound within a signed 64big value. This is reflective of storage limitations of the SQLITE and PSQL storage backings. This ensures that we cannot make a node in one storage backing that cannot be moved to another storage backing because of storage-specific issues.
- 499 - The storm pivot operator was not runt-node aware; so it was unable to pivot to runt nodes. This has been fixed.
- 498 - Telepath's ``Proxy`` object was unable to successfully reconnect to a shared object if the Proxy object had event handlers registered to it. The order of operations for handling a reconnection has been changed to allow this to function properly.

Documentation
- 488 - Updated scheduler persec/loop function docstrings to clarify the return values and ability to cancel future tasks.