Synapse

--------------------

New Features
- 504 - Universal node properties, ``tufo:form`` and ``node:created``, are now model properties. Those properties do not have a form associated with them. In addition, the universal node property ``node:ndef`` was added. This is the guid derived from the primary property and primary property together, giving a way to universally represent a node value in a anonymous form. Universal properties are now added to the the datamodel documentation generated by autodoc. The associated migration for adding ``node:ndef`` values to nodes migrates all forms loaded into the Cortex datamodel at the time of startup. Depending on the size of a Cortex, this migration may take a long time to complete and it is encouraged that large (10 million+ node) Cortexes have a test migration done on a backup of the Cortex.
- 515 - Add a ``inet:addr`` type, which normalizes both IPV4 and IPV6 values to a single IPV6 value, which will produce a IPV4 sub if the address is part of the v6 -> v4 mapped space.
- 515 - Add a ``inet:dns:req`` form to record a DNS request which was made by an IP at a given time.
- 515 - Add a ``inet:dns:type`` type to enumerate different types of DNS requests.

Enhancements
- 516 - The ``task:<taskname>`` events fired by the Storm task() operator includes all the nodes in the current query set under the ``nodes`` value, instead of firing a single event per node under the ``node`` key.
- 504 - The msgpack helpers, ``synapse.common.msgenpack``, ``synapse.common.msgunpack`` and ``synapse.common.msgpackfd`` were removed. They are duplicates of functionality present in ``synapse.lib.msgpack`` content. They are replaced by ``synpase.lib.msgpack.en``, ``synapse.lib.msgpack.un`` and ``synapse.lib.fd respectively``.

Bugs
- 517 - The ``Cortex.delTufoTag`` API did not return the tufo to the caller. It now returns the modified tufo to the caller.
- 518 - Ensure Axon resources are fini'd during Axon related tests.
- 519 - The tests for normalizing the string ``'now'`` as a ``time`` type are more forgiving of system load.

Documentation
- 512 - Added style guide notes to prefer returning None over raising exceptions.
- 513 - Added filter documentation for storm
- 520 - Added a link to the Synapse slack chat to the readme.rst file. Invite your friends, they're welcome here!

--------------------

New Features
- 502 - Added the ``dir:///`` handler for opening a Cortex (currently SQLite backed) by file path.
- 502 - Added a Telepath reminder API to facillitate server side statefullness on Proxy reconnect.
- 502 - Added a Cortex metadata directory configable option and helpers for CoreModules to access that directory.
- 507 - Added ``inet:dns:rev6`` form for recording IPV56 PTR lookups.

Enhancements
- 502 - Added a reqPerm() API helper to require a user have a given permission.
- 502 - Removed old/broken session management code.

Bugs
- 170, 501 - Replaced the Python ``re`` module with ``regex``. This addresses a unicode parsing error in the ``re`` module which prevented the correct identification of some punycode encoded FQDN values.
- 508 - Add a signal handler for ``SIGTERM`` to the ``Eventbus.main()`` function. This allows for gracefully shutting down a dmon which was started in a Docker contain. Previously, ``SIGTERM`` was not caught and caused the Python process to close ungracefully.
- 509 - Removed Python 3.7 from test matrix until ``synapse.async`` library is removed.
- 509 - Fixed a bug in formTufoByProp which allowed the formation of nodes which were valid props, but not actually forms.
- 509 - Fixed a bug in storm that prevented setting read-only properties on nodes which may not have had the read-only property present.

Documentation
- 503 - Added docstrings to inet.py, dns.py and files.py models.
- 505 - Added link to docker for the ``vertexproject/synapse`` images.
- 510 - Added docs for running PostgreSQL cortex tests manually with Docker.
- 511 - Added code style guidelines to indicate the preference of the ``regex`` module over the use of ``re``.

--------------------

New Features
- 480 - Added a fully asynchronous push FIFO structure in order to support future Synapse built services.
- 490 - Added a ``make:json`` typecast which can be used to cast an object into a JSON string.
- 492 - Added a JSONL to messagepack tool. ``synapse.tools.json2mpk`` can be used to convert a JSONL file to a stream of messagepack's blobs.
- 496 - Added a ingest helper ``setGestFunc`` to the IngestApi mixin. This allows a function to be registered which performs data ingest without relying on a full ingest definition being created.
- 480 - Configable objects have a new method, `reqConfOpts()`. This method checks all configabl options; if an option has the property 'req' which evaluates to True and the value is not set on the object, a ReqConfOpt exception is thrown. This can be used to enforce an object to have specific configuration options set.

Enhancements
- 490 - Ensured Synapse was generating pretty JSON strings in places where a human may end up reading the JSON directly.
- 497 - Made axon exception logging more verbose.
- 489 - Docker images are now built in DockerCloud using a Dockerfile contained in the Synapse repository. The ``vertexproject/synapse`` image will use the ``python:3.6.3-slim`` base image moving forward, as to keep container size smaller.
- 480 - During a graceful shutdown, an atexit handler will now attempt to ``fini()`` all EventBus objects which have not been fini()'d and have had the ``self._fini_atexit`` flag set to True on them.
- 480 - AtomFile objects may now be truncated to reduce their size.


Bugs
- 487 - Removed Python 2.7 from the list of suppported Python versions in setup.py trove classifiers.
- 491 - Fixed a race condition in splicepump tests for ``node:created`` values.
- 494 - Added a minimum and maximum value for the ``IntType`` integer value to ensure it is bound within a signed 64big value. This is reflective of storage limitations of the SQLITE and PSQL storage backings. This ensures that we cannot make a node in one storage backing that cannot be moved to another storage backing because of storage-specific issues.
- 499 - The storm pivot operator was not runt-node aware; so it was unable to pivot to runt nodes. This has been fixed.
- 498 - Telepath's ``Proxy`` object was unable to successfully reconnect to a shared object if the Proxy object had event handlers registered to it. The order of operations for handling a reconnection has been changed to allow this to function properly.

Documentation
- 488 - Updated scheduler persec/loop function docstrings to clarify the return values and ability to cancel future tasks.

--------------------

New Features
- 477 - Added ``node:created`` universal TUFO property. This is set when a node is created via formNodeByProp, and enables lifting/sorting nodes by the time they are created. Existing nodes will have ``node:created`` props set on them based on the Cortex timestamp value of their ``node:form`` property. Since this requires lifting every ``tufo:form`` row in a Cortex, it is reccomended that this is first tested in a copy of any production cortexes before doing a production deployment; so any neccesary outage windows can be planned.
- 484 - The Cortex ``axon:url`` configable option now accepts a URL to a service bus. It will create an AxonCluster object if that is the case.

Enhancements
- 478 - Properties which are read-only will be able to be set on a node if that property does not exist on the node.
- 485 - Test context managers in ``synapse.lib.iq`` now properly clean up after themselves in the event of a test failure/error.
- 485 - Added setTstEnvars context manager to the SynTest class to enable running tests with specific environmental variables set.

Bugs
- 459, 478 - Refactor how nodes are created using formTufoByProp. This has the impact that nodes automatically created via the autoadds mechanism will now have any secondary properties available added to them from the process of doing data normalization.

Documentation
- 483 - Remove outdated readme examples
- 486 - Add docstrings to inet:iface properties.

--------------------

New Features
- 473 - Added ``it:prod:soft``, ``it:prod:softver``, ``it:hostsoft`` types and associated forms, to allow tracking software, versions of software and software installed on a given host.
- 473 - Added ``it:semver`` data type for doing type normalization of Semantic Version numbers. Added helper functions for both Semantic version parsing and generic version parsing to ``synapse.lib.version``
- 473 - Added ``it:version:brute`` typecast to attempt parsing a version string into a normalized system value that can be used to do ordered comparison of version strings.
- 476 - Added ``inet:iface`` type and form for modeling a network interface on a particular device being bound to a particular IP, host, phone or wifi SSID. Added ``inet:wifi:ssid`` type.
- 476 - Added ``ps:contact`` type and form to act as a conglomerate of contact information for a individual.
- 476 - Added ``tel:mob:tac``, ``tel:mod:imei`` and ``tel:mob:imsi`` types and forms for modeling cellphone related information. This includes parsing and validation of pre-2004 IMEI/IMSI numbers.
- 482 - Moved test helper functions from ``synapse.tests.common`` to ``synapse.lib.iq`` so other users of Synapse can reuse our pre-existing test helpers (SynTest, TestEnv and TstOutput).

Enhancements
- 465 - Added ``axon:listener``, ``axon:tags`` and ``axon:syncopts`` to the AxonHost class, so these default values can be passed to Axons made by an AxonHost.
- 479 - Add test for calling the storm ``task()`` operator on a remote cortex with a local calback handler.

Bugs
- 475 - Changed PropValuType to use reqPropNorm instead of getPropNorm to enforce that the property referred to BY the type must be a modeled property.

--------------------

New Features
- 471 - The dmon tool, ``synapse.tools.dmon`` can now accept the log level via a environmental variable, ``SYN_DMON_LOG_LEVEL``. This can be added as an environmental variable in a docker compose file using the Vertex Project Synapse Docker images to configure the logging level.

Enhancements
- 467 - Added ``it:exec:proc:path``, ``it:exec:proc:src:proc`` and ``it:exec:proc:src:exe`` properties to the ``it:exec:proc`` form. Removed the ``it:exec:subproc`` form since it is not needed with the ``:src:`` properties on ``it:exec:proc``.
- 467 - Removed the computer science model (``compsci.py``) since it was superseded by the host execution model.
- 467 - Added ``inet:flow:src:exe`` and ``inet:flow:dst:exe`` properties to ``inet:flow`` to allow modeling data between ``file:bytes`` nodes.
- 468, 469, 472 - Added pytest-xdist to the testrunner.sh script to speed up local (dev) test runs of synapse.
- 470 - Remove unnecessary docker related functionality.

Bugs
- 466, 474 - Fixed bug in non-blocking SSL link which would sometimes prevent data from being transmitted

Documentation
- 462 - Added documentation for the host execution model in ``infotech.py``.