Synapse

--------------------

New Features
- 590 - Added ``synapse.lib.remcycle.fetch()`` API which allows retrieving a URL using Tornado and executing a user provided callback to process the response.

Enhancements
- 589 - Added the ability to set a ``tags`` directive in an Ingest ``forms`` definition; which allows setting per-form tags without adding the tags to the current or child scopes.

--------------------

New Features
- 588 - Added a RFC2822 address parser and type/form, ``inet:rfc2822:addr``. This normalizes and parses string identifiers and attempts to extract email addresses as a secondary property.

Bugs
- 587 - Make `synapse.lib.msgpack` helper functions resilient to unmatched unicode surrogate characters. This also affects the ``synapse.lib.socket.Socket`` and ``synapse.lib.persist.Dir`` msgpack unpackers. This is done by passing ``unicode_errors='surrogatepass'`` to the msgpack pack/unpack functions. This makes synapse more resilient to malformed string data which may be encountered in the real world.

Documentation
- 587 - Removed ``synapse.statemach``. It was previously used for providing object persistence at an API level but has been unused within Synapse and was generally not a safe tool to use for third party use.

--------------------

New Features
- 574 - Added ``EventBus.onWith()`` API. This is a context manager which acts like ``.on()``, but the callback is removed when the context manager is exited.
- 575 - Added ``synapse.lib.iq.CmdGenerator()`` class to allow testing CLI command loops using unittest mock.
- 577 - Added ``synapse.lib.certdir.genClientCert()`` API to allow creation of a PKCS12 certificate bundle for a user certificate, private key and CA cert.
- 577 - Added a ``--p12`` option to the easycert tool. This allows a user to bundle their certificate, private key and CA cert into a PKC12 formatted file.
- 578 - Added the Storm macro syntax ``<-`` to represent a ``join()`` operation.

Enhancements
- 576 - The ``Daemon`` now fini's ``EventBus()`` objects in LIFO order when it is fini'd. In other words, objects created in a dmon configuration are torn down in reverse order that they are created.
- 576, 581 - Increased test coverage for CLI related tests.
- 577 - Updated tests for ``synapse.lib.certdir`` to ensure that the certificates made by ``certdir`` were correct.
- 577 - Updated tests for ``synapse.tools.easycert`` to ensure that the certificates made by ``easycert`` were correct.
- 578 - Updated the storm ``join()`` operator syntax to behave exactly like the ``pivot()`` operator with respect to source and destination properties. ``join()`` still is an additive operator which does not consume the source nodes.
- 583 - ``synapse.lib.remcycle.Hypnos`` now registers and persists ingest definitions as ``syn:ingest`` nodes in its Cortex.

Bugs
- 579 - Fix a reference to ``onCtx`` (the original name for the ``EventBus.onWith()`` function).
- 584 - ``Cortex`` Storage backings had different behaviors when limit=0 was passed to functions which joined rows together to make tufos. This has been fixed, so that a limit=0 API parameter will return 0 rows.

Documentation
- 573 - Added telepath docstrings for ``evalurl()``.
- 577 - Rewrote API documentation for ``synapse.lib.certdir``.
- 586 - Updated the Storm ``join()`` documentation to reflect changes to its arguments.

--------------------

New Features
- 564 - Added a ``int:2:str10`` typecast to convert a integer value to the base10 string value.
- 569 - The Telepath ``Proxy`` object now fires the local event ``tele:sock:runsockfini`` when the Proxy is attempting to reconnect to its remote ``Daemon``. This can be hooked by implementers to detect the loss of an existing connection to a remote ``Daemon``.
- 572 - The ``synapse.axon.AxonCluster`` helper object is now a subclass of ``EventBus``. This will now automatically close its corresponding ``SvcProx`` instance on its own ``.fini()``.
- 572 - Added the ``synapse.lib.iq.objhierarchy()`` function to extract a type hierarchy from nested tufo-style objects. This can be a useful for debugging.

Enhancements
- 565 - Increase Tornado related test timeouts from 5 to 30 seconds.
- 570 - Additional debug logging added to ``synapse.axon.Axon`` actions.
- 572 - Additional debug logging added to ``synapse.lib.persist.Dir`` actions.

Bugs
- 561 - When a telepath Proxy object is made, by default it will create its own socket Plex object. This Plex is now fini’d when the proxy is fini’d.
- 568 - Add ``synapse.glob`` to ``synapse.lib.socket`` imports to fix a missing reference.
- 567- The ``synapse.lib.persist.Dir.items()`` method yielded offsets which could switch from being relative offsets to being absolute offsets. This has been changed to always yield absolute offsets.
- 570 - The ``synapse.axon.Axon._fireAxonClone`` thread now uses the ``tele:sock:runsockfini`` event to detect that a disconnect has occurred for its remote Proxy, and uses that to break out of its innermost ``synapse.lib.persist.Dir.items()`` loop. The previous behavior would fail to reconnect if part of an Axon cluster went down and then came back up, as the Proxy contained potentially invalid link information instead of getting an updated link tufo from the Axon’s ServiceBus.
- 571 - Fini ``Axon`` threads after calling ``self.syncdir.fini()`` to ensure that any threads which are in the real time event pump from the syncdir can safely close their resources before attempting to stop any threads.

Documentation
- 572 - Miscellaneous API docstring improvements.

--------------------

New Features
- 553 - Synapse Docker images will now have ``-debug`` images available as well for all master and tagged builds. This image is built off of the ``vertexproject/pydebug:3.6-slim`` image. This image has full GDB support for Python enabled. See https://github.com/vertexproject/pydebug for more information.

Enhancements
- 551 - Make 4096 bit RSA keys by default when using the ``synaspse.tools.easycert`` tool. Thanks galyac!
- 550 - Add additional debug logging to Synapse when starting up a Cortex, executing remote methods via Telepath and around SSL errors.
- 554 - Validate the Storm ``tree()`` ``recurlim`` argument is greater than or equal to zero.
- 557 - ``inet:dns:look:tcp4:ipv4`` and ``inet:dns:look:tcp4:udp4`` sub properties of ``port`` and ``ipv4`` have been added, so a DNS response can be recorded from a server even if the server source port is unknown.
- 559 - Add a limit parameter to the Storm ``totags()`` operator.
- 560 - Add unit test for ``synapse.lib.dark``.

Bugs
- 555 - Set the x509 extension number of certificates made by ``synaspse.tools.easycert`` to version 3 (0x02). This allows Chrome to validate the certificates starting in Chrome 63.
- 558 - When an exception is raised during the ``synapse.lib.socket.Plex._plexMainLoop`` call to ``select.select()``, perform a check to see if any sockets are fini'd or have fileno() == -1 and perform a cleanup operation on that socket. This can reduce a race condition where a bad socket can get wedged and prevent the multiplexor from working correctly.
- 558 - Change the ``synapse.link.ssl.Socket.send()`` implementation to call ``self.sock.send()`` itself instead of calling through the default ``Socket.send()`` method. This allows us to properly catch ``SSLWantReadError`` and ``SSLWantWriteError`` exceptions; instead of allowing teh default method to catch them as ``OSError`` exceptions and fini the socket. The previous behavior lead to SSL Sockets being unstable.
- 556 - Fix the Storm filter function for ``+range()`` so that filter operator works properly.
- 562 - Fix the Storm filter function for ``+in()`` so that filter operator works properly.

Documentation
- 552 - Add documentation for the Storm operators to the user guide: ``pivot()``, ``join()``, ``refs()``, ``fromtags()``, ``totags()``, ``jointags()``, and ``tree()``.

--------------------

New Features
- 545 - Added storm macro function ``get:tasks`` and an API for introspecting tasks which have been registered on a Cortex.

Enhancements
- 544 - Added new fields (``url`` and ``whois:fqdn``) to ``whois:contact``.
- 547 - Enabled pyup monitoring for Synapse release notes and added config file to disable pyup update checks.
- 549 - Removed ``cryptography`` from setup.py.

Documentation
- 548 - Added CHANGELOG.md to maintain release notes within the repository.