Synapse

--------------------

New Features
- 477 - Added ``node:created`` universal TUFO property. This is set when a node is created via formNodeByProp, and enables lifting/sorting nodes by the time they are created. Existing nodes will have ``node:created`` props set on them based on the Cortex timestamp value of their ``node:form`` property. Since this requires lifting every ``tufo:form`` row in a Cortex, it is reccomended that this is first tested in a copy of any production cortexes before doing a production deployment; so any neccesary outage windows can be planned.
- 484 - The Cortex ``axon:url`` configable option now accepts a URL to a service bus. It will create an AxonCluster object if that is the case.

Enhancements
- 478 - Properties which are read-only will be able to be set on a node if that property does not exist on the node.
- 485 - Test context managers in ``synapse.lib.iq`` now properly clean up after themselves in the event of a test failure/error.
- 485 - Added setTstEnvars context manager to the SynTest class to enable running tests with specific environmental variables set.

Bugs
- 459, 478 - Refactor how nodes are created using formTufoByProp. This has the impact that nodes automatically created via the autoadds mechanism will now have any secondary properties available added to them from the process of doing data normalization.

Documentation
- 483 - Remove outdated readme examples
- 486 - Add docstrings to inet:iface properties.

--------------------

New Features
- 473 - Added ``it:prod:soft``, ``it:prod:softver``, ``it:hostsoft`` types and associated forms, to allow tracking software, versions of software and software installed on a given host.
- 473 - Added ``it:semver`` data type for doing type normalization of Semantic Version numbers. Added helper functions for both Semantic version parsing and generic version parsing to ``synapse.lib.version``
- 473 - Added ``it:version:brute`` typecast to attempt parsing a version string into a normalized system value that can be used to do ordered comparison of version strings.
- 476 - Added ``inet:iface`` type and form for modeling a network interface on a particular device being bound to a particular IP, host, phone or wifi SSID. Added ``inet:wifi:ssid`` type.
- 476 - Added ``ps:contact`` type and form to act as a conglomerate of contact information for a individual.
- 476 - Added ``tel:mob:tac``, ``tel:mod:imei`` and ``tel:mob:imsi`` types and forms for modeling cellphone related information. This includes parsing and validation of pre-2004 IMEI/IMSI numbers.
- 482 - Moved test helper functions from ``synapse.tests.common`` to ``synapse.lib.iq`` so other users of Synapse can reuse our pre-existing test helpers (SynTest, TestEnv and TstOutput).

Enhancements
- 465 - Added ``axon:listener``, ``axon:tags`` and ``axon:syncopts`` to the AxonHost class, so these default values can be passed to Axons made by an AxonHost.
- 479 - Add test for calling the storm ``task()`` operator on a remote cortex with a local calback handler.

Bugs
- 475 - Changed PropValuType to use reqPropNorm instead of getPropNorm to enforce that the property referred to BY the type must be a modeled property.

--------------------

New Features
- 471 - The dmon tool, ``synapse.tools.dmon`` can now accept the log level via a environmental variable, ``SYN_DMON_LOG_LEVEL``. This can be added as an environmental variable in a docker compose file using the Vertex Project Synapse Docker images to configure the logging level.

Enhancements
- 467 - Added ``it:exec:proc:path``, ``it:exec:proc:src:proc`` and ``it:exec:proc:src:exe`` properties to the ``it:exec:proc`` form. Removed the ``it:exec:subproc`` form since it is not needed with the ``:src:`` properties on ``it:exec:proc``.
- 467 - Removed the computer science model (``compsci.py``) since it was superseded by the host execution model.
- 467 - Added ``inet:flow:src:exe`` and ``inet:flow:dst:exe`` properties to ``inet:flow`` to allow modeling data between ``file:bytes`` nodes.
- 468, 469, 472 - Added pytest-xdist to the testrunner.sh script to speed up local (dev) test runs of synapse.
- 470 - Remove unnecessary docker related functionality.

Bugs
- 466, 474 - Fixed bug in non-blocking SSL link which would sometimes prevent data from being transmitted

Documentation
- 462 - Added documentation for the host execution model in ``infotech.py``.

--------------------

New Features
- 456 - A global thread pool has been added to Synapse and a Task object convention added for executing tasks in the pool. This is in preparation of future feature support.

Enhancements
- 461 - The storm query operator ``refs()`` now also lifts nodes by prop-valu combination in order to get nodes which may refer to the source nodes. This allows identifying XREF nodes which point TO the inbound node.
- 463 - The TimeType now norms the string "now" as the current system time.
- 464 - Added a "guid" helper for Ingest to assist in making GuidType nodes without having to form strings out of variables.

--------------------

New Features
- 446, 450 - Adds the ability for GuidType nodes to normalize a list of items, in order to generate stable guids for potentially re-encounterable data. This only works when generating a property norm value (getPropNorm) and does not work for purely type normalization (getTypeNorm). Storm keyword list argument parsing can be used to generate stable GUID using the CLI, Ingest or Storm mechanisms.
- 452 - Synapse now stores the current version of the Synapse library in the Cortex blob store at the end of Cortex initialization. This was done in order to prepare for eventually enforcing required upgrade paths for data migrations or other features.
- 447 - Added the new form ``inet:web:postref`` XREF to track an ``inet:web:post`` which refers to another node.
- 447 - Added the new form ``inet:web:action`` GUID to to track an arbitrary action by an ``inet:web:acct`. The actions tracked by this are by defined by [Synapse] user convention.
- 447 - Added the new form ``inet:web:actref`` XREF to track how an ``inet:web:action`` may have interacted with another node.
- 454 - Added the ``inet:web:chprofile`` GUID to track previous values of a ``inet:web:acct`` node, representing changes to user accounts or profiles.
- 454 - Added the ``inet:web:post:repost`` property to track the concept of a ``inet:web:post`` being a copy of another post.
- 455 - Added a pair of Storm (and Cortex) Configable options to enable and set logging levels for Storm queries. These are ``storm:query:log:en`` and ``storm:query:log:level``. This logs what the query is and what the user execution context was.
- 426 - Axon and AxonHost objects are now Configable objects with configuration definitions that are used to define their behavior.

Enhancements
- 442 - Python 2.7 support dropped from Synapse.
- 447 - Migrated all inet:net* forms to the inet:web:* space. The following is a map of the migrated forms and their corresponding new forms:

('inet:netuser', 'inet:web:acct')
('inet:netgroup', 'inet:web:group')
('inet:netmemb', 'inet:web:memb')
('inet:follows', 'inet:web:follows')
('inet:netpost', 'inet:web:post')
('inet:netfile', 'inet:web:file')
('ps:hasnetuser', 'ps:haswebacct')
('ou:hasnetuser', 'ou:haswebacct')

These forms will automatically be migrated in existing Cortexes. If XREF types were used to point to any of these forms and the cortex was not first migrated to v0.0.26, the XREF type migration will fail. It is recommended that users first upgrade to v0.0.26 prior to upgrading to v0.0.27.
- 447 - Added Storage.updateProperty() and Storage.updatePropertyValu() APIs to the Cortex storage layer for doing bulk property and property-by-value updates. These are explicitly NOT exposed in the Cortex class.
- 449 - Thinned out some components of the EventBus class for performance reasons. This did result in the removal of the synapse.eventbus.on() decorator for decorating functions to be used as event callbacks.
- 456 - Removed unused Synapse modules: synapse.hivemind, synapse.mindmeld, synapse.lib.moddef.
- 426 - Logging in tests is now controlled by the environmental variable ``SYN_TEST_LOG_LEVEL`` which, as an integer, will set the logging level used by the root logger.
- 426 - The environmental variable ``SYN_TEST_SKIP_LONG`` can be set to a non-zero integer to skip potentially long running tests. This can shave up to a minute of test execution time.
- 426 - Axons now have the in-memory cache enabled on their Cortex by default.
- 458 - The ``inet:web:acct:occupation`` property has been changed from a ``str:txt`` type to ``str:lwr`` to allow for better foldability between user-declared occupations.

Bugs
- 443 - Make the daemon return more useful error messages when an exception has occurred during execution of a remote request.
- 444, 445 - Allow an inet:srv4 type to accept an integer string as input. Also adds additional boundary checking when norming an ip:port string to ensure that irreversible inputs are not accepted.
- 453 - Cleaned up skifIfNoInternet() test helpers. They will now be allowed to fail unless the ``SYN_TEST_SKIP_INTERNET`` environmental variable is set to a non-zero integer.
- 426 - The synapse.lib.heap.Heap class was not properly responding to ``heap:resize`` events. This was remedied.
- 426 - Wrapped a .items() iterator in synapse.daemon.OnHelp with a list to prevent a RuntimeError.
- 426 - Fix the synapse.lib.service.SvcProxy.getSynSvcs() method to return a Telepath safe list instead of a dict.values() view object.
- 426 - Fix the synapse.lib.service.SvcProxy.__init__ to strap in event handlers AFTER initializing instance variables to avoid a race condition on startup.
- 426 - AxonHost now waits before advertising itself on the bus, and properly calculates the number of axons it needs to make. This addressed an issue where the AxonHost was generating a non-deterministic number of Axons.
- 426 - The Axon’s thread to make clones for itself on a ServiceBus now waits until a remote clone is made. This addressed an issue where the Axon would make extra clones for itself.

Documentation
- 448 - Added in-model documentation for the file: model defined in files.py.
- 451 - Added user guide information for Storm lift operations, lift(), guid() and alltags().
- 426 - Docstrings in the synapse.axon module were rewritten or added when needed.

--------------------

New Features
- 438 - Added PropValu datatype to synapse. This allows a secondary (or primary) property to be modeled as a string in the form `property=<repr valu>`. This type also yields sub of "prop" representing the property, and "strval" or "intval" being the system normalized value of the property. This allow for node creation where a reference to another node property is needed but cannot be defined up front in the model. The additional subs allow for filtering/pivoting operations on nodes which use the PropValu type. A simple example of the string form for a PropValue is `inet:ipv4=1.2.3.4`. A more complex example of the string form for a PropValu is `inet:passwd=oh=my=graph!`
- 438 - XREF types have been updated to use the PropValu datatype instead of storing data in the property columns. This removes any extra-model data from the property columns in Cortex rows, and allows the implementation of pivotable XREF nodes. The string syntax for XREF creation was changed to be in line with the Comp datatype syntax, which looks now looks like ``. This will affect any ingests or programmatic creation of xref nodes done by users. An example of the string form of a Xref now looks like the following: `(98db59098e385f0bfdec8a6a0a6118b3,"inet:passwd=oh=my=graph!")`. Note that the PropValue portion of the value is quote delimited.

Enhancements
- 438 - The `refs()` storm operator is updated to be aware of secondary properties which are PropValu types and will pivot off of them, even if the props themselves are not forms.
- 438 - The file:imgof and file:txtref nodes no longer have glob property xref:*, instead :xref is now a PropValu and has the additional :xref:prop, :xref:intval and :xref:strval secondary props. This does require a data migration, so deployment of v0.0.26 should be tested on Cortexes which use those forms prior to production use.
- 438 - Migrated unittests to using getRamCore() helper when possible.
- 438 - Added unittest self.len() helper. Started some migration there.
- 439 - Cleaned up Socket() class implementation to be better, stronger and faster.
- 440 - Moved the syn:type, syn:form and syn:prop forms (and other items declared in DataModel) to be part of the "syn" model. These core elements are now themselves introspect-able.
- 440 - Added syn:prop:base and syn:prop:relname to syn:prop nodes, so those nodes can now be lifted by a basename or a relative property name.
- 441 - Added Python 3.7 RC1 to the test matrix. Pinned postgres test image to postgres:9.6.
- The Storm runtime now respects the `storm:limit:lift` configable value as the default limit when lifting nodes.

Bugs
- 440 - Changed how runtime nodes (runts) used to represent the data model in a cortex are created. These are now made from the type and property definitions based on the loaded data model; having been guided by the data model being processed. This restores things like "syn:prop:form" which was accidentally dropped from nodes.