Synapse

--------------------

New Features
- 456 - A global thread pool has been added to Synapse and a Task object convention added for executing tasks in the pool. This is in preparation of future feature support.

Enhancements
- 461 - The storm query operator ``refs()`` now also lifts nodes by prop-valu combination in order to get nodes which may refer to the source nodes. This allows identifying XREF nodes which point TO the inbound node.
- 463 - The TimeType now norms the string "now" as the current system time.
- 464 - Added a "guid" helper for Ingest to assist in making GuidType nodes without having to form strings out of variables.

--------------------

New Features
- 446, 450 - Adds the ability for GuidType nodes to normalize a list of items, in order to generate stable guids for potentially re-encounterable data. This only works when generating a property norm value (getPropNorm) and does not work for purely type normalization (getTypeNorm). Storm keyword list argument parsing can be used to generate stable GUID using the CLI, Ingest or Storm mechanisms.
- 452 - Synapse now stores the current version of the Synapse library in the Cortex blob store at the end of Cortex initialization. This was done in order to prepare for eventually enforcing required upgrade paths for data migrations or other features.
- 447 - Added the new form ``inet:web:postref`` XREF to track an ``inet:web:post`` which refers to another node.
- 447 - Added the new form ``inet:web:action`` GUID to to track an arbitrary action by an ``inet:web:acct`. The actions tracked by this are by defined by [Synapse] user convention.
- 447 - Added the new form ``inet:web:actref`` XREF to track how an ``inet:web:action`` may have interacted with another node.
- 454 - Added the ``inet:web:chprofile`` GUID to track previous values of a ``inet:web:acct`` node, representing changes to user accounts or profiles.
- 454 - Added the ``inet:web:post:repost`` property to track the concept of a ``inet:web:post`` being a copy of another post.
- 455 - Added a pair of Storm (and Cortex) Configable options to enable and set logging levels for Storm queries. These are ``storm:query:log:en`` and ``storm:query:log:level``. This logs what the query is and what the user execution context was.
- 426 - Axon and AxonHost objects are now Configable objects with configuration definitions that are used to define their behavior.

Enhancements
- 442 - Python 2.7 support dropped from Synapse.
- 447 - Migrated all inet:net* forms to the inet:web:* space. The following is a map of the migrated forms and their corresponding new forms:

('inet:netuser', 'inet:web:acct')
('inet:netgroup', 'inet:web:group')
('inet:netmemb', 'inet:web:memb')
('inet:follows', 'inet:web:follows')
('inet:netpost', 'inet:web:post')
('inet:netfile', 'inet:web:file')
('ps:hasnetuser', 'ps:haswebacct')
('ou:hasnetuser', 'ou:haswebacct')

These forms will automatically be migrated in existing Cortexes. If XREF types were used to point to any of these forms and the cortex was not first migrated to v0.0.26, the XREF type migration will fail. It is recommended that users first upgrade to v0.0.26 prior to upgrading to v0.0.27.
- 447 - Added Storage.updateProperty() and Storage.updatePropertyValu() APIs to the Cortex storage layer for doing bulk property and property-by-value updates. These are explicitly NOT exposed in the Cortex class.
- 449 - Thinned out some components of the EventBus class for performance reasons. This did result in the removal of the synapse.eventbus.on() decorator for decorating functions to be used as event callbacks.
- 456 - Removed unused Synapse modules: synapse.hivemind, synapse.mindmeld, synapse.lib.moddef.
- 426 - Logging in tests is now controlled by the environmental variable ``SYN_TEST_LOG_LEVEL`` which, as an integer, will set the logging level used by the root logger.
- 426 - The environmental variable ``SYN_TEST_SKIP_LONG`` can be set to a non-zero integer to skip potentially long running tests. This can shave up to a minute of test execution time.
- 426 - Axons now have the in-memory cache enabled on their Cortex by default.
- 458 - The ``inet:web:acct:occupation`` property has been changed from a ``str:txt`` type to ``str:lwr`` to allow for better foldability between user-declared occupations.

Bugs
- 443 - Make the daemon return more useful error messages when an exception has occurred during execution of a remote request.
- 444, 445 - Allow an inet:srv4 type to accept an integer string as input. Also adds additional boundary checking when norming an ip:port string to ensure that irreversible inputs are not accepted.
- 453 - Cleaned up skifIfNoInternet() test helpers. They will now be allowed to fail unless the ``SYN_TEST_SKIP_INTERNET`` environmental variable is set to a non-zero integer.
- 426 - The synapse.lib.heap.Heap class was not properly responding to ``heap:resize`` events. This was remedied.
- 426 - Wrapped a .items() iterator in synapse.daemon.OnHelp with a list to prevent a RuntimeError.
- 426 - Fix the synapse.lib.service.SvcProxy.getSynSvcs() method to return a Telepath safe list instead of a dict.values() view object.
- 426 - Fix the synapse.lib.service.SvcProxy.__init__ to strap in event handlers AFTER initializing instance variables to avoid a race condition on startup.
- 426 - AxonHost now waits before advertising itself on the bus, and properly calculates the number of axons it needs to make. This addressed an issue where the AxonHost was generating a non-deterministic number of Axons.
- 426 - The Axon’s thread to make clones for itself on a ServiceBus now waits until a remote clone is made. This addressed an issue where the Axon would make extra clones for itself.

Documentation
- 448 - Added in-model documentation for the file: model defined in files.py.
- 451 - Added user guide information for Storm lift operations, lift(), guid() and alltags().
- 426 - Docstrings in the synapse.axon module were rewritten or added when needed.

--------------------

New Features
- 438 - Added PropValu datatype to synapse. This allows a secondary (or primary) property to be modeled as a string in the form `property=<repr valu>`. This type also yields sub of "prop" representing the property, and "strval" or "intval" being the system normalized value of the property. This allow for node creation where a reference to another node property is needed but cannot be defined up front in the model. The additional subs allow for filtering/pivoting operations on nodes which use the PropValu type. A simple example of the string form for a PropValue is `inet:ipv4=1.2.3.4`. A more complex example of the string form for a PropValu is `inet:passwd=oh=my=graph!`
- 438 - XREF types have been updated to use the PropValu datatype instead of storing data in the property columns. This removes any extra-model data from the property columns in Cortex rows, and allows the implementation of pivotable XREF nodes. The string syntax for XREF creation was changed to be in line with the Comp datatype syntax, which looks now looks like ``. This will affect any ingests or programmatic creation of xref nodes done by users. An example of the string form of a Xref now looks like the following: `(98db59098e385f0bfdec8a6a0a6118b3,"inet:passwd=oh=my=graph!")`. Note that the PropValue portion of the value is quote delimited.

Enhancements
- 438 - The `refs()` storm operator is updated to be aware of secondary properties which are PropValu types and will pivot off of them, even if the props themselves are not forms.
- 438 - The file:imgof and file:txtref nodes no longer have glob property xref:*, instead :xref is now a PropValu and has the additional :xref:prop, :xref:intval and :xref:strval secondary props. This does require a data migration, so deployment of v0.0.26 should be tested on Cortexes which use those forms prior to production use.
- 438 - Migrated unittests to using getRamCore() helper when possible.
- 438 - Added unittest self.len() helper. Started some migration there.
- 439 - Cleaned up Socket() class implementation to be better, stronger and faster.
- 440 - Moved the syn:type, syn:form and syn:prop forms (and other items declared in DataModel) to be part of the "syn" model. These core elements are now themselves introspect-able.
- 440 - Added syn:prop:base and syn:prop:relname to syn:prop nodes, so those nodes can now be lifted by a basename or a relative property name.
- 441 - Added Python 3.7 RC1 to the test matrix. Pinned postgres test image to postgres:9.6.
- The Storm runtime now respects the `storm:limit:lift` configable value as the default limit when lifting nodes.

Bugs
- 440 - Changed how runtime nodes (runts) used to represent the data model in a cortex are created. These are now made from the type and property definitions based on the loaded data model; having been guided by the data model being processed. This restores things like "syn:prop:form" which was accidentally dropped from nodes.

--------------------

New Features
- 404 - Added model for inet:web:netlogon to track netuser's logging into services.
- 433 - Added inet:dns:mx, inet:dns:cname, inet:dns:soa, inet:dns:txt node types for recording different DNS responses.
- 436 - Added the ability to define triggers. Triggers are predefined actions which can react to events and trigger storm queries, in order to automate activities and actions.
- 436 - Rewrote the user authentication system to be more integrated with Cortex (and used within the trigger subsystem). This allows adding user roles and permissions to add, delete, or update nodes.

Enhancements
- 404 - Added inet:whois:nsrec comp nodes to track nameserver's associated with whois record. This removed the inet:whois:rec:ns* properties and migrates existing props into the new nodes.

Bugs
- 437 - Fix the tag interval filtering in the Storm query system.

Documentation
- 435 - Rewrote docstrings for synapse.lib.queue subsystem.

--------------------

Enhancements
- 430 - CortexTest class was split into CortexBaseTest and CortexTest. CortexBaseTest is used for running the
basic_core_expectations tests with the different storage backing; and CortextTest is used for more generic API tests.
This allows running the basic test suite against the different storage types without running the entire test suite, in
order to isolate possible storage related issues.
- 434 - Migrated from using nosetestes to using py.test as the Synapse testrunner for CI.

Bugs
- 422 - Fixed the order of autoadds being added to the DB during the addTufoEvents / addTufoEvent Cortex API. This
could manifest itself as tufos made by these events to not contain their properties.
- 428 - Fix a issue with the LMDB cortex where parameters passed to pylmdb were not bool types (as that library
expected), causing the parameters to be ignored in effect.

Documentation
- 422 - The inet:dns model docstrings were updated to be more comprehensive.

--------------------

New Features
- 423 - Added ephemeral runtime-only nodes as a concept to Synapse. The data model has been migrated to utilize these for loading and storing the data model, so the data model (syn:type, syn:prop, syn:form) itself no longer lives within the Cortex database itself. This makes data model updates much easier, requiring only data migration functions to be written for future updates. This does mean that custom models are no longer persistent and are required to be loaded into a Cortex to be made available.

- 423 - Added a initial model for capturing software execution knowledge on hosts. These models are primarily comp types, which allow modeling varying levels of knowledge which may be available from different data sources.

Enhancements
- 423 - The setModlVers API was moved to the Storage layer (and simply called through by the Cortex) to allow hooking model revision function execution with events, to allow for easier testing of Model data changes requiring data migrations.
- 429 - Updated .drone.yml to run all tests in parallel again to take advantage of infrastructure updates.

Bugs
- 421 - Fixed getConfDefs() API in Configable. Made it so that default values (defval) items are copy.deepcopy'd, so that mutable defval's are not overwritten by later use.
- 425 - Prevent the Ingest tool from attempting to ingest data to a remote Cortex connected to over Telpath. Attempting to do so previously would result in a esoteric error message. This means that the --sync option must be used when syncing data up to a remote Cortex.

Documentation
- 424 - Docstrings for the synaspe.lib.config.Config class were written to clarify that class APIs and add notes about possible race conditions when using onSetConfOpt handlers to respond to configable events.