Synapse

--------------------

New Features
- 404 - Added model for inet:web:netlogon to track netuser's logging into services.
- 433 - Added inet:dns:mx, inet:dns:cname, inet:dns:soa, inet:dns:txt node types for recording different DNS responses.
- 436 - Added the ability to define triggers. Triggers are predefined actions which can react to events and trigger storm queries, in order to automate activities and actions.
- 436 - Rewrote the user authentication system to be more integrated with Cortex (and used within the trigger subsystem). This allows adding user roles and permissions to add, delete, or update nodes.

Enhancements
- 404 - Added inet:whois:nsrec comp nodes to track nameserver's associated with whois record. This removed the inet:whois:rec:ns* properties and migrates existing props into the new nodes.

Bugs
- 437 - Fix the tag interval filtering in the Storm query system.

Documentation
- 435 - Rewrote docstrings for synapse.lib.queue subsystem.

--------------------

Enhancements
- 430 - CortexTest class was split into CortexBaseTest and CortexTest. CortexBaseTest is used for running the
basic_core_expectations tests with the different storage backing; and CortextTest is used for more generic API tests.
This allows running the basic test suite against the different storage types without running the entire test suite, in
order to isolate possible storage related issues.
- 434 - Migrated from using nosetestes to using py.test as the Synapse testrunner for CI.

Bugs
- 422 - Fixed the order of autoadds being added to the DB during the addTufoEvents / addTufoEvent Cortex API. This
could manifest itself as tufos made by these events to not contain their properties.
- 428 - Fix a issue with the LMDB cortex where parameters passed to pylmdb were not bool types (as that library
expected), causing the parameters to be ignored in effect.

Documentation
- 422 - The inet:dns model docstrings were updated to be more comprehensive.

--------------------

New Features
- 423 - Added ephemeral runtime-only nodes as a concept to Synapse. The data model has been migrated to utilize these for loading and storing the data model, so the data model (syn:type, syn:prop, syn:form) itself no longer lives within the Cortex database itself. This makes data model updates much easier, requiring only data migration functions to be written for future updates. This does mean that custom models are no longer persistent and are required to be loaded into a Cortex to be made available.

- 423 - Added a initial model for capturing software execution knowledge on hosts. These models are primarily comp types, which allow modeling varying levels of knowledge which may be available from different data sources.

Enhancements
- 423 - The setModlVers API was moved to the Storage layer (and simply called through by the Cortex) to allow hooking model revision function execution with events, to allow for easier testing of Model data changes requiring data migrations.
- 429 - Updated .drone.yml to run all tests in parallel again to take advantage of infrastructure updates.

Bugs
- 421 - Fixed getConfDefs() API in Configable. Made it so that default values (defval) items are copy.deepcopy'd, so that mutable defval's are not overwritten by later use.
- 425 - Prevent the Ingest tool from attempting to ingest data to a remote Cortex connected to over Telpath. Attempting to do so previously would result in a esoteric error message. This means that the --sync option must be used when syncing data up to a remote Cortex.

Documentation
- 424 - Docstrings for the synaspe.lib.config.Config class were written to clarify that class APIs and add notes about possible race conditions when using onSetConfOpt handlers to respond to configable events.

--------------------

New Features
- 395 - Add formTufosByProps() API to the Cortex() class to do bulk tufo creation. This has significant performance for doing bulk tufo creation by utilizing a single storage layer transaction.
- 359, 405 - All splices now represent atomic actions. This is a breaking change from v0.0.21, since the node:set events no longer multiple props, but instead a single prop per event. This requires an ecosystem wide upgrade for users utilizing the splice subsystem.
- 408 - Provisional task() operator added to the storm runtime. This fires events in the form of "task:<name>" on the storm core eventbus. This behavior may change in the future.
- 409, 414, 416 - Synapse model properties which had req=1 set on them now actually require that property to be present when model enforcement is in place. This check occurs after node:form events, allowing code to hook the node formation process and provide required properties or set them as needed.
- 417 - Added a tree() operator to storm for doing recursive pivot operations on a set of nodes.
- 419 - Added a delprop() operator to storm to remove properties from nodes.

Enhancements
- 410 - pivot() operator syntax updated to match that of the macro syntax operator. This may be a breaking change for any programmatic use of the pivot operator syntax.
- 411 - Added pycodestyle checks to CI builds to identify code style issues.
- 412 - Fix resourcewarnings with unclosed file handles.

Bugs
- 399 - Fix Cortex.__init__() on Python 2.7 where a list comprehension smashed function locals. This prevented configable options from being set at initialization.
- 401 - Fix setTufoProps() to add nodes created from secondary properties when appropriate.

Documentation
- 403 - Initial docs for Ingest subsystem,
- 408, 420 - Updates to Storm runtime documentation.
- 418 - Added markdown template to release documentation.

--------------------

The 0.0.20 release is not small - so here are some notes for it!

Since there are significant changes to how we handle models and storage layers in this version, we recommend that you make a backup of any production Cortexes you have before deploying this, and test your systems prior to deployment.

-- vEpiphyte

New Features

- 277 - Add the ability to load python modules via dmon.
- 275 - All Synapse core models have been cutover to using CoreModule implementations for doing model revisioning.
- 274 - Optional Time boundaries added to tags for doing timeboxing of tags.
- 279 - Added columns support and additional help framework for cmdr CLI
- 285 - Easycert tool now makes server certificates with the subjectAltName (SAN) field populated. Tell your fiends - this a way easier tool to use to make SAN certs than using the openssl binary itself. This allows certificate pairs to be used in chrome 58+.
- 177 - Added a ndef() function to synapse.lib.tufo to get a tufo type and primary prop.
- 286 - Easycert tool can make PKCS12 client certs.
- 333 - Comp types now support optional kw fields, allowing recording of varying levels of knowledge for a given type.
- 321 - SSL Support added to the webapp
- 289, 290, 301 - Add support for a blob key/value store to the Cortex which exists separately from the Row layer storage.
- 291, 292, 301 - Add support for storage layers to be revisioned independently of models.
- 300, 342 - Add support for sending BODY content via remcycle
- 348, 350 - Add a guid() operator to storm for lifting a node by iden.
- 358 - Add a delTufoProp() API to cortex for deleting a tufo property. This changed splice contention and event handlers, node:set is no longer fired; node:prop:set and node:prop:del are now fired.
- 374, 378 - Add a new limit() operator for Storm.
- 320, 362 - Storage layers are now separated from the Cortex class by an API boundary. This allows for future custom storage layers to be implemented easily.
- 319, 262 - Add dumprows and loadrows tools to dump a cortex to a savefile or create a new storage object from a existing savefile.

Improvements
- 282 - Code style cleanup
- 293 - Prevent reference databases used in tests from being modified accidently.
- 288, 296, 332 - Storm setprop operator (and prop edit mode) now respect relative prop values. Addnode also uses relative props now.
- 272, 342 - Rewrote remcycle tests to no longer require external resources. They run consistently now.
- 302, 342 - Remcycle now uses configable definitions in a consistent manner.
- 364 - Restored pre v0.0.15 axon path behaviors.
- 378 - Add a plan step to storm query parsing - allows for future optimizations.
- 338, 362 - Configable objects may now use a decorator to declare all of their options, which will be loaded at runtime.

Bugfixes
- 276 - Fixes for eq/lt/le/gt/ge operators in storm
- 236, 295 - Allow cache disable on a cortex to actually clear the cache.
- 287, 294 - Fix delnode storm operator and delTufo() API
- 347, 349 - File:bytes nodes created from seed ctors (file:bytes:md5, file:bytes:sha1, etc) now have stable guids. Previously guids were case sensitive.
- 335, 352 - Fix cmdr quit function on Python 2.7
- 345, 351 - Fix inet:cidr range lookups.
- 367, 370 - Regex macro filter "~=" accidently ate whitespace. This is fixed.
- 380, 382 - Fix a issue with tag caches upon tag deletion.
- 378 - Fix for comp type arg parsing in long form storm operators.
- 339, 346 - Fix for inet:url:ipv4 and inet:url:fqdn parsing.
- 354, 355 - Fix for inet:tcp4 / inet:udp4 :ipv4 and :port parsing. Also make ipv6 repr's consistent.

Documentation
- 273, 278 - Initial performance benchmarks for Synapse
- 281, 283 - Initial User Guide for Synapse
- 284 - Change docs to using the easier to read RTD theme.
- 368 - ADditional User Guide documentation.
- 338, 362 - Automatic documentation is now generator for configable objects, detailing their options.
- 323, 324, 362 - Synapse devops documentation moved together.