Msticpy

New Features
- db86480:
- LocalDataDriver for using CSV and pickled DF files as a QueryProvider (64)
This is primary for demonstration and test purposes where you do not have access to online data sources. It
replicates the functionality of QueryProvider allowing drop-in replacement in existing notebooks.
- Updated DataQueries.rst ReadtheDocs page with new queries
- Add documentation for LocalDataDriver to DataProviders.rst and updated section on creating query files.
- 66a66d2:
- Checked in notebook to create DataQueries.rst
- Added "AzureSentinel" alias for LogAnalytics DataEnvironment

Breaking Changes
- db86480:
- Removed deprecated kql.py, query_builtin_queries, query_mgr.py, query_schema.py
- Changed location of query_defns.py and made pkg reference updates in several modules and notebooks.
- Some fixes to support local_data_driver in query_store.py, driver_base.py and data_providers.py
- Unit test - test_localdata_queries.yaml and supporting data and query files.
- Fixed test in test_utils.py to work on Linux
- Reduced warnings produced during pytest run to something more reasonable (mainly by removing
deprecated code
- 8a32ad5:
- Changed tilookup and kql_base/kql_driver so that handling failure to load is a bit friendlier. E.g. running
TILookup in a non-IPython environment (with ASTI provider) will now just cause a warning, not an exception.
- kql_driver.py also updated to check for get_ipython() returning None and output friendlier message.
- Changed driver_base.py and derived class to take additional QuerySource parameter for query() method -
not yet used but required so that we can implement driver-specific checks on query parameters.

This release includes:

Anomaly sequence analysis and visualization using Markov chain karishma-dixit
Morph Chart visualization of log events petebryan

(originally released as v0.4.1 but updated to v0.5.0)

New Features

- Anomalous sequences (60)
Markov Chain anomaly analysis for sequences of commands/patterns in a session
- Morph Charts visualization - 3D visualization of event data using experimental (58)
Morph Charts exploration
- nbinit: a neater and more robust startup/setup function for Jupyter notebooks
handling package installs, imports and option setting (62)
- Azure Sentinel Queries
- Added two Logon fail queries for Linux (62)
- Add Linux logons for host
- Added msticpy.common.pkg_config.validate_config() to validate current config
or external config file (62)

Fixes

- f78a29e:
- Change return type on for bokeh graphs to return whole layout
- Improved geoip error messages when Api key is missing
- Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
- 31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
- 827477b: make titles consistent on the widgets page (59)
- 7964b5f: Fix to utility.py - check_and_install_missing_packages to all package version to be specified.
- f793d55:
- Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
- Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
- Fixed an error in test-pypi-test-pkg.cmd
- 3e42e42: Doc fix and OutOfBoundsDatetime catch
- efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting (55)
- 0ad166a: fixing headings in rst docs for timeseries
- 606fc8f: Fixing broken Readthedocs link (53)
- 4810e1f: Fixing some documentation omissions/errors (52)
- 43bbd3c: Updating pylintrc to change limits for some checks.
- f50eec2: Notebooklet queries and timeline hide option
- 13c3f3f Flake8 error with unknown "QuerySource" (63)
- 9921352 Adding pkgs to conda-reqs-pip.txt Removing Python 3.7 version setting from pre-commit
- 921370c (63)
- requirements.txt and setup.py changes to avoid version conflicts (causing sphinx to fail)updated version to 5.0
- c900386 Fixed issue causing test failure (63)
- 5c9db2d Adding get_all_entities feature used in Alerts Notebook (63)

This release includes:

Anomaly sequence analysis and visualization using Markov chain karishma-dixit
Morph Chart visualization of log events petebryan

New Features

- Anomalous sequences
Markov Chain anomaly analysis for sequences of commands/patterns in a session
- Morph Charts visualization - 3D visualization of event data using experimental
Morph Charts exploration
- nbinit: a neater and more robust startup/setup function for Jupyter notebooks
handling package installs, imports and option setting
- Azure Sentinel Queries
- Added two Logon fail queries for linux
- Add Linux logons for host
- Added msticpy.common.pkg_config.validate_config() to validate current config
or external config file

Fixes

- f78a29e:
- Change return type on for bokeh graphs to return whole layout
- Improved geoip error messages when Api key is missing
- Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
- 31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
- 827477b: make titles consistent on the widgets page (59)
- 7964b5f: Fix to utility.py - check_and_install_missing_packages to all package version to be specified.
- f793d55:
- Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
- Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
- Fixed an error in test-pypi-test-pkg.cmd
- 3e42e42: Doc fix and OutOfBoundsDatetime catch
- efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting
- 0ad166a: fixing headings in rst docs for timeseries
- 606fc8f: Fixing broken Readthedocs link (53)
- 4810e1f: Fixing some documentation omissions/errors (52)
- 43bbd3c: Updating pylintrc to change limits for some checks.
- f50eec2: Notebooklet queries and timeline hide option

This release includes:

- Expansion of Azure Data API for retrieving additional data about subscriptions and resources from Azure APIs.
- Time Series anomaly detection for arbitrary Kusto data sets together with visualization of time series charts in
Jupyter Notebooks using Bokeh Charts.
- Using KeyVault and Python Keyring to store secrets used to authenticate to web data providers.
Examples include API keys for Threat Intel and Geo IP Providers. Other provider types will be included in
a future release.

New Features
- Azure data expansion and documentation
- Keyvault and keyring secrets management with support for multiple Azure clouds
- config2kv.py KV secret update tool
- Timeseries - Bokeh with KQL and documentation
- KQL generic time series decomposition queries
- Bokeh time series visualization
- Added pandas version of get_whois_info and added as DataFrame accessor function.
- Added cmd script to test PyPi test deployment
- Added Conda package requirements files
- Updated TI providers to provide more consistent output and reduce false positives
- Using text rather than number to express severity
- Made TISeverity class comparable and parsable from string or int
- Added mp_demo_data.py notebook helper to tools.
- SecurityAlert has more flexible recognition of entities
- Added additional dependencies for azure mgmt, keyvault and others.

Fixes
- Fixed get_ip_type ordering to return more accurate IP types
- Fix entity extraction in SecurityAlert to allow nested entities to work correctly
- Additional test cases

This release includes early implementations of **pandas extensions** so that you can invoke msticpy functionality directly from a DataFrame:
python
my_events_df.mp_timeline.plot()
my_proc_events_df.mp_process_tree.plot()

So far, [IoCExtract](https://msticpy.readthedocs.io/en/latest/data_analysis/IoCExtract.html#pandas-extension), [Base64Unpack](https://msticpy.readthedocs.io/en/latest/data_analysis/Base64Unpack.html#pandas-extension), Timeline and ProcessTree have these extensions but we will be adding them to more msticpy modules over time.

Also, IoCExtract and Base64 decode functions have **IPython magics** (`%%ioc` and `%%b64`) allowing you to paste a block of text into a notebook cell and run the function directly on that text.

Most other changes are primarily maintenance and house-keeping improvements such as increasing unit testing code coverage.

New Features
- **process_tree** - added pandas extension and changed main function so that it returns the plot figure and layout
- **timeline** - added pandas extension. added support for DateTime column in Tooltips (display as date time rather
than number)
- **base64unpack** - added pandas extension, added IPython `%%b64` magic
- **iocextract** - added pandas extension, added IPython `%%ioc` magic
- Added documentation and notebook examples for the pandas extensions and magics.
- **wsconfig** - added method to display available workspaces
- **README.MD** - added some graphics to brighten the page up a little
- Added unit test test_folium.py
- Adding **FoliumMap.ipynb** sample notebook
- Added additional geolocation centering functions for **FoliumMap**
- Updates to **GeoIPLookups.ipynb**
- Add parameter checks to timeline.py and process_tree.py so that invalid **kwargs produced a helpful error message.
- Added **requirements-dev.txt**

Fixes
- Typos in AzureData.rst
- Adding GeoIP tests.
- Removing deprecated lines from coverage reports.
- Cleaned up pytest coverage report.
- Adding suppression file for credscan false positives
- Removing SecurityAlertandEntities notebook with misleading content
- Removed failing cell from end of GeoIPLookups notebook
- Fixed a few errors in foliummap.py
- Fixed bug in GeoIP DB downloader
- Changed foliummap center functions to use median by default
- Removed largely redundant os_family param from iocextract.py functions
- Fixed sectools_magics iocextract class
- Update test_ioc_extractor for new parameters
- domain_tools - changed tld_index and ssl_bl attributes to properties that auto-load on first use (prevents remote
http request if data on class instantiation)
- Added more tests for utility.py
- Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
- Tidied up/refactored some code in base64unpack.py

New Features
The documentation now includes a user guide covering many aspects of `msticpy`
It includes the following sections:
- Getting started section (Installation and configuration)
- Data Acquisition (querying and data)
- Data Enrichment (GeoIP, Threat Intel)
- Data Analysis (IoC extraction, decoding, clustering
- Visualization (Event timeline, Process tree, Mapping, widgets)

Documentation is on [ReadTheDocs](https://msticpy.readthedocs.io/en/latest/index.html)

Fixes
- Broken links and outdated docs updated
- Fixes to some unit tests