Msticpy

This release includes:

- Expansion of Azure Data API for retrieving additional data about subscriptions and resources from Azure APIs.
- Time Series anomaly detection for arbitrary Kusto data sets together with visualization of time series charts in
Jupyter Notebooks using Bokeh Charts.
- Using KeyVault and Python Keyring to store secrets used to authenticate to web data providers.
Examples include API keys for Threat Intel and Geo IP Providers. Other provider types will be included in
a future release.

New Features
- Azure data expansion and documentation
- Keyvault and keyring secrets management with support for multiple Azure clouds
- config2kv.py KV secret update tool
- Timeseries - Bokeh with KQL and documentation
- KQL generic time series decomposition queries
- Bokeh time series visualization
- Added pandas version of get_whois_info and added as DataFrame accessor function.
- Added cmd script to test PyPi test deployment
- Added Conda package requirements files
- Updated TI providers to provide more consistent output and reduce false positives
- Using text rather than number to express severity
- Made TISeverity class comparable and parsable from string or int
- Added mp_demo_data.py notebook helper to tools.
- SecurityAlert has more flexible recognition of entities
- Added additional dependencies for azure mgmt, keyvault and others.

Fixes
- Fixed get_ip_type ordering to return more accurate IP types
- Fix entity extraction in SecurityAlert to allow nested entities to work correctly
- Additional test cases

This release includes early implementations of **pandas extensions** so that you can invoke msticpy functionality directly from a DataFrame:
python
my_events_df.mp_timeline.plot()
my_proc_events_df.mp_process_tree.plot()

So far, [IoCExtract](https://msticpy.readthedocs.io/en/latest/data_analysis/IoCExtract.html#pandas-extension), [Base64Unpack](https://msticpy.readthedocs.io/en/latest/data_analysis/Base64Unpack.html#pandas-extension), Timeline and ProcessTree have these extensions but we will be adding them to more msticpy modules over time.

Also, IoCExtract and Base64 decode functions have **IPython magics** (`%%ioc` and `%%b64`) allowing you to paste a block of text into a notebook cell and run the function directly on that text.

Most other changes are primarily maintenance and house-keeping improvements such as increasing unit testing code coverage.

New Features
- **process_tree** - added pandas extension and changed main function so that it returns the plot figure and layout
- **timeline** - added pandas extension. added support for DateTime column in Tooltips (display as date time rather
than number)
- **base64unpack** - added pandas extension, added IPython `%%b64` magic
- **iocextract** - added pandas extension, added IPython `%%ioc` magic
- Added documentation and notebook examples for the pandas extensions and magics.
- **wsconfig** - added method to display available workspaces
- **README.MD** - added some graphics to brighten the page up a little
- Added unit test test_folium.py
- Adding **FoliumMap.ipynb** sample notebook
- Added additional geolocation centering functions for **FoliumMap**
- Updates to **GeoIPLookups.ipynb**
- Add parameter checks to timeline.py and process_tree.py so that invalid **kwargs produced a helpful error message.
- Added **requirements-dev.txt**

Fixes
- Typos in AzureData.rst
- Adding GeoIP tests.
- Removing deprecated lines from coverage reports.
- Cleaned up pytest coverage report.
- Adding suppression file for credscan false positives
- Removing SecurityAlertandEntities notebook with misleading content
- Removed failing cell from end of GeoIPLookups notebook
- Fixed a few errors in foliummap.py
- Fixed bug in GeoIP DB downloader
- Changed foliummap center functions to use median by default
- Removed largely redundant os_family param from iocextract.py functions
- Fixed sectools_magics iocextract class
- Update test_ioc_extractor for new parameters
- domain_tools - changed tld_index and ssl_bl attributes to properties that auto-load on first use (prevents remote
http request if data on class instantiation)
- Added more tests for utility.py
- Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
- Tidied up/refactored some code in base64unpack.py

New Features
The documentation now includes a user guide covering many aspects of `msticpy`
It includes the following sections:
- Getting started section (Installation and configuration)
- Data Acquisition (querying and data)
- Data Enrichment (GeoIP, Threat Intel)
- Data Analysis (IoC extraction, decoding, clustering
- Visualization (Event timeline, Process tree, Mapping, widgets)

Documentation is on [ReadTheDocs](https://msticpy.readthedocs.io/en/latest/index.html)

Fixes
- Broken links and outdated docs updated
- Fixes to some unit tests

Overview
Note: This release is consolidation of v0.2.8 and some additional features and fixes. Adding everything to the release notes for this version since the previous one did not get published to PyPi.

This release contains three important features:
- Query support for Microsoft Defender ATP
- Interactive Process Tree browser.
- Support for querying Azure properties for subscriptions and resources

New Features
- Microsoft Defender Query Support. Added a query provider/driver to
query Defender alerts, machines, processes and arbitrary KQL queries of
the Hunting data
- Template queries for MDATP for hunting and standard entities
- Process Tree Viewer - Bokeh interactive graphical view for one or
more Process Trees in a data set. Supports both Windows and Linux.
- Process tree utilities - data library to create and query process trees.
- Azure properties of subscriptions and resources such as VMs can be
queried from Notebooks.
- Query providers now accept ISO-string format for datetime fields
for queries (in addition to datetime and timedelta)
- Added Progress widget to nbwidgets.
- Added config support for GeoIP providers from msticpyconfig.yaml
- GeoIP classes try to obtain API key from config if not supplied
- Refactored ti_provider_settings to generic provider_settings module

Fixes
- Miscellaneous linting/checker bugs
- Spelling and path errors in docs
- Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
- Updating dependency to Bokeh 1.40
- Fixed timeline legend bug
- Fix for Maxmind requiring authentication for GeoLite DB download (GeoIP classes
are no longer loaded by default.
- Added missing pytz and pyyaml packages to requirements.txt and setup.py.
- GeoLite2 url and archive extraction changes

Overview
This release contains three important features:
- Query support for Microsoft Defender ATP
- Interactive Process Tree browser.
- Support for querying Azure properties for subscriptions and resources

New Features
- Microsoft Defender Query Support. Added a query provider/driver to
query Defender alerts, machines, processes and arbitrary KQL queries of
the Hunting data
- Template queries for MDATP for hunting and standard entities
- Process Tree Viewer - Bokeh interactive graphical view for one or
more Process Trees in a data set. Supports both Windows and Linux.
- Process tree utilities - data library to create and query process trees.
- Azure properties of subscriptions and resources such as VMs can be
queried from Notebooks.
- Query providers now accept ISO-string format for datatime fields
for queries (in addition to datetime and timedelta)
- Added Progress widget to nbwidgets.

Fixes
- Miscellaneous linting/checker bugs
- Spelling and path errors in docs
- Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
- Updating dependency to Bokeh 1.40
- Fixed timeline legend bug

Overview
Most of the features and fixes this month were done to support the new Entity Explorer series of notebooks
published on [Azure-Sentinel-Notebooks repo](https://github.com/Azure/Azure-Sentinel-Notebooks)
New Features
- Added syslog_utils module for common syslog data manipulation
- Added cmd_line module for syslog CMD analysis
- Added ip_utils module for common IP Address operations
- Added domain_utils module for common domain/DNS/URL operations
- Added new TI providers - open_page_rank and tor_exit_nodes
- Added package auto-install function added to utility.py
- nbwidgets: added filtering to all select/list widgets
- In query templates you can now express date parameters (and query defaults) as
a pos/neg number relative to current date or a KQL-like range (-30d, 15h, etc.)
- wsconfig.py - can create a Connection string directly from wsconfig.
- Added settings as a top-level msticpy attribute.
- TI dataframes derive their column names directly from LookupResult so don't need to keep two lists.

- A number of new queries were added in the following categories:
- Syslog/linux
- Office Activity
- Azure/Azure Active Directory
- Azure Network Analytics
- KQL time series queries
- Hunting bookmarks
- Documentation updates:
- new documentation page on msticpy configuration - msticpyconfig.yaml
- Updated Readme.md for new features.

Fixes
- Typo in wsconfig.py - wsconfig throws meaningful error if config values are not found
- fix yaml parsing error in timeseries kql
- Miscellaneous fixes from notebook testing for new Explorer notebooks
- Changed param_extractor to always prefer supplied params over defaults
- tilookup fix - exception thrown if an empty IoCs list sent to it
- geoip - fixed multiple problems with the DF lookup version of the API
- Updated Pandas requirement to 0.25
- Query fixes - miscellaneous
- Added dependencies on cryptography, tqdm, ipwhois