Msticpy

This release includes early implementations of **pandas extensions** so that you can invoke msticpy functionality directly from a DataFrame:
python
my_events_df.mp_timeline.plot()
my_proc_events_df.mp_process_tree.plot()

So far, [IoCExtract](https://msticpy.readthedocs.io/en/latest/data_analysis/IoCExtract.html#pandas-extension), [Base64Unpack](https://msticpy.readthedocs.io/en/latest/data_analysis/Base64Unpack.html#pandas-extension), Timeline and ProcessTree have these extensions but we will be adding them to more msticpy modules over time.

Also, IoCExtract and Base64 decode functions have **IPython magics** (`%%ioc` and `%%b64`) allowing you to paste a block of text into a notebook cell and run the function directly on that text.

Most other changes are primarily maintenance and house-keeping improvements such as increasing unit testing code coverage.

New Features
- **process_tree** - added pandas extension and changed main function so that it returns the plot figure and layout
- **timeline** - added pandas extension. added support for DateTime column in Tooltips (display as date time rather
than number)
- **base64unpack** - added pandas extension, added IPython `%%b64` magic
- **iocextract** - added pandas extension, added IPython `%%ioc` magic
- Added documentation and notebook examples for the pandas extensions and magics.
- **wsconfig** - added method to display available workspaces
- **README.MD** - added some graphics to brighten the page up a little
- Added unit test test_folium.py
- Adding **FoliumMap.ipynb** sample notebook
- Added additional geolocation centering functions for **FoliumMap**
- Updates to **GeoIPLookups.ipynb**
- Add parameter checks to timeline.py and process_tree.py so that invalid **kwargs produced a helpful error message.
- Added **requirements-dev.txt**

Fixes
- Typos in AzureData.rst
- Adding GeoIP tests.
- Removing deprecated lines from coverage reports.
- Cleaned up pytest coverage report.
- Adding suppression file for credscan false positives
- Removing SecurityAlertandEntities notebook with misleading content
- Removed failing cell from end of GeoIPLookups notebook
- Fixed a few errors in foliummap.py
- Fixed bug in GeoIP DB downloader
- Changed foliummap center functions to use median by default
- Removed largely redundant os_family param from iocextract.py functions
- Fixed sectools_magics iocextract class
- Update test_ioc_extractor for new parameters
- domain_tools - changed tld_index and ssl_bl attributes to properties that auto-load on first use (prevents remote
http request if data on class instantiation)
- Added more tests for utility.py
- Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
- Tidied up/refactored some code in base64unpack.py

New Features
The documentation now includes a user guide covering many aspects of `msticpy`
It includes the following sections:
- Getting started section (Installation and configuration)
- Data Acquisition (querying and data)
- Data Enrichment (GeoIP, Threat Intel)
- Data Analysis (IoC extraction, decoding, clustering
- Visualization (Event timeline, Process tree, Mapping, widgets)

Documentation is on [ReadTheDocs](https://msticpy.readthedocs.io/en/latest/index.html)

Fixes
- Broken links and outdated docs updated
- Fixes to some unit tests

Overview
Note: This release is consolidation of v0.2.8 and some additional features and fixes. Adding everything to the release notes for this version since the previous one did not get published to PyPi.

This release contains three important features:
- Query support for Microsoft Defender ATP
- Interactive Process Tree browser.
- Support for querying Azure properties for subscriptions and resources

New Features
- Microsoft Defender Query Support. Added a query provider/driver to
query Defender alerts, machines, processes and arbitrary KQL queries of
the Hunting data
- Template queries for MDATP for hunting and standard entities
- Process Tree Viewer - Bokeh interactive graphical view for one or
more Process Trees in a data set. Supports both Windows and Linux.
- Process tree utilities - data library to create and query process trees.
- Azure properties of subscriptions and resources such as VMs can be
queried from Notebooks.
- Query providers now accept ISO-string format for datetime fields
for queries (in addition to datetime and timedelta)
- Added Progress widget to nbwidgets.
- Added config support for GeoIP providers from msticpyconfig.yaml
- GeoIP classes try to obtain API key from config if not supplied
- Refactored ti_provider_settings to generic provider_settings module

Fixes
- Miscellaneous linting/checker bugs
- Spelling and path errors in docs
- Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
- Updating dependency to Bokeh 1.40
- Fixed timeline legend bug
- Fix for Maxmind requiring authentication for GeoLite DB download (GeoIP classes
are no longer loaded by default.
- Added missing pytz and pyyaml packages to requirements.txt and setup.py.
- GeoLite2 url and archive extraction changes

Overview
This release contains three important features:
- Query support for Microsoft Defender ATP
- Interactive Process Tree browser.
- Support for querying Azure properties for subscriptions and resources

New Features
- Microsoft Defender Query Support. Added a query provider/driver to
query Defender alerts, machines, processes and arbitrary KQL queries of
the Hunting data
- Template queries for MDATP for hunting and standard entities
- Process Tree Viewer - Bokeh interactive graphical view for one or
more Process Trees in a data set. Supports both Windows and Linux.
- Process tree utilities - data library to create and query process trees.
- Azure properties of subscriptions and resources such as VMs can be
queried from Notebooks.
- Query providers now accept ISO-string format for datatime fields
for queries (in addition to datetime and timedelta)
- Added Progress widget to nbwidgets.

Fixes
- Miscellaneous linting/checker bugs
- Spelling and path errors in docs
- Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
- Updating dependency to Bokeh 1.40
- Fixed timeline legend bug

Overview
Most of the features and fixes this month were done to support the new Entity Explorer series of notebooks
published on [Azure-Sentinel-Notebooks repo](https://github.com/Azure/Azure-Sentinel-Notebooks)
New Features
- Added syslog_utils module for common syslog data manipulation
- Added cmd_line module for syslog CMD analysis
- Added ip_utils module for common IP Address operations
- Added domain_utils module for common domain/DNS/URL operations
- Added new TI providers - open_page_rank and tor_exit_nodes
- Added package auto-install function added to utility.py
- nbwidgets: added filtering to all select/list widgets
- In query templates you can now express date parameters (and query defaults) as
a pos/neg number relative to current date or a KQL-like range (-30d, 15h, etc.)
- wsconfig.py - can create a Connection string directly from wsconfig.
- Added settings as a top-level msticpy attribute.
- TI dataframes derive their column names directly from LookupResult so don't need to keep two lists.

- A number of new queries were added in the following categories:
- Syslog/linux
- Office Activity
- Azure/Azure Active Directory
- Azure Network Analytics
- KQL time series queries
- Hunting bookmarks
- Documentation updates:
- new documentation page on msticpy configuration - msticpyconfig.yaml
- Updated Readme.md for new features.

Fixes
- Typo in wsconfig.py - wsconfig throws meaningful error if config values are not found
- fix yaml parsing error in timeseries kql
- Miscellaneous fixes from notebook testing for new Explorer notebooks
- Changed param_extractor to always prefer supplied params over defaults
- tilookup fix - exception thrown if an empty IoCs list sent to it
- geoip - fixed multiple problems with the DF lookup version of the API
- Updated Pandas requirement to 0.25
- Query fixes - miscellaneous
- Added dependencies on cryptography, tqdm, ipwhois

New Features
- Major update to display_timeline control:
- allows arbitrary number of event series
- input as dict of data series or grouped DataFrame
- added interactive visual time range control
- added display_timeline_values to display timelines with a scalar value (line, circle, vbar)
- added sample notebook
- added ReadtheDocs page describing usage
- Moved to timeline.py module

Other Improvements/Additions
- Updated Base64Unpack, EventClustering, NotebookWidgets and TIProviders notebooks.
- Added unit test capability for UI-dependent packages by running notebooks within the unit test
- nbwidgets: added filtering text box to all select widgets
- nbwidgets: added SelectSubset widget allowing you to pick from one list and add to selected subset
- Updates to documentation/README.md
- Added checks for no TI Providers or missing keys and updated TIProvider docs for this.
- Added network data query yaml - kql_sent_winevent.yaml
- Added WinSecurityEvent.json events file
- Added pre-commit hooks including local hook script download_tlds.py

Fixes
- Fixes from testing notebook development:
- Minor change to base64unpack.py to prevent pandas warning
- entityschema: fixing __repr__ to always return a string
- security_base: removing broken and deprecated properties adding __repr__
- ti_lookup - remove unneeded import
- nbwidgets - bug in restoring current index in selected items list
- eventcluster - first/last time range for clustered events was not properly calculated.
- Fixed foliummap error to display in notebook (implemented _repr_html_ so that instances display directly in notebook.)