Highlights
The highlights of this release (which is really 1.2.0 with some last-minute fixes) are: two new data providers for Azure Resource Graph and Sumologic and a DataViewer control for pandas dataframes.
Azure Resource Graph provider
The [Azure Resource graph](https://azure.microsoft.com/en-us/features/resource-graph/)
provider lets you query Azure resources using KQL queries. This works much like our other
query providers and gives you a lot more flexibility in navigating around Azure resources
than the current AzureData provider.
Explore more in the [Resource Graph Driver notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/ResourceGraphDriver.ipynb) and the [online docs](https://msticpy.readthedocs.io/en/latest/data_acquisition/ResourceGraphDriver.html)
Many thanks to Ryan Cobb rcobb-scwx for creating and contributing this.
Sumologic provider
[Sumo Logic Inc. ](https://www.sumologic.com), is a cloud-based machine data analytics service
focusing on security, operations and BI use cases. This provider allows you to connect to
and query your data from MSTICPy via their [Search API](https://help.sumologic.com/APIs/Search-Job-API/About-the-Search-Job-API)
Explore more in the [Sumologic Data Connector notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Sumologic-DataConnector.ipynb)
Many thanks to Julien (juju4) for building and contributing this.
DataViewer
The data viewer uses the [Bokeh DataTable control[(https://docs.bokeh.org/en/latest/docs/user_guide/interaction/widgets.html#datatable)
to display and browse through data in a pandas DataFrame. It lets you sort by column, choose which columns to display and filter by multiple columns. It keeps a synchronized copy of the DataFrame with column choice and filtering applied, so that you can always access the data as it appears in the control.
Explore more in the [DataViewer notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/DataViewer.ipynb)
and the [Dataview online docs](https://msticpy.readthedocs.io/en/latest/visualization/DataViewer.html)
New Features and Updates
ab6eb73 - (164) DataViewer control
- DataViewer control - Bokeh based data table control
- Documentation in DataViewer.rst
- Notebook in DataViewer.ipynb
3ab31ad- Azure Resource Graph Data Provider (167)
- Co-authored-by: Ryan Cobb <rcobbsecureworks.com>
- Co-authored-by: Pete Bryan <peter.bryanmicrosoft.com>
2a33d61- Sumologic Data Provider (165)
- Co-authored-by: juju4 <juju4users.noreply.github.com>
- Co-authored-by: Ian Hellen <ianhellemicrosoft.com>
4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (170)
- Updates to init_notebook in nb_init.py
- Moved most of the logic of the Azure Sentinel notebooks nb_check script into msticpy - azure_ml_tools
- Will now create msticpyconfig.yaml if it doesn't exist
- Output message from init_notebook when msticpyconfig.yaml is created in nbinit.py
8214c90 - Minor fixes to VT TI Provider and TIBrowser (173)
- Adding QueryTime instance to Dataproviders in data-data_providers.py. Let's you specify time range (or use default 1 day range) for all queries from provider
- Added Browshot to mpconfig_defaults.yaml so that it can be edited in MP settings editor
ab6eb73 (164)
- added recent blog articles and video to a renamed resources page for RTD - blob_articles.
Fixes
8214c90 - Minor fixes to VT TI Provider and TIBrowser (173)
- Updating VT TI Provider for better severity accuracy with additional "detected_" types.
- Changing ti_browser to display "information" results if no "warning" or "high" items in dataset.
- Updated compound_ctrls.py and mp_config_file.py to work when azure-keyvault-secrets not installed
- Update to QueryTime to fix default max time when "day" is used for units.
e6ea232- Pebryan/2021 6 7 grey noise updates (171)
- Update GreyNoise naming and added custom UA
4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (170)
- Fixing a query error in kql_sent_azuresentinel.yaml
- Fixing an issue with contain naming when creating pivot functions
- Fixing link in TIProviders.rst
- Fixing mypy error in azure_auth_core
- Adding check for KV settings before trying to initialize secrets client in provider_settings.py
- Adding function to search for file in utility.py
- mp_config_edit.py:
- Renaming "Save File" button to "Save Settings"
- Add config_filepath parameter to specify file to save when settings not imported from a file
- Changing item update button to "Update" rather than Save in comp_edit.py
- Strip extraneous quotes from string "default" value in mp_config_control.py
- Corrected open params to "w" rather than "w+" (write append)
- Added "browse" as alias for browse_queries in data_providers.py.
- Fixed misreporting of DataFrame column name in exception when column parameter does not match a column in the input data. in pivot_register.py
- Adding azure_ml_tools.py (from nb_check.py script)
- Added "severities='all'" parameter option. Added friendly warning (rather than exception) when no data is present in the input (either empty DF or nothing above the default severities - in ti_browser.py
- Fixing additional areas breaking in timeline plot module if you don't supply "source_columns" parameter.
- Fixing mypy warning in domain_utils.py
- Fixed allowing geolite to create a database folder even if parent folders don't exist.
- Adding "col" and "column" aliases for lookup_iocs in tilookup.py
- Changed score logic for IBM XForce - prevent misreporting of score==1 as warning.
- Updating version in _version.py
- Added test_azure_ml_tools.py
- unit_test_lib updates:
- Added change_directory context manager (with lock) to allow individual test cases to change directory
- Fixed TEST_DATA_PATH to always return absolute path
- Fixing pylint warnings about the lack of use of context managers in mordor_driver.py, base64unpack.py, morph_charts.py
- Pylint warnings suppression about lowercase enums in tilookup.py
- Fixing version number format problem in azure_ml_tools
- Fixing warning that wasn't processing an f-string in tilookup.py