Highlights
We've added comprehensive support for Azure sovereign clouds (China, Germany and US Government).
To set the appropriate cloud open MpConfigEdit and choose the **Azure** tab.
Read more about this [here](https://msticpy.readthedocs.io/en/latest/getting_started/SettingsEditor.html#azure-cloud-and-authentication-settings)
There are also additions and improvements to the visualizations.
Matrix Plot is an interaction plot letting you visualize the interactions between two sets of entities.
A typical use would be seeing connections between source and destination IP Addresses.
Read more about this [here](https://msticpy.readthedocs.io/en/latest/visualization/MatrixPlot.html)
Another visualization change is a major overhaul of the ProcessTree visualization. This now accepts generic data
sets (not specifically Azure Sentinel) and has special support for Microsoft Defender for Endpoint logs.
The data format should be auto-detected.
Finally, we've also collected our visualization methods into a single pandas accessor - `mp_plot`.
Using this accessor you can plot any of the visualizations from a pandas dataframe:
python
df.mp_plot.timeline()
df.mp_plot.process_tree()
New Features
889112dMatrix plot visualization (197)* Matrix plot visualization
Includes
- vis/matrix_plot.py - main plotting code
- MatrixPlot.ipynb - notebook
- visualization/MatrixPlot.rst - documentation
- tests/vis/test_matrix_plot.py - unit test
- mp_pandas_plot - generic pandas extension "mp_plot" that includes all
visualizations
d425c77Process tree updates to handle more generic data (178)*
- Tidying up some of the code in timeline.py
- Moving the pd accessors to timeline_pd_accessor.py
- Updating notebook EventTimeline.ipynb and documentation EventTimeline.rst
- Unit tests for timeline controls in test_timeline.py
- Removed line to slim down columns - need to keep original data columns because they may be
referenced in source_columns, etc.
- Process tree updates for MDE process data
- Updating docs, adding pid_fmt parameter
- Adding explanation of schema to NB and doc
- Adding updated notebook
- Expanded the schema table to show all types.
a00741dAdding support for multiple Azure Clouds (193)
Co-authored-by: Pete Bryan <pebryanmicrosoft.com>
- Adding cloud options to settings, kql_driver and keyvault
- Updated help text and combined CESimpleSettings class.
- Azure auth multicloud support
- Replacing hard-coded URIs with msrestazure enums
- Modularize multicloud support
- Refactoring code to use AzureCloudConfig instead of reading directly from config
- azure_auth.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init, spelling corrections
- azure_auth_core.py - create AzureCloudConfig, and _AzCachedConnect classes to read cloud config values and cache tokens
- cred_wrapper.py - minor formatting
- keyvault_settings.py - use AzureCloudConfig
- ce_common.py - uses AzureCloudConfig in get_def_tenant_id function
- azure_data.py - uses AzureCloudConfig, added credential_scopes option to client class inits, spelling corrections
- azure_sentinel.py - uses AzureCloudConfig. added functions to set default subscription and workspace, changed _build_paths to method
- kql_driver.py - uses AzureCloudConfig, added load_mode == Silent, added mp_az_auth to control using Azure auth instead of native Kqlmagic auth,
- resource_graph_driver.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init
- test_azure_auth_core - Added test for AzureCloudConfig class
- test_resource_graph_driver.py, test_azure_data.py, test_azure_sentinel.py - added real credential classes to test
- msticpyconfig-test.yaml, msticpyconfig.yaml - added Azure sections to test config.
- Update azure-mgmt-subscription and azure_mgmt-resource versions
Fixes
d81801dIanhelle/tests and tidy 2021 09 01 (200)*
- Added separate unit test for nbwidgets to cover cases missed in notebook.
- Removing deprecated code from utility.py
- Removing unneeded function from mordor_driver.py
- Sourcery code refactor in entity.py
- Deprecating create_ip_record function in ip_utils.py
- Split wsconfig tests out of test_pkg_config.py
- Adding test_wsconfig.py. Added new test for couple of utility methods
- Removing redundant import from test_nbwidgets.py
- Moved lots of test files into sub-package folders corresponding to tested modules
- Merged a few tests into single files - test_entity.py and test_timeline.py
- Re-enabling IP_stack/geoip notebook test in test_geoip.py
- Updating use of tqdm to remove deprecated tqdm_notebook in utility.py
- Adding some debugging output to geoip GeoLite class.
- Updating test_geoip.py to use predictable msticpyconfig-test.yaml
- Updating test to restore env var values after test in test_pkg_config.py
- Fixing aliases element in kql query records
- Handle aliases as list or string in help
* Updates from PR review
f58e35cChanging behavior so that vtlookupv3.py does not throw an exception when an item is not found. (199)
- This is especially annoying when doing multiple look-ups and one item is not found.
- Minor changes to VTLookupV3.ipynb - to use get_object API from the vtlookupv3 library
d425c77Process tree updates to handle more generic data (178)
- Fixing bug in customjs and adding mde test files
- Update NOTICE.txt
- Fixing progress widget to use correct visibility settings.
- Changing user_config to be more defensive against bad settings.
- Error in query_store passing list instead of strings to MsticpyUserConfigError exception
* Adding a bit more text/warning to check the browser.
Co-authored-by: Pete Bryan <pebryanmicrosoft.com>
ff87ebbcred-scan suppression (196)