Msticpy

In case you're wondering - we never released 1.6.0 to PyPI so this is the official 1.6.0 release.

Summary

Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class (227)
The FoliumMap class now supports layering and subgrouping. This was contributed by tj-senserva
(from Senserva)

Adding New Features to Sentinel APIs (280)
Sentinel APIs has had signficant rework and expansion to cover new public APIs. These include Analytics and Watchlists as
well as Incidents and Bookmarks from earlier releases. Each API now includes modification
operations such as Create, Delete, Update (not all types support all types of modification).

Note: the module and class names drop the "Azure" prefix following the
renaming of Azure Sentinel to Microsoft Sentinel

VirusTotal FileBehavior, FileBrowser and Pivot functions (260)
The VirusTotal V3 module has been expanded to add additional operations and data visualization:
- VT FileBehavior class lets you download and view file detonation data.
- VT ObjectBrowser lets you download and view basic attributes of a file object
- Pivot functions - these expose the VT3 Relationship APIs allowing you to pivot between
related IPs, URLs, Files and Domains.

Added instance handling to Dataproviders (218)
Several providers now support instances - e.g. Splunk, Sumologic can support multiple
configuration entries in msticpyconfig.yaml with the use of an instance suffix, e.g.
Spunk-Env1, Splunk-Evn2.
This change also makes it possible to combine these instances into a single aggregate
data provider and run the same query across the combined instances. This also
works for multiple Microsoft Sentinel workspaces.

Process tree visualization updates to make schema-agnostic (256)
The process tree visualization code previous had several dependencies on Sentinel or MDE
schema specifics. The code should now accept process event data sets from many sources.

**Adding Sysmon ProcSchema** (267) - A schema mapping for Sysmon data was contribute by nbareil

Build pipeline fixes for external forks (270)
Previously builds triggered from external forks would fail for a variety of reasons
(no access to pipeline secrets, MS-Internal-only build actions). This is now fixed so that
external contributors should expect a clean build.

Added default caching option for mordor driver (257)
The OTRF Security Datasets driver and browser now cache security data set and MITRE ATT&CK
data for quicker startup. By default, the cache files are stored in $HOME/.msticpy/mordor
or the location specified in the Dataproviders/Mordor section of the msticpyconfig.

What's Changed
* Ianhelle/proc tree fixes 2022 01 03 by ianhelle in https://github.com/microsoft/msticpy/pull/256
* Bump sphinx from 4.2.0 to 4.3.2 by dependabot in https://github.com/microsoft/msticpy/pull/241
* Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class by tj-senserva in https://github.com/microsoft/msticpy/pull/227
* 20220105 hackathon blob storage conn string by lucky-luk3 in https://github.com/microsoft/msticpy/pull/262
* Added instance handling to Dataproviders by ianhelle in https://github.com/microsoft/msticpy/pull/218
* Added default caching option for mordor driver by ianhelle in https://github.com/microsoft/msticpy/pull/257
* Ianhelle/virustotalv3 additions 2021 10 22 by ianhelle in https://github.com/microsoft/msticpy/pull/260
* Initial attempt at masking failing tests for external forks. by ianhelle in https://github.com/microsoft/msticpy/pull/270
* Adding Sysmon Process Create's ProcSchema by nbareil in https://github.com/microsoft/msticpy/pull/267
* Removing validate configuration for external forks by ianhelle in https://github.com/microsoft/msticpy/pull/273
* Update AzureData to use correct Azure subscription management API by liamkirton in https://github.com/microsoft/msticpy/pull/269
* Added details to README on upgrading msticpy by danielc-evans in https://github.com/microsoft/msticpy/pull/274
* Updated IPython elements and requirements by petebryan in https://github.com/microsoft/msticpy/pull/278
* Fixes and improvements to KqlDriver and KustoDriver Azure auth. by liamkirton in https://github.com/microsoft/msticpy/pull/277
* Miscellaneous fixes for VT, Auth and other items by ianhelle in https://github.com/microsoft/msticpy/pull/281
* Adding New Features to Sentinel APIs by petebryan in https://github.com/microsoft/msticpy/pull/280
* Correcting error introduced in nbinit refactoring by ianhelle in https://github.com/microsoft/msticpy/pull/286

New Contributors
* dependabot made their first contribution in https://github.com/microsoft/msticpy/pull/241
* tj-senserva made their first contribution in https://github.com/microsoft/msticpy/pull/227
* lucky-luk3 made their first contribution in https://github.com/microsoft/msticpy/pull/262
* nbareil made their first contribution in https://github.com/microsoft/msticpy/pull/267
* liamkirton made their first contribution in https://github.com/microsoft/msticpy/pull/269
* danielc-evans made their first contribution in https://github.com/microsoft/msticpy/pull/274

Additional feature details

7fa1598
- Add ability to filter queries by substring in `list_queries` in data_providers.py
- Adding coordinates property to GeoLocation entity geo_location.py
- Simplify verbose option and refactoring complex method in init_notebook
c0a32a0Merge pull request 277 from liamkirton/liamkirton/kql-kusto-driver-auth
- Fixes and improvements to KqlDriver and KustoDriver Azure auth.
ae6bcd7Updated IPython elements and requirements (278)* Updated IPython elements and requirements
79910e2Merge pull request 274 from danielc-evans/updated-readmeAdded details to README on upgrading msticpy
7038e28Changed type hints in enum_parse in utility.py
- Added autoload of Pivot and vt_pivot (if VT config detected)
- Moved some common methods to IPyDisplayMixin class in nbwidgets.py
- Added extra parameter for VT Private API definition to mpconfig_defaults.yaml. Also changed defaults for some providers to Primary=False
- Updates to test_mp_config_controls.py to deal with new VT parameter
- Add fixed vtobject_browser.py + unit test in test_vtlookupv3.py
8308849Added the ability to connect to a blob storage using the connection string. Container name parameter changed to optional.

Additional fix details

7fa1598Miscellaneous fixes for VT, Auth and other items (281)* Updates to Timeseries analysis and plotting modules
- Avoiding exception when cmdline is NaN or non-string in process_tree.py
- Removing "Authenticating to Azure" output since it fires on every query.
- Removing credential caching in azure_auth_core.py

f730ba2Moved geoip GeoLiteLookup to check for and update DB before first query (rather than in init)
Added full copy of GeoLite2-City.mmdb to test data
Linting fixes in mordor_driver.py and test-test_mordor_driver.py
Cherry picked commit for python-package and azure-pipelines yamls

569eb75Forcing install of pandas>=1.30 in pytest block
857b08eSaved mde_proc_pub.pkl as pickle protocol 4 to work in Python 3.6
70b5c55Adding updated notice file.
4599d6eFixed use of hard-code global endpoint in azure_blob_storage.py

cad8256Added create incident and create rule features
70b5c55Adding upgrade to pandas to so that it uses latest version and avoids pickle compatability issue with test data pickled with pandas > 1.2.5Also added higher version constraint to requirements-dev.txt and conda-reqs-dev.txt
Adding updated notice file.

4599d6eAdded instance handling to Dataproviders (218)
- Fixed use of hard-code global endpoint in azure_blob_storage.py
- Adding better handling of response data in kql_driver.py
- Removing default start and end parameters from queries. We should rely on the builtin querytimes control
- Adding issue tracker URL and additional classifiers to setup.py
- Updating import_analyzer.py to include resolved and unresolved paths for determining whether a package belongs to the std library. (these paths may be different if some of them are links)
0b9c81aBump sphinx from 4.2.0 to 4.3.2 (241)

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v.1.5.2...v1.6.1

v.1.5.2
Hotfix release to correct incorrect capitalization reading values for Splunk settings

Summary
This release includes two minor changes:
1. Fixed schema for Microsoft Defender Device Process events - this should allow it to work with the Process Tree visualization
2. Single-sign-on in Azure ML notebooks using MSI

Fixes and New Features

MDE Process Tree
The schema originally deployed for generating process tree data from Microsoft Defender for Endpoint data was incorrect - we were using an internal schema that did not match the data retrieved via the Defender APIs.
This has now been fixed along with some related items:
- DateTime fields returned in the Defender data are automatically converted to pandas Timestamps (previously they were left as strings)
- The process tree and other visualizations will automatically convert required timestamp columns to datetime format - this is useful where data is retrieved from an API or file where the required datetime columns are stored as date strings

Single Sign-On for notebooks in Azure ML (in Microsoft Sentinel)
When you sign on an Azure ML workspace, Managed Identity (MSI) credentials are automatically enabled on the Azure Computes used to run the notebooks. The MSI credentials use the user principal of the user that has logged on (not the machine identity). This allows processes on the Computes to use these credentials to authorize to other Azure resources.
The Log Analytics team made some changes to allow the Managed Identity credentials from Azure ML to authorize to Log Analytics/Microsoft Sentinel data API.

With this release, connecting to Microsoft Sentinel will automatically try to use the MSI credentials. Existing authentication flow options such as Azure CLI and Device Code authentication are still available.

To use MSI-based SSO:
python
qry_prov = QueryProvider("AzureSentinel") we will change this to MicrosoftSentinel soon!
qry_prov.connect(WorkspaceConfig())
`
To force specific authentication flavors, you can set your preferences explicitly in the [msticpconfig.yaml](https://msticpy.readthedocs.io/en/latest/getting_started/SettingsEditor.html#default-azure-authentication-methods). You can also do this at the command line.

To specify an explicit set of authentication types to use (they are tried in the order specified):
python
qry_prov.connect(WorkspaceConfig(), mp_az_auth=["cli", "interactive"])


To skip integrated Azure authentication altogether and use interactive Device Code authentication:
python
qry_prov.connect(WorkspaceConfig(), mp_az_auth=False)
or
qry_prov.connect(WorkspaceConfig(), mp_az_auth=["interactive"])


Detailed changes

4fdf286Ianhelle/mde proctree fixes 2021 12 16 (239)* Changes:

- Removed checking of DataFamily in data_query_reader.py
- Changed default behavior of kql_driver.py to use Azure authentication
- Change security_base.py to use entities.OSFamily rather than query_defns.DataFamily
- Added ensure_df_datetimes function to auto-convert specified columns from string to datetime
- Added unit test test_query_defns.py
- Added ensure_df_datetime to timeline.py
- Added ensure_df_datetime to timeline_duration.py
- Added ensure_df_datetime to proc_tree_build_mde.py
- Added ensure_df_datetime to proc_tree_build_winlx.py
- Changed mpconfig_defaults.yaml to include "msi"
- Switching "Microsoft Sentinel" to "Azure Sentinel" in UI/messages.
- Updated black version in .pre-commit-config.yaml
- Using returned schema in mdatp_driver.py to auto-format datetimes to pandas timestamp
- Returning full response from query_with_results in odata_driver.py
- Some fixes in convert_mde_schema_to_internal - field naming and using tz-aware Unknown time value
- Moved the proc tree schema code from proc_tree_builder.py to proc_tree_schema.py
- Add mocked az_connect to test_kql_driver.py tests - since the change to defaulting to AZ auth in the kql_driver means that tests fail.
- Added test case for MDE public data - mde_proc_pub.pkl
- Adding tooltips to settings editor buttons
- also changing button text for simple setting editor to "Update" (from "Save")
- changing order of checks in base64unpack to let you specify UTF-16 decoding
- updating version to 1.5.1

Release Summary

There are several cool new features in this release:
- Microsoft Sentinel Incident exploration
- RiskIQ Threat Intellligence provider and pivot functions
- Microsoft Defender 365 API/Data provider support
- Preview Azure Data Explorer/Kusto data provider support
- Microsoft graph provider now supports sovereign clouds
- ProcessTree visualization support for Microsoft Defender data in Microsoft Sentinel

Microsoft Sentinel Incident Exploration

0037dd6Merge pull request 211 from microsoft/pebryan/2021-8-18_IncidentTriage

Provides new features to support Incident Triage scenarios (as included in the new Microsoft Sentinel Incident Triage notebook https://github.com/Azure/Azure-Sentinel-Notebooks/pull/120).

Entity Graph Visualization
This new visualization allows for the graphing of an investigation including any alerts, incidents or entities associated with that investigation.
The graph can be created, updated and visualized either as a graph or with a graph and a timeline of the graph event times.

python
from msticpy.vis.entity_graph_tools import EntityGraph
graph = EntityGraph(incident)
graph.plot()


<img title="A sample entity graph" src="https://github.com/microsoft/msticpy/blob/e0b8c371c109d0d26115746192db589e40d70af9/docs/source/visualization/_static/entitygraph.png">

The Entity Graph can also be created from a dataframe of events, either by passing in the dataframe or as a Pandas accessor.

python
df.mp_plot.incident_graph(timeline=True)


<img title="A sample entity graph with timeline" src="https://github.com/microsoft/msticpy/blob/main/docs/source/visualization/_static/entitygraph_w_timeline.png">

Microsoft Sentinel Incident API support

---

RiskIQ PassiveTotal TI provider and Pivot provider

9353d9fRiskIQ threat intel provider & pivots (175)

Delivers a new "RiskIQ" threat intel provider, enabling access to RiskIQ Summary Card and Reputation
datasets for IPs and hostnames with the same syntax and response format used with other TI Providers.
python
ti_lookup = TILookup()
lookup_result = ti_lookup.lookup_ioc(observable="137.184.153.130", providers="RiskIQ")
ti_lookup.result_to_df(lookup_result).T


Introduces a new mechanism for registering MSTICPy Pivot Functions provided by threat intel providers,
then uses that code to register capabilities for nearly all RiskIQ API endpoints on hostnames, FQDN and
IPAddress entities. These pivot functions enable easy integration of RiskIQ-provided intelligence in
existing pivot pipelines, and makes it easy to build new pipelines leveraging unique RiskIQ datasets.
python
(
Dns.RiskIQ.resolutions("jquery.su")
.query("recordtype=='A'")
.mp_pivot.run(Dns.RiskIQ.reputation, column='resolve', join='left')
.mp_pivot.run(IpAddress.RiskIQ.resolutions, column='resolve', join='left')
)


More documentation on this will follow.

Authored by Mark Kendrick <mark.kendrickmicrosoft.com>
Co-authored-by: (some minor contribs!) Ian Hellen <ianhellemicrosoft.com>

---
Preview support for Kusto (Azure Data Explorer)

98864aaIanhelle/kusto and md365 support 2021 11 01 (225)

This release contains a preview (i.e. not fully baked) of support for Kusto cluster queryingand Microsoft Defender 365 data providers.
The Kusto/Azure Data Explorer support is based on the existing Microsoft Sentinel provider.
python
kusto_provider = QueryProvider("Kusto")
kusto_provider.connect(
"azure_data-Explorer://code;cluster='my_cluster';database='my_db';alias='my_alias'"
)

Some more documentation to follow about configuration in msticpyconfig and
how to build template queries for Kusto/Azure Data explorer.

Support for Microsoft 365 Defender data providers
This is slight expansion of our current support for MS Defender for endpoint. Using the new "M365D" provider name
will cause the provider to use the new combined Microsoft Defender APIs, giving you consolidated access
to data from multiple Defender services.

To create a provider for Microsoft 365 Defender:
python
m365d_provider = QueryProvider("M365D")
m365d_provider.connect()


All template queries for that were available for MDE are also loaded for M365D.

Using Microsoft Defender for Endpoint is still supported. To use only
the Microsoft Defender for Endpoint APIs use "MDE".

You can specify your connection parameters in msticpconfig.yaml
`yaml
DataProviders:
...
MicrosoftDefender:
Args:
ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
ClientSecret:
KeyVault:
TenantId: 8360dd21-0294-4240-9128-89611f415c53
`
This configuration expects to find the client secret in your configured Key Vault with the name
"DataProviders-MicrosoftDefender-Args-ClientSecret" - the msticpy config editor will create this for you.
You can store the secret in your config file but, obviously, we would not recommend this.

You can also specify configuration/secret values in the call to m365d_provider.connect().
For more information see the [Defender provider MSTICPy documentation](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#microsoft-365-defender)


Other improvements

The Graph provider now supports sovereign clouds as well as the Azure global cloud.
You can configure settings for MS Defender and MS Graph via the MSTICPy settings editor.
If you have configured settings for these you can now connect with a simplified
syntax, with no parameters required.
python
graph_prov = QueryProvider("SecurityGraph")
graph_prov .connect()


The schema for the configuration is the same format as the Defender format described above.

`yaml
DataProviders:
...
MicrosoftGraph:
Args:
ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
ClientSecret:
KeyVault:
TenantId: 8360dd21-0294-4240-9128-89611f415c53
`

Process Tree visualization supports MS Sentinel schema for MDE process data

The schema of the DeviceProcessEvents table in Microsoft Sentinel is different
from the data queried directly from the MS Defender advance hunting API. This
update allows you to query MS Defender process data from Azure Sentinel and
display in the Process Tree visualization.

New CI Pipelines using GitHub Actions and Azure Pipelines

The old Azure DevOps pipeline has been decommissioned and replaced by two new ones - GitHub actions and an improved version of the Azure DevOps pipeline.

The main advantage of the GitHub actions pipeline is that the results
of tests and code checks are visible to external contributors.

However, there are some component governance and security checks that
we are required to run by Microsoft but which are not currently available
in GitHub actions. We've opted to run both in parallel.

The new Azure DevOps pipeline is more sophisticated and comprehensive than
the previous one. It allows us to run builds and tests on Windows, Linux and
Mac platforms and to cover Python versions 3.6 and 3.8.
(Note: we will likely switch to 3.8 and 3.10 in the near future)

Other changes and fixes
98864aaIanhelle/kusto and md365 support 2021 11 01 (225)
- Fixing issue in odata_driver.py where api_root would keep getting suffix appended if you connected multiple times.
- Adding mutli-cloud support for MS Graph API in security_graph_driver.py
- Fixing syntax errors in graph_alerts.yaml
- Updating tests for test_drivers and test_kql_driver
- Fixed an issue in kql_driver with Kusto returning a different status code format
1c2c1beDisabling IPstack tests because of restricted API allowance (224)*
- Disabling IPstack tests because of restricted API allowance
- Fixing incorrect assignment in test_geoip.py
- 56fcb81Updating readme badge to reflect new build pipeline
- cdff1ffUpdate sphinx requirements in azure-pipelines.yml for Azure Pipelines
- ce8cbe1Update README.md to include downloads
- 0cb20fdUpdated way of getting instance and domain name for azure_ml_tools.py (217)
- Updated checking for version of msticpy using setuptools/pkg_resources in azure_ml_tools.py
- Added warning if running with stale import of newly updated msticpy.
- 8043207Ianhelle/timeline args check 2020 09 21 (219)* Adding some error checking of column names to timeline_values and timeline_duration
- a7498a3Ianhelle/azure pipelines yaml 2021 09 09 (201)*
- Pylint and mypy fixes in:
- anomalous_sequence/model.py
- anomalous_sequence/utils/cmds_params_values.py
- common/pkg_config.py
- common/wsconfig.py
- config/comp_edit.py
- config/mp_config_file.py
- data/data_providers.py
- data/data_query_reader.py
- data/query_store.py
- data/sql_to_kql.py
- datamodel/entities/host.py
- datamodel/entities/ip_address.py
- datamodel/pivot_register_reader.py
- nbtools/morph_charts.py
- nbtools/nbdisplay.py
- nbtools/security_alert.py
- nbtools/security_base.py
- sectools/auditdextract.py
- sectools/base64unpack.py
- sectools/cmd_line.cmd_line.py
- Adding pylint --disable=duplicate-code
- Updating pylint requirements in requirements-dev.txt and .pre-commit-config.yaml
- Add [with_pyroma] to prospector install
- Duplicating noqa MC0001 suppressions since prospector/McCabe now seem to ignore decorators an locate the offending line as the "def" line rather than the decorator line.
- Fixing MyPy-discovered error in nbwidgets.AlertSelector
9f122eerefactored code in test_geoip
907e486Remove un-needed code in test_geoip
8ae01f7Updated test_geoip to fix failure case
e5aadacAdded missing parameters to list_all_signins_geo


v.1.5.0pre3
Release Summary

There are several cool new features in this release:
- Microsoft Sentinel Incident exploration
- RiskIQ Threat Intellligence provider and pivot functions
- Microsoft Defender 365 API/Data provider support
- Preview Azure Data Explorer/Kusto data provider support
- Microsoft graph provider now supports sovereign clouds
- ProcessTree visualization support for Microsoft Defender data in Microsoft Sentinel

Microsoft Sentinel Incident Exploration

0037dd6Merge pull request 211 from microsoft/pebryan/2021-8-18_IncidentTriage

Provides new features to support Incident Triage scenarios (as included in the new Microsoft Sentinel Incident Triage notebook https://github.com/Azure/Azure-Sentinel-Notebooks/pull/120).

Entity Graph Visualization
This new visualization allows for the graphing of an investigation including any alerts, incidents or entities associated with that investigation.
The graph can be created, updated and visualized either as a graph or with a graph and a timeline of the graph event times.

python
from msticpy.vis.entity_graph_tools import EntityGraph
graph = EntityGraph(incident)
graph.plot()


<img title="A sample entity graph" src="https://github.com/microsoft/msticpy/blob/e0b8c371c109d0d26115746192db589e40d70af9/docs/source/visualization/_static/entitygraph.png">

The Entity Graph can also be created from a dataframe of events, either by passing in the dataframe or as a Pandas accessor.

python
df.mp_plot.incident_graph(timeline=True)


<img title="A sample entity graph with timeline" src="https://github.com/microsoft/msticpy/blob/main/docs/source/visualization/_static/entitygraph_w_timeline.png">

Microsoft Sentinel Incident API support

---

RiskIQ PassiveTotal TI provider and Pivot provider

9353d9fRiskIQ threat intel provider & pivots (175)

Delivers a new "RiskIQ" threat intel provider, enabling access to RiskIQ Summary Card and Reputation
datasets for IPs and hostnames with the same syntax and response format used with other TI Providers.
python
ti_lookup = TILookup()
lookup_result = ti_lookup.lookup_ioc(observable="137.184.153.130", providers="RiskIQ")
ti_lookup.result_to_df(lookup_result).T


Introduces a new mechanism for registering MSTICPy Pivot Functions provided by threat intel providers,
then uses that code to register capabilities for nearly all RiskIQ API endpoints on hostnames, FQDN and
IPAddress entities. These pivot functions enable easy integration of RiskIQ-provided intelligence in
existing pivot pipelines, and makes it easy to build new pipelines leveraging unique RiskIQ datasets.
python
(
Dns.RiskIQ.resolutions("jquery.su")
.query("recordtype=='A'")
.mp_pivot.run(Dns.RiskIQ.reputation, column='resolve', join='left')
.mp_pivot.run(IpAddress.RiskIQ.resolutions, column='resolve', join='left')
)


More documentation on this will follow.

Authored by Mark Kendrick <mark.kendrickmicrosoft.com>
Co-authored-by: (some minor contribs!) Ian Hellen <ianhellemicrosoft.com>

---
Preview support for Kusto (Azure Data Explorer)

98864aaIanhelle/kusto and md365 support 2021 11 01 (225)

This release contains a preview (i.e. not fully baked) of support for Kusto cluster queryingand Microsoft Defender 365 data providers.
The Kusto/Azure Data Explorer support is based on the existing Microsoft Sentinel provider.
python
kusto_provider = QueryProvider("Kusto")
kusto_provider.connect(
"azure_data-Explorer://code;cluster='my_cluster';database='my_db';alias='my_alias'"
)

Some more documentation to follow about configuration in msticpyconfig and
how to build template queries for Kusto/Azure Data explorer.

Support for Microsoft 365 Defender data providers
This is slight expansion of our current support for MS Defender for endpoint. Using the new "M365D" provider name
will cause the provider to use the new combined Microsoft Defender APIs, giving you consolidated access
to data from multiple Defender services.

To create a provider for Microsoft 365 Defender:
python
m365d_provider = QueryProvider("M365D")
m365d_provider.connect()


All template queries for that were available for MDE are also loaded for M365D.

Using Microsoft Defender for Endpoint is still supported. To use only
the Microsoft Defender for Endpoint APIs use "MDE".

You can specify your connection parameters in msticpconfig.yaml
`yaml
DataProviders:
...
MicrosoftDefender:
Args:
ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
ClientSecret:
KeyVault:
TenantId: 8360dd21-0294-4240-9128-89611f415c53
`
This configuration expects to find the client secret in your configured Key Vault with the name
"DataProviders-MicrosoftDefender-Args-ClientSecret" - the msticpy config editor will create this for you.
You can store the secret in your config file but, obviously, we would not recommend this.

You can also specify configuration/secret values in the call to m365d_provider.connect().
For more information see the [Defender provider MSTICPy documentation](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#microsoft-365-defender)


Other improvements

The Graph provider now supports sovereign clouds as well as the Azure global cloud.
You can configure settings for MS Defender and MS Graph via the MSTICPy settings editor.
If you have configured settings for these you can now connect with a simplified
syntax, with no parameters required.
python
graph_prov = QueryProvider("SecurityGraph")
graph_prov .connect()


The schema for the configuration is the same format as the Defender format described above.

`yaml
DataProviders:
...
MicrosoftGraph:
Args:
ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa
ClientSecret:
KeyVault:
TenantId: 8360dd21-0294-4240-9128-89611f415c53
`

Process Tree visualization supports MS Sentinel schema for MDE process data

The schema of the DeviceProcessEvents table in Microsoft Sentinel is different
from the data queried directly from the MS Defender advance hunting API. This
update allows you to query MS Defender process data from Azure Sentinel and
display in the Process Tree visualization.

New CI Pipelines using GitHub Actions and Azure Pipelines

The old Azure DevOps pipeline has been decommissioned and replaced by two new ones - GitHub actions and an improved version of the Azure DevOps pipeline.

The main advantage of the GitHub actions pipeline is that the results
of tests and code checks are visible to external contributors.

However, there are some component governance and security checks that
we are required to run by Microsoft but which are not currently available
in GitHub actions. We've opted to run both in parallel.

The new Azure DevOps pipeline is more sophisticated and comprehensive than
the previous one. It allows us to run builds and tests on Windows, Linux and
Mac platforms and to cover Python versions 3.6 and 3.8.
(Note: we will likely switch to 3.8 and 3.10 in the near future)

Other changes and fixes
98864aaIanhelle/kusto and md365 support 2021 11 01 (225)
- Fixing issue in odata_driver.py where api_root would keep getting suffix appended if you connected multiple times.
- Adding mutli-cloud support for MS Graph API in security_graph_driver.py
- Fixing syntax errors in graph_alerts.yaml
- Updating tests for test_drivers and test_kql_driver
- Fixed an issue in kql_driver with Kusto returning a different status code format
1c2c1beDisabling IPstack tests because of restricted API allowance (224)*
- Disabling IPstack tests because of restricted API allowance
- Fixing incorrect assignment in test_geoip.py
- 56fcb81Updating readme badge to reflect new build pipeline
- cdff1ffUpdate sphinx requirements in azure-pipelines.yml for Azure Pipelines
- ce8cbe1Update README.md to include downloads
- 0cb20fdUpdated way of getting instance and domain name for azure_ml_tools.py (217)
- Updated checking for version of msticpy using setuptools/pkg_resources in azure_ml_tools.py
- Added warning if running with stale import of newly updated msticpy.
- 8043207Ianhelle/timeline args check 2020 09 21 (219)* Adding some error checking of column names to timeline_values and timeline_duration
- a7498a3Ianhelle/azure pipelines yaml 2021 09 09 (201)*
- Pylint and mypy fixes in:
- anomalous_sequence/model.py
- anomalous_sequence/utils/cmds_params_values.py
- common/pkg_config.py
- common/wsconfig.py
- config/comp_edit.py
- config/mp_config_file.py
- data/data_providers.py
- data/data_query_reader.py
- data/query_store.py
- data/sql_to_kql.py
- datamodel/entities/host.py
- datamodel/entities/ip_address.py
- datamodel/pivot_register_reader.py
- nbtools/morph_charts.py
- nbtools/nbdisplay.py
- nbtools/security_alert.py
- nbtools/security_base.py
- sectools/auditdextract.py
- sectools/base64unpack.py
- sectools/cmd_line.cmd_line.py
- Adding pylint --disable=duplicate-code
- Updating pylint requirements in requirements-dev.txt and .pre-commit-config.yaml
- Add [with_pyroma] to prospector install
- Duplicating noqa MC0001 suppressions since prospector/McCabe now seem to ignore decorators an locate the offending line as the "def" line rather than the decorator line.
- Fixing MyPy-discovered error in nbwidgets.AlertSelector
9f122eerefactored code in test_geoip
907e486Remove un-needed code in test_geoip
8ae01f7Updated test_geoip to fix failure case
e5aadacAdded missing parameters to list_all_signins_geo

Minor release to address some performance problems with the config editor on Azure ML notebooks platform.
The control uses lots of ipywidgets and the AML notebook front-end has some perf issues with lots of widgets.
The improvement is to not load all controls/widgets for the settings editor - only load each tab as it is selected.

The release also includes a minor change to the entities classes - to allow them to be JSON serializable by adding a
JSON Encoder class to entities.Entity.
python
json.dumps(my_entity_list, cls=entities.Entity.JSONEncoder)

Minor release that implements check for existing Azure CLI credentials and warns user if these are not valid/expired.

Pull request 209 Azure CLI credentials check

I introduced a bug in the previous hotfix that caused the settings editor to ignore changes made.
This should be fixed along with a couple of other related items.