Msticpy

Minor release addressing the following:
* Azure-monitor-query release 1.2.0 changed the format of the endpoint URLs that it accepts. Fixed the azure_kusto driver (currently invoked with the "Kusto_New" data environment) so that it will provide the correct format for 1.2.0+ and pre 1.2.0 versions
* Bug in the kql_driver (MS Sentinel) was causing the kusto_driver to fail when querying. The latter is a subclass of the former and was failing due to an attribute that was defined in the parent (kql_driver) but not in the child (kusto_driver). This affected the older (current) Kusto driver version and does not affect the new azure_kusto ("Kusto_New") driver.
* Updated requirements to allow ipywidgets 8.x to install by default (this is now supported by vs code
* Updated documentation for the new Sentinel and Kusto drivers to add instructions for manually installing the required SDK components (azure-monitor-query and azure-kusto-data)


What's Changed
* Azure monitor endpoint URL has changed format in v1.2.0 by ianhelle in https://github.com/microsoft/msticpy/pull/677


**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.5.2...v2.5.3

* Release is mainly to align *bokeh* version requirements with the new release of Holoviz panel.
- moved bokeh from <3.0.0 to < 4.0.0
* Also fixes an issue with the MicrosoftSentinel attribute disappearing from msticpy

What's Changed

* Ianhelle/hotfix 2.5.2 2023 06 08 by ianhelle in https://github.com/microsoft/msticpy/pull/676


**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.5.1...v2.5.2

Some minor fixes that address:
- importing msticpy without some non-default azure packages installed failed
- added more resiliency to query reader so that the whole thing does not fail if there is bad query file.
- removed initialization dependency on azure-resourcegraph in MicrosoftSentinel class.

What's Changed
* Hotfix for v2.5.1 by ianhelle in https://github.com/microsoft/msticpy/pull/672

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.5.0...v2.5.1

Summary of main changes

* New MS Sentinel and Azure Kusto drivers/data providers - these include support for multi-threaded parallel queries, proxies and user-defined query timeouts.
* Extensibility model for MSTICPy - you can create private data providers, TI and Context providers and load them into MSTICPy alongside the built-in providers.
* MS Sentinel repo query download - add current detection and hunting queries from the Sentinel repo as Sentinel queries runnable from MSTICPy/notebooks
* OSQuery data provider - makes it easy to import OS Query logs to dataframes to do additional processing/analysis on them.
* Panel tabulator now supported as default data viewer (a million times better than the one we built!)

More details on these changes below

Sentinel and Kusto provider new drivers

This change adds replacement drivers for the MSSentinel and Kusto data providers.
In place of Kqlmagic, these drivers use the [azure-kusto-data](https://learn.microsoft.com/azure/data-explorer/python-query-data) and [azure-monitor-query](https://learn.microsoft.com/python/api/overview/azure/monitor-query-readme?view=azure-python) SDKs, respectively.

Currently these drivers are enabled alongside the existing versions - in a future version we will make these the defaults for Sentinel and Kusto.

Some of the main changes with these new versions:

* They use the provider names `MSSentinel_New` and `Kusto_New` when creating a QueryProvider instance.
* Both drivers support setting proxies for firewall-protected networks
* Both drivers support custom configuration of the server timeout via a `timeout` parameter
* Both drivers use integrated Azure authentication by default and support the `auth_types` and `tenant_id` parameters used elsewhere
in MSTICPy
* Both drivers support threaded execution for parallelizing queries (across multiple workspaces/clusters or split by time) - this functionality, however, will be exposed in v2.6.0 via a separate feature.
* The MSSentinel_New driver allows you to execute the same query across multiple workspaces in parallel and returns the results as a combined dataframe.
* Some of the previous parameters have been deprecated:
* ``mp_az_auth`` is replaced by ``auth_types`` (the former still works but will be removed in a future release).
* `mp_az_auth_tenant_id` is replaced by `tenant_id` (the former is not supported in the new providers).

**Note**: in order to use these new versions you must have the azure-kusto-data and/or azure-monitor-query Python packages
installed. You can install these using `pip install msticpy[azure]` or install them separately using pip.

For more details on how to use these providers, see:
* [Documentation for the new Sentinel provider](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProv-MSSentinel-New.html)
* [Documentation for the new Kusto provider](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProv-Kusto-New.html)

Changes specific to the MS Sentinel provider

Connecting to multiple workspaces allows you to run queries across these workspaces and return the combined results as a single Pandas DataFrame. The workspaces must use common authentication credentials and should have the same data schema.
python3
use workspace names if these workspaces are configured in msticpyconfig.yaml
qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"])

or use a list of workspace IDs
qry_prov.connect(workspaces=["e6b4bc15-119b-45a2-8f3d-c39ed384ed37", "b17e0e5a...."])

run query against connected workspaces
qry_prov.SecurityAlert.list_alerts()


Changes specific to the Kusto provider

* The settings format has changed (although the existing format is still supported albeit with some limited functionality).
See the Kusto provider documentation for details.
* In the earlier implementation of driver you can specify a new cluster to connect to in when executing a query. This is no longer supported.
Once the provider is connected to a cluster it will only execute queries against that cluster. (You can however, call the connect() function to
connect the provider to a new cluster before running the query.)
* Filtering pre-defined queries by cluster. If you have MSTICPy query definitions for the Kusto provider, these will all be attached as methods
of the QueryProvider, when it is created. However, as soon as you connect to a specific cluster, the queries will be filtered down to show
only the queries that are intended to run on that cluster.
* New APIs (exposed via the query_provider):
* get_database_names() - return list of databases for the connected cluster
* get_database_schema() - return table schema for a database in the cluster
* configured_clusters() - return a list of clusters configured in *msticpyconfig.yaml*
* set_cluster() - switch connected to cluster to a different one (you can use the connect method to do this, which also lets you specify
additional connection parameters).

Extend MSTICPy with Data provider, TI provider and Context provider plugins

This adds the ability to "side-load" data providers, TI providers and context providers. If you have a data/TI/context source that you want to use in MSTICPy you can write a provider (deriving from one of the base provider classes) and tell MSTICPy where to load it from.

In a future release we'll build on this framework to let you install plugins from external packages and provide some cookie-cutter templates to generate skelton provider classes.

Writing a TI provider or Context provider (partial example)

python

class TIProviderHttpTest(HttpTIProvider):
"""Custom IT provider TI HTTP."""

PROVIDER_NAME = "MyTIProvider"
_BASE_URL = "https://api.service.com"
_QUERIES = _QUERIES = {
"ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"),
"ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"),


Telling MSTICPy to load the plugins

Load on demand
python

import msticpy as mp

mp.load_plugins(plugin_paths="/my_modules")

or multiple paths
mp.load_plugins(
plugin_paths=["./my_modules", "./my_other_modules"]
)


Or specify in `msticpyconfig.yaml`

yaml
...
Custom:
- "testdata"
PluginFolders:
- tests/testdata/plugins
Azure:
...

See the new [Extending Msticpy](https://msticpy.readthedocs.io/en/latest/ExtendingMsticpy.html) section in our docs.
If you want to contribute any of the drivers you write, also check out the new [Development](https://msticpy.readthedocs.io/en/latest/Development.html) section in the MSTICPy docs.

OS Query Provider
Great contribution from juju4 here (with a bit of collaboration with ianhelle).
Create a MSTICPy QueryProvider with the data environment name "OSQueryLogs" and load forensic logs from OSQuery.

python3
specify one or more paths to folders where the dumped JSON OSQuery logs can be found
qry_prov = mp.QueryProvider("OSQueryLogs", data_paths=["~/logs1", "~/logs2"])
qry_prov.connect()
qry_prov.list_queries()


['osquery.acpi_tables',
'osquery.device_nodes',
'osquery.dns_resolvers',
'osquery.events',
'osquery.fim',
'osquery.last',
'osquery.listening_ports',
'osquery.logged_in_users',
'osquery.mounts',
'osquery.open_sockets',
...

Each event type is available as a separate function that returns a pandas DataFrame with the combined events from the logs for that type
python3
qry_prov.osquery.processes()


Downloading Sentinel Detection and Hunting queries for the Sentinel Query Provider

We haven't finished documenting this or integrating it fully, so will leave the full announcement of this until the next release. If you want to play around with it look at the following module:
python3
from msticpy.data.drivers.sentinel_query_reader import download_and_write_sentinel_queries

download_and_write_sentinel_queries(
query_type="Hunting", or "Detections"
yaml_output_folder="./sentinel_hunting",
)
qry_prov = mp.QueryProvider("Sentinel_New", query_paths=["./sentinel_hunting"])

Since there are lots of queries, the import might take a little while in its current form.

Panel Tabulator now available as a DataViewer control.

HoloViz Panel is a powerful Bokeh-based data exploration & web app framework for Python. It has an immense amount of functionality that you can read about at the [Panel documentation site](https://panel.holoviz.org/index.html). You need to have **panel** installed for the Tabulator-based viewer to run (`pip install panel`).

Unfortunately, the documentation for our Tabulator view never made it into this release but most of the functionality should be obvious from the UI. There are some useful load-time parameters that you can use at startup for things like:
* selecting an initial column set.
* adding columns to a per-row expando pane - useful for viewing long column values such as command-line.

We also kept the column chooser widget from the previous data viewer so that you can interactively select which columns to display. The [Tabulator MSTICPy initialization parameters](https://github.com/microsoft/msticpy/blob/1f87529a5217af6c0c56d7364850394d557d4b69/msticpy/vis/data_viewer_panel.py#L34) are documented in the code.

Most of the Tabulator `init` parameters are also passed through to the underlying control - which give you an immense amount of control over the viewer. These are described in the [Panel Tabulator documentation](https://panel.holoviz.org/reference/widgets/Tabulator.html)


Big thanks to our contributors in this release!
juju4
jannieli
ianhelle
Tatsuya-hasegawa
FlorianBracq
danielyates2
petebryan
ashwin-patil

What's Changed PR Reference
* Updated Sentinel incident docs to reflect filtering options by petebryan in https://github.com/microsoft/msticpy/pull/648
* Read the docs update for Managed spark installation by ashwin-patil in https://github.com/microsoft/msticpy/pull/647
* Added documentation for the polling detection module by danielyates2 in https://github.com/microsoft/msticpy/pull/601
* Add PyVis panel version of DataViewer. by ianhelle in https://github.com/microsoft/msticpy/pull/646
* add LocalOsquery driver based on LocalData one by juju4 in https://github.com/microsoft/msticpy/pull/624
* Bump httpx from 0.23.3 to 0.24.0 by dependabot in https://github.com/microsoft/msticpy/pull/655
* Sentinel and Kusto new providers by ianhelle in https://github.com/microsoft/msticpy/pull/656
* Fix a critical bug of Splunk results reader, lack of pagination by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/657
* Update azure_kusto_driver.py by FlorianBracq in https://github.com/microsoft/msticpy/pull/664
* Ianhelle/mp extensibility 2023 02 09 by ianhelle in https://github.com/microsoft/msticpy/pull/632
* Format of cluster name has changed in new KustoClient. by ianhelle in https://github.com/microsoft/msticpy/pull/667
* Write Sentinel queries to YAML for Github Browser by jannieli in https://github.com/microsoft/msticpy/pull/491

New Contributors
* danielyates2 made their first contribution in https://github.com/microsoft/msticpy/pull/601
* Tatsuya-hasegawa made their first contribution in https://github.com/microsoft/msticpy/pull/657
* jannieli made their first contribution in https://github.com/microsoft/msticpy/pull/491

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.4.0...v2.5.0

Main changes for this release

There are no huge changes in this release but a good variety of important updates and fixes.
We're also delighted to welcome 3 new contributors to the MSTICPy family:
* ZeArioch
* ctoma73
* jllangley

Thanks so much!

New Threat Intel provider for Pulsedive from fr0gger 609

This includes a standard MSTICPy TI provider (so you can include it in you collection of providers used for
regular TI checks on IPs, URLs, etc. This provider also contain a few custom methods that let to query
some other facets of the Pulsedive data. For example, the `explore` function that allows you to use
the pulsedive query language
python
pddetail = pdlookup.explore(query="ioc=pulsedive.com or threat=AgentTesla")
pddetail

You can also request a can on a domain or URL
python
pdscan = pdlookup.scan(observable= "alvoportas.com.br")
pdscan

To use any of the Pulsedive features you'll need an account and API key from [Pulsedive](https://pulsedive.com/api/)
See more details of the usage in the [Pulsedive notebook](https://github.com/microsoft/msticpy/blob/main/docs/notebooks/PulsediveLookup.ipynb)

Process tree updates 637
- ZeArioch added Process Tree support for FireEye HX data so it should be automatically recognized and render correct
- We also added the ability to export a process tree as a text object - which is useful if you want to copy and paste
a tree or part of it into a non-HTML document. See the [Process Tree docs](https://msticpy.readthedocs.io/en/latest/visualization/ProcessTree.html#tree-to-text) for more details

+-- Process: C:Program FilesMicrosoft Monitoring AgentAgentMonitoringHost.exe
PID: 0x888
Time: 1970-01-01 00:00:00+00:00
Cmdline: nan
Account: nan LoginID: 0x3e7
+-- Process: C:WindowsSystem32cscript.exe PID: 0x364
Time: 2019-01-15 04:15:26+00:00
Cmdline: "C:Windowssystem32cscript.exe" /nologo
"MonitorKnowledgeDiscovery.vbs"
Account: WORKGROUPMSTICAlertsWin1$ LoginID: 0x3e7
+-- Process: C:Program FilesMicrosoft Monitoring AgentAgentHealth Service
StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe PID:
0x1c4
Time: 2019-01-15 04:16:24.007000+00:00
Cmdline: "C:Program FilesMicrosoft Monitoring AgentAgentHealth
Service
StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe"
GetInventory "C:Program FilesMicrosoft Monitoring
AgentAgentHealth Service
StateCT_602681692workServiceStateServiceState.mof" "C:Program
FilesMicrosoft Monitoring AgentAgentHealth Service
StateCT_602681692workServiceState"
Account: WORKGROUPMSTICAlertsWin1$ LoginID: 0x3e7


Miscellaneous fixes 644
This sounds like a small item but contain several important fixes:
- Azure authentication (az_connect) now avoids throwing exceptions if you ask it to use authentication types (e.g. clientsecret) where parameters are not passed (or available in environment variables). It will now just ignore those credential types and only throw an exception if no usable credential types remain.
- Updates to API documentation
- A new IPython magic "%save_to_cell" - this lets you save a Python object (e.g. a DataFrame to a base64-encoded blob in a new cell. The cell contains code to restore the original data. This is subject to the usual caveats about pickle - including the security ones. *Do Not* run a cell that unpickles some arbitrary data in notebooks that you do not trust.
- A bunch of changes/fixes to the Sentinel APIs
- Most of these are fixes related to the newly-supported Sentinel Dynamic Summaries feature
- Some minor fixes also to Sentinel core

Python Logging support 640
We should have had this from the beginning but it's never too late to start correcting your mistakes.
We've implemented a central logging module and started to instrument some of the code that is especially complex
and where people often get stuck with cryptic errors. E.g. the `init_notebook` function.
We also enabled in in the authentication modules (`az_connect`) in 644
Most of the time, this will be invisible. However, if you need it you can just do the following:
python
import msticpy as mp if not already imported
mp.set_logging_level("INFO")

Then re-run the function that you are having trouble with again.
You can also use the `MSTICPYLOGLEVEL` variable to control this. And, if you want to log to a file, set the env variable `MSTICPYLOGFILE` to the path of your log file. (You'll need to restart the kernel/python session and reload MSTICPy for this to take effect).

Support for Bokeh 3.0 630 642 and 650
ctoma73 did some awesome work to track down problems with compatibility with Bokeh 3.0 and fix all of them (a lot were tedious mypy/linting fixes due to some of the more dynamic nature of the Bokeh 3.0 object model).
You'll notice in 650 that we still have Bokeh 2.4.3 in the MSTICPy requirements. We're not going to change that just yet since we want compatibility with [PyViz/HoloViz panel](https://panel.holoviz.org/) - you will likely see some panel-related features in the next minor release.
Despite this (and assuming you can ignore some pip warning about MSTICPy not being compatible with Bokeh 3.x) you can install Bokeh 3.0 after MSTICPy and enjoy the delights of the new release. All of our code should be compatible (tested with 3.0.0 and 3.1.0).

That's all for this release.
We'll likely be doing a follow-on 2.5.0 release that will include several contributions from our 2023 Hackmonth (which turned into a HackNMonths event).


What's Changed
* Add support for FireEye HX acquisition packages in `process_tree` by ZeArioch in https://github.com/microsoft/msticpy/pull/616
* Adding Pulsedive as Threat Intel provider by fr0gger in https://github.com/microsoft/msticpy/pull/609
* Fix error when latest version 3.0.3 of bokeh is installed by ctoma73 in https://github.com/microsoft/msticpy/pull/630
* Adding logging and updating settings access by ianhelle in https://github.com/microsoft/msticpy/pull/640
* ProcTree and init_notebook fixes by ianhelle in https://github.com/microsoft/msticpy/pull/637
* Adding data query paths test for DEX support by ianhelle in https://github.com/microsoft/msticpy/pull/638
* Fixing RangeTool with bokeh 3.1.0 not a GestureTool by ctoma73 in https://github.com/microsoft/msticpy/pull/642
* Modified the upload_df method to split the data into batches of 10,00… by jllangley in https://github.com/microsoft/msticpy/pull/633
* Misc updates for 2.3.2 release: by ianhelle in https://github.com/microsoft/msticpy/pull/644
* Reverting to bokeh version 2.4.3 for default install by ianhelle in https://github.com/microsoft/msticpy/pull/650

New Contributors
* ZeArioch made their first contribution in https://github.com/microsoft/msticpy/pull/616
* ctoma73 made their first contribution in https://github.com/microsoft/msticpy/pull/630
* jllangley made their first contribution in https://github.com/microsoft/msticpy/pull/633

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.3.1...v2.4.0

This is minor release with mostly fixes.

Some higlights from the 631 PR

629 - You can now suppress progress bar for Threat Intel lookups (useful to avoid screen mess
when running multiple lookups from other code)
python
tilookup.lookup_iocs(data, progress=False)


572 - We've had a long-running issue in Azure Machine Learning where the UI does not correctly
handle javascript written by the notebook. This results in JS code in the output cells. While we're waiting
for AML to re-adopt the latest Azure Notebooks package and get rid of this bug altogether we've
added a fix to suppress javascript text for out Kqlmagic data provider

* Fix to Azure ML use - automatic creation of msticpyconfig.yaml was writing the file to
the wrong place, so users always got the message that no config file was found.

* We had a request (again for batch jobs) to remove automatic display of license information in the geoip module.

* Using MSTICPy offline or in isolated environment - it has always been our goal to support this but
we recently discovered that we were running a `check_version` call from `init_notebook`. This function
did not handle network failure and crashed the entire init_notebook process. This has been fixed
so should be runnable offline or in air-gapped networks.

* Related to this we've also cleaned up remaining units tests that make outbound network requests.

Full Changelist
* Adding job to file issue if main build fails. by ianhelle in https://github.com/microsoft/msticpy/pull/613
* Removing prospector from CI build by ianhelle in https://github.com/microsoft/msticpy/pull/619
* Reverting PR 496 - Removing blank sub-id from resource graph list by ianhelle in https://github.com/microsoft/msticpy/pull/621
* Resolved issues with nextLink following in Sentinel API calls by petebryan in https://github.com/microsoft/msticpy/pull/617
* Fix MDE procschema by rrevuelta in https://github.com/microsoft/msticpy/pull/626
* Bump sphinx-rtd-theme from 1.1.1 to 1.2.0 by dependabot in https://github.com/microsoft/msticpy/pull/628
* Bump sphinx from 5.3.0 to 6.1.3 by dependabot in https://github.com/microsoft/msticpy/pull/610
* Ianhelle/misc fixes 2023 02 17 by ianhelle in https://github.com/microsoft/msticpy/pull/631

New Contributors
* rrevuelta made their first contribution in https://github.com/microsoft/msticpy/pull/626

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.2.3...v2.3.1