One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to
use and maintain. Several years of organic growth had seen modules created in places that
seemed like a good idea at the time but did not age well.
The discussion about the V2 structure can be found here 320.
This first pre-release of V2.0 has a small number of new features but is mostly
about testing compatibility. We'll be adding features to future pre-releases in
the next week or two.
**Due to the re-organization, many features are no longer in places where they used
to be imported from! **
We have tried to maintain compatibility with old locations by adding "glue" modules.
These allow import of many modules from their previous locations but will issue a Deprecation warning if loaded from here.
The warning will contain the new location of the module - so you should update your code to point to this new location.
This table gives a quick overview of the V2.0 structure
| folder | description |
|-----------|----------------------------------------------------------------------------------|
| analysis | data analysis functions - timeseries, anomalies, clustering |
| auth | authentication and secrets management |
| common | common used utilities and definitions (e.g. exceptions) |
| config | configuration and settings UI |
| context | enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup |
| data | data acquisition/queries/storage/uploaders |
| datamodel | entities, soc objects |
| init | package loading and initialization - nbinit, pivot modules |
| nbwidgets | nb widgets modules |
| transform | simple data processing - decoding, reformatting, schema change, process tree |
| vis | visualization modules including browsers |
Notable things that have moved:
* most things from the sectools folder have migrated to context, transform or analysis
* most things from the nbtools folder have migrated to:
* init - (not to be confused with __init__) - package initialization
* vis - visualization modules
* pivot functionality has moved to init
If you are having trouble finding a module, we have added a simple search function
python
import msticpy
msticpy.search("riskiq")
Any matches will be returned in a table with links to the module documentation
---
<h4>Modules matching 'riskiq'</h4>
<table class='table_mod'>
<tr class='cell_mod'><th>Module</th><th>Help</th></tr>
<tr class='cell_mod'><td>msticpy.context.tiproviders.riskiq</td><td>
<a href={https://msticpy.readthedocs.io/en/latest/api/msticpy.sectools.tiproviders.html#msticpy-sectools-tiproviders-riskiq-module} target='_blank'>msticpy.context.tiproviders.riskiq</a></td>
</tr>
</table>
---
The Help link is to the current release version of the documentation, so will not
map to the correct path on ReleaseTheDocs for the release documentation. You should be able to
map this to the pre-release doc version by adding v2.0.0.rc1 into the URL.
https://msticpy.readthedocs.io/en/v2.0.0rc1/api/msticpy.sectools.tiproviders.html
Other Features/Changes
Pandas accessors have been consolidated into two main ones:
* df.mp.*func* - contains all of the transformation functions like base64 decoding, ioc searching, etc.
* df.mp_plot.*func* - contains all of the visualization accessors (timeline, process tree, etc.)
We've also done a lot of work to clean up warnings, move some remaining Python unittest tests to pytest
format and random other things that you do when you're cleaning house. A fuller but
not especially readable list of changes follows.
Please let us know if you hit any issues with the re-jigged structure
or any other bugs. File them as issues or reach out to msticpymicrosoft.com
List of changes
Co-authored-by: Ian Hellen <ianhellemicrosoft.com>
Co-authored-by: Pete Bryan <pebryanmicrosoft.com>
* 857835dIanhelle/mpconfigedit fix from main 2022 05 22 (396)* Fix for MpConfigEdit ValueError
* Updating Dockerfile source to mcr anaconda
* bd96d40Ianhelle/v2 reorg directories 2 2022 04 12 (377)* Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver
* Some fixes to Kusto common_imports
* now works with Kusto config entry without instance suffix
* can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
* added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
in the data_families key.
* Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
* Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
* Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.
* Reorganized logic for handling parameters and failing on invalid file path in config module.
* Moved analysis.data to transform folder
* Moved data.context to context
* Moved auth and secrets modules to auth folder
* Moved nbwidgets to new folder
* Updated deprecation warning in glue modules to v2.2
* Fix to URLs in README.md
* Adding init folder - moved:
* nbinit.py, user_config.py, pivot.py, azure_ml_tools.py to here
* also moved vt_pivot.py and pivot_ti_provider.py to init/pivot_init
* Renaming datamodel/pivots to datamodel/pivot
* Moved azure_blob_storage.py to data/storage folder
* Refactored query_container to data_types.py - to be separate types for queries (query_container.py) and pivots (pivot_container.py)
* Moved browsers to vis folder
* Updating API docs for moves
* Moved all pivot functions to init folder.
* Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD)
* Added automatic acquisition of globals() in nbinit.py
* Fixed a couple of bugs in pivot_pipeline.py
* Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization.
* Added trap to timeline when supplied with no data.
* Fixed incorrect escaping in regex in kql_driver.py
* Notebook updates for errors and invalid links.
* Added script to run all notebooks for testing
* Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x). Other components use the default.
* Updated typing rigor for pkg_config::get_http_timeout
* Fixed bugs in test test_pkg_config.py and test_code_view.py
* Fixed re-auth on query issue in KQL driver
* Fixed kql_driver tests
* Kusto provider
* now works with Kusto config entry without instance suffix
* can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
* added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
in the data_famiies key.
* Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
* Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
* Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.
* Reorganized logic for handling parameters and failing on invalid file path in config module.
* e2e743aIanhelle/main mergeback 2022 04 05 (355)* Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters.
* Fixed query_time widget so that you can reset time range from parameter
* Added additional unit test for QueryTime setter
* Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line.
* Changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName.
* Updated TimeSpan class so it has more flexible constructor
* Added account_id as a parameter for list_aad_signins_by_account query
* Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts
to defaults. This can cause errors if the defaults are different to user-specified parameters.
There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings.
* aa14784moved list_sentinel_workspaces to AzureData
* f3e82d9New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from 352.
* ba33c38Ianhelle/pivot dataprov selfload 2022 03 15 (343)* Query providers load pivots dynamically when created.
* Renamed query paths and changed data_providers so that only env-specific queries are loaded.
* Moving ensure_df_datetimes to common/data_utils.py to avoid circular imports
* Consolidated data-related pandas accessors into single module.
* Fixing circular dependency in iocextract
* Fixing linting errors in data_providers.py, azure_resource.py, host.py, process.py, pivot_data_queries.py
* Adding default timeout values to httpx calls.
* Changing tor_exit_nodes.py Tor provider to defer download of tor list until first lookup
* Fixing test for trigger Tor node download before running test.
* Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. (from main branch)
* Now loads with empty settings rather than throw exception.
* Added unit test case
* Aligning splunk_uploader params with base class
* 243f196Ianhelle/implement isort branch post-fixes 2022 03 21 (346)* move query files
* Stub files and restructure fixes
* Merging in updates to Azure auth
* Making tests more resilient for multiple environments
* restructure cyberreason and splunk queries
* Fixed incorrect vtlookup
* Updating missed Conda version for respx
* Updated cybereason folder names
* Fixed Sentinel APIs
* Updated test mocked data to match new Sentinel APIs
* Renamed data.context_providers to context
* Renamed data.common to data.core
* Added docstrings to redirection files so that they link to right location in read-the-docs
* Updated docs with new paths
* Updated notebooks with new module paths
* Changed RTD to generate an API page for each module.
* Add text to deprecation warning that we'll remove in v2.0.0
* Adding isort to pre-commit.yaml
* Isorting all of the imports in msticpy, tests and tools
* Renaming query folders
* Removing some automatic imports from msticpy.__init__.py
* Adding search function to find modules to utility.py
* Fixing old paths in test_timeline.py
* Adding triggers for release branches
* Adding isort to requirements-dev and conda-reqs-dev.txt
* Bandit FPs in anomaly sequence modules
* Fixed failing clustering notebook
* Errors in notebook and keyvault tests
* Fixing test issues in MicrosoftDefender.ipynb and EventClustering.ipynb
* Bug using wrong httpx code syntax in odata_driver.py
* Removing auto-load of VTLookup in nbinit
* Forcing notebook tests to use test msticpyconfig-test.yaml
* Added missing __init__.py to tests/data/browsers
* Removing vtlookup import from sectools init because of circular import error
* Moving IPStack check for API key to first call (rather than __init__) to avoid error on load.
* prospector config changed produces deprecation warning and non-zero exit code.
* Fixing misconfigured prospector.yaml
* Getting rid of warning from test_nbinit
* Adding McCabe suppression to ip_utils.py
* removing version restriction for prospector in Github actions python-package.yaml
* Powershell viewer PR
* Elastic driver skeleton and changes to allow driver-specific param substitution
* Cleaning up some import redirections to point to new locations
* Fixing circular import in vtfile_behavior
* Adding placeholder class to allow imports to sort of work even if sub-modules fail to import
* Refactored _value_or_default in query_source to reduce complexity
* Fixed import errors in elastic_driver.py and splunk_driver.py
* Fixed import from old location in nbinit
* Fixing warning in code_view.py
* Fixed test failure in test_code_view
* Changing test_timeline.py to use new mp_plot accessor in place of deprecated one.
* Fixing test failure in test_timeline.py
* 3f3140aSync changes to main into v2 branch (330)* adding devcontainer files (321)
* Syncing recent IPython-related changes (to skip magic creation if not in IPython)
* f7923e5Minor quality of life improvements to AzureData & MicrosoftSentinel (331)* Updated names in AzureData to match MicrosoftSentinel
* Added connection checks to Sentinel features
PRs
* Sync changes to main into v2 branch by ianhelle in https://github.com/microsoft/msticpy/pull/330
* Ianhelle/msticpy v2.0.0 merge updates 2022 03 14 by ianhelle in https://github.com/microsoft/msticpy/pull/338
* Ianhelle/implement isort 2022 02 15 by ianhelle in https://github.com/microsoft/msticpy/pull/327
* Ianhelle/implement isort branch post-fixes 2022 03 21 by ianhelle in https://github.com/microsoft/msticpy/pull/346
* Ianhelle/pivot dataprov selfload 2022 03 15 by ianhelle in https://github.com/microsoft/msticpy/pull/343
* Ianhelle/main mergeback 2022 04 05 by ianhelle in https://github.com/microsoft/msticpy/pull/355
* Merging changes from main for geoip.py, config editor and kusto_driver by ianhelle in https://github.com/microsoft/msticpy/pull/359
* Pebryan/2022 4 14 auth merge by petebryan in https://github.com/microsoft/msticpy/pull/368
* Fixed minor issues by petebryan in https://github.com/microsoft/msticpy/pull/372
* Ianhelle/v2 reorg directories 2 2022 04 12 by ianhelle in https://github.com/microsoft/msticpy/pull/377
* Ianhelle/mpconfigedit fix from main 2022 05 22 by ianhelle in https://github.com/microsoft/msticpy/pull/396
**Full Changelog**: https://github.com/microsoft/msticpy/compare/v.1.5.2...v2.0.0.rc1