Pysigma-backend-kusto

🚀 Release Notes

🐛 Bug Fixes

- **query_table param fix**: Resolved issue where `query_table` would not persist when provided in another pipeline merged by `sigma-cli`

🚀 Release Notes

🐛 Bug Fixes

- **Keyword Detection Items**: Resolved an issue where "Invalid SigmaDetectionItem field name encountered: None" was incorrectly raised for keyword detection items. (Fixes 27 )

- **Table Name Prepending**: Fixed a bug where table names were only prepended to queries in `SigmaCollection` objects and not `SigmaRule` objects. This was addressed by implementing a postprocessing item on each rule instead of using a finalizer. (Fixes 28 )

🧪 Testing Improvements

- **Comprehensive Test Coverage**: Added tests to cover both SigmaCollection and SigmaRule objects for all pipelines, ensuring more robust functionality across different use cases.

📚 Documentation

- **README Update**: The README file has been updated to reflect recent changes and provide more accurate information.

🛠 Maintenance

- **Code Formatting**: Applied minor formatting updates.

- **Development Dependencies**: Updated the dev dependency group in pyproject.toml.

🚀 Release Notes

🌟 Major Changes

🛡️ Microsoft XDR Pipeline (formerly Microsoft 365 Defender)
- 🔄 Microsoft 365 Defender pipeline renamed to Microsoft XDR
- ⚠️ Users should migrate to the new Microsoft XDR pipeline

🆕 Azure Monitor Pipeline (NEW!)
- 🧪 New Azure Monitor pipeline introduced (alpha status)
- 🗃️ Supports field mappings for SecurityEvents and SigninLogs tables
- 📊 All 698 Azure Monitor tables supported in final queries

🔍 Enhanced Sentinel ASIM Pipeline (Beta)
- 🔑 Additional field mappings added (beta status)

📈 Expanded Table Support
- Microsoft XDR: 38 tables
- Sentinel ASIM: 8 tables
- Azure Monitor: 698 tables

🏗️ Codebase Refactoring
- 🧱 Improved organization and structure
- 🔄 Better sharing of components across pipelines

✨ New Features

🎛️ Custom Table Name Support
- 🆕 Set custom table names with `query_table` parameter
- 🐍 Configurable via YAML or Python

🔀 Flexible Rule Category Handling
- 🚫 "Unsupported rule category" error suppressed when the following conditions are met:
- Rule category is absent or category not in `mappings.py` for each pippeline
- A valid table is supplied via `query_table` param

🛠️ Technical Improvements

📜 Table Generation Scripts
- 🤖 New scripts in `utils` folder
- 🔄 Auto-populate valid tables and field schema in `tables.py` for each pipeline

🗺️ Field Mappings
- 🔨 Ongoing improvements for all pipelines

📊 Rule-to-Table Mapping
- 🚧 Work in progress on advanced mapping methods

📚 Documentation

- 📝 Updated README with `query_table` usage
- 💡 New examples for YAML and Python implementations
- FAQ/Troubleshooting section

⚠️ Deprecation Notices

- 🚫 Microsoft 365 Defender pipeline is deprecated
- 🔜 Users should migrate to Microsoft XDR pipeline

🔮 Future Work

- 🔍 Expanding field mappings across pipelines
- 🧠 Developing sophisticated rule-to-table mapping
- 🔧 Refining Azure Monitor and Sentinel ASIM pipelines

📘 Please refer to the updated README for detailed usage instructions and examples of the new features.

🐛 Bug Fixes

🛡️ Sentinel ASIM Pipeline
- Resolved issues with the `sentinelasim` pipeline, improving its stability and reliability.
- Fixes 25

🧪 Testing Improvements

📊 Sentinel ASIM Pipeline
- Added basic test coverage for the `sentinelasim` pipeline, enhancing our ability to catch potential issues early.

🔮 Ongoing Development

We're actively working on expanding and refining our pipeline support:

- The `sentinelasim` pipeline and other pipelines are under active development.
- We're continuously adding more tables and features to these pipelines.
- We're also working on refactoring the codebase to make it more organized and easier to contribute to
- Your patience and feedback during this process are greatly appreciated!

🏷️ Repository Rename
We've renamed the repository from `pySigma-backend-microsoft365defender` to `pySigma-backend-kusto`. This change reflects our expanded focus on supporting the Kusto Query Language (KQL) as a backend for various platforms.

🆕 New Features

📊 Sentinel ASIM Pipeline
- Introduced the `sentinel_asim` pipeline, providing initial beta support for the Sentinel Advanced Security Information Model (ASIM) tables. (Thanks adonm!)

🛠️ Backend Enhancements
- Renamed `Microsoft365DefenderBackend` to `KustoBackend` for clarity and broader applicability.
- The `microsoft_365_defender_pipeline` is no longer automatically applied by the backend, allowing for more flexible configurations.

🔧 Microsoft 365 Defender Pipeline Improvements
- Table renaming is now performed within the pipeline rather than the backend.
- Added the ability to override Sigma Rule category -> table name mappings with a custom `query_table` parameter.

📘 Usage Example
For details on using custom table names, see the [README](https://github.com/AttackIQ/pySigma-backend-kusto?tab=readme-ov-file#%EF%B8%8F-custom-table-names-new-in-030-experimental).

🔮 Future Plans
We're actively working on expanding support for additional KQL-based platforms. Stay tuned for updates on Microsoft XDR and other integrations!

🙏 Acknowledgements
Special thanks to the SigmaHQ Discord community for their valuable input on the repository direction.