Threatingestor

Changelog

Breaking Changes
- Due to the recent Twitter API changes, the Twitter operator is no longer supported (https://github.com/InQuest/ThreatIngestor/pull/157)

Features
- ThreatIngestor now offers the ability to include program monitoring via BugSnag. You can include your BugSnag credentials the traditional way by adding them to the `config.yml` (https://github.com/InQuest/ThreatIngestor/pull/157)
- The ability to exclude URLs from RSS and sitemap feeds is now available. This utilizes raw regular expressions (regex) and is not filtered like the `include` option (https://github.com/InQuest/ThreatIngestor/issues/148)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.4.0/

**Changelog**: https://github.com/InQuest/ThreatIngestor/compare/v1.3.3...v1.4.0

Changelog

Breaking Changes
- Due to the recent API paid transition from Twitter ("X"), we had to rebuild our Twitter source from the ground up to accommodate their new API schema. While almost verbatim to the old structure, we did have to make some modifications to the configuration. (https://github.com/InQuest/ThreatIngestor/pull/155)

Bug Fixes
- Sitemap ingestion was missing certain IOCs due to some HTML content being skipped, this is now fixed (https://github.com/InQuest/ThreatIngestor/commit/b661a082744c571aebe61317a60794f020c8a06f)

Features
- Improved `config.yml` validation script
- It now includes a verbosity (`-v`) flag to debugging and cleaner output (https://github.com/InQuest/ThreatIngestor/commit/a646830fd047fb221774f19694b829289e418404)
- Better check when validating operators (https://github.com/InQuest/ThreatIngestor/commit/c59af56f86e21808b9af363e760339fc8f4c7257)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.3.3/

**Changelog**: https://github.com/InQuest/ThreatIngestor/compare/v1.2.0...v1.3.3

Changelog

Bug Fixes
- A small patch was made to update how the sitemap source ingests artifacts. Certain blog URLs should no longer be skipped (https://github.com/InQuest/ThreatIngestor/commit/5dc79f6dc24b8bee67d3710dff8df5bca2e2a4c1)

Features
- New independent `config.yml` validation script for verifying the configuration is appropriately structured as both a YAML file and the minimum requirements for ThreatIngestor are met (https://github.com/InQuest/ThreatIngestor/issues/149)
- Script: `scripts/validate.py`
- New source now allows for VirusTotal user comments ingestion (https://github.com/InQuest/ThreatIngestor/issues/87) (https://github.com/InQuest/ThreatIngestor/commit/f08946d96aed778a59261e61860cb7afe9f7fcaa, https://github.com/InQuest/ThreatIngestor/commit/de66d6e0d2d5d1b76ea679613b5533c458c4a173)
- Web source now runs an extra check against the modified header and saves the status code in the "saved_state" as an additional validation checkpoint before ingesting (https://github.com/InQuest/ThreatIngestor/issues/101) (https://github.com/InQuest/ThreatIngestor/commit/d91e6f1afdc0e1e58f848897aa770f0d58c16e97)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.2.0/

**Changelog**: https://github.com/InQuest/ThreatIngestor/compare/v1.1.0...v1.2.0

Changelog

Bug Fixes
- Merged the url_controller utility into the `twitter` source due to a broken import (https://github.com/InQuest/ThreatIngestor/issues/144)
- Restructured imports for the `image` and `twitter` sources. This should improve compatibility with Python 3.6 (https://github.com/InQuest/ThreatIngestor/commit/d3ecc5a8de3ed1f9eacae52485db317dbc02103e)
- No longer uses urllib module for the `sitemap` source. Now uses the requests module (https://github.com/InQuest/ThreatIngestor/commit/d3ecc5a8de3ed1f9eacae52485db317dbc02103e)
- `RSS` and `sitemap` sources now have better ingestion thanks to improvements made to the HTML content parsing (https://github.com/InQuest/ThreatIngestor/issues/140)

Features
- Automated image extraction from `twitter` sources (https://github.com/InQuest/ThreatIngestor/issues/132)
- New indicator of comprise type for ingested sources: `email` (https://github.com/InQuest/ThreatIngestor/issues/122)
- Updated codebase to match the newest version of [iocextract](https://github.com/InQuest/iocextract) (https://github.com/InQuest/ThreatIngestor/issues/143)
- Regex parsing is now supported for `RSS` sources (https://github.com/InQuest/ThreatIngestor/issues/142)

Hot Fix
- v1.1.1 - Fixed suffocating ingestion when working with RSS and sitemap feeds (https://github.com/InQuest/ThreatIngestor/commit/2b6446162ee04454c71bec9affc15463b8c90697)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.1.1/

**Changelog**: https://github.com/InQuest/ThreatIngestor/compare/v1.0.3...v1.1.0

Changelog

Bug Fixes
- Improved URL extraction for Twitter by utilizing the `pyshorteners` module. Now when the expansion attempt fails the first time, it'll attempt a different method for expanding the URL before returning the artifact (https://github.com/InQuest/ThreatIngestor/issues/128)

Features
- Now offers custom regex filtering for the sitemap ingestion source (https://github.com/InQuest/ThreatIngestor/issues/129)
- Modernized documentation (https://github.com/InQuest/ThreatIngestor/commit/f394da0b5c21bd172529b15effe7810993694cc3, https://github.com/InQuest/ThreatIngestor/commit/d2a8ab317e5483361c508ab49bcf21bffb2f56a1, https://github.com/InQuest/ThreatIngestor/commit/31dd2b3ee851684ecbb044ba6c2fcdeb7ea66271)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.0.3/

**Changelog**: https://github.com/InQuest/ThreatIngestor/compare/v1.0.2...v1.0.3

Changelog

Bug Fixes
- Updated Dockerfile to now include more pip packages and Google tesseract (https://github.com/InQuest/ThreatIngestor/pull/126/commits/126eb85cd028c7046df6d0610458fcbba5050ee6)
- Converted versioning to remove the 'beta' tag (https://github.com/InQuest/ThreatIngestor/pull/126/commits/126eb85cd028c7046df6d0610458fcbba5050ee6)

Features
- New sources: image, sitemap
- `image`: Allows for image string extraction to parse out IOCs (https://github.com/InQuest/ThreatIngestor/pull/123/commits/1b066835415447fa128012f2d17df2665b5b1462)
- `sitemap`: Parses sitemap XML data to locate blogs (https://github.com/InQuest/ThreatIngestor/pull/127/commits/079985e8b6814649ab31d9ac134a98d761f5ab84)

New build info and documentation can be found here: https://pypi.org/project/threatingestor/1.0.2/