Misp-stix

Overview

- Introducing the first version of a more generic way of parsing Observable objects from different sources.
- as the maintenance and extension of the Observable objects conversion to MISP mapping is a continuous task, some content might be missed in the conversion. Feel free to report any issue using the github issues system.
- Merged pull requests:
- Use MISP event UUID for bundle ID by coolacid in 26
- Fix naive timestamp by SYNchroACK in 35
- Work in Progress to be released soon: STIX 2.0 & 2.1 patterning expressions parsing

Detailed changelog

Changes

* [poetry] Bumped latest dependencies versions. [Christian Studer]

* [package] Updated poetry & pymisp requirement. [Christian Studer]

- In order to better support git dependencies, we
updated poetry because it is required in order
to use git dependencies.
- With the change on poetry we can then use the
git dependency for pymisp - for now

* [package] Bumped version. [Christian Studer]

* [submodules] Bumped latest versions. [Christian Studer]

* [documentation] Regenerated the documentation. [Christian Studer]

* [stix2 export] A quick reuse of an existing SDO creation function. [Christian Studer]

* [documentation] Regenerated the MISP -> STIX documentation with the recent mapping updates. [Christian Studer]

* [documentation] Updated mapping documentation following some recent changes. [Christian Studer]

* [github] Enabling github actions on dev branch. [Christian Studer]

* [poetry, package] Updated python & the library versions. [Christian Studer]

* [github, python] Removing support for 3.7 and added 3.11. [Christian Studer]

Fix

* [stix2 import] Fixed missing imports removed by mistake. [Christian Studer]

* [stix2 import] Some cleanup. [Christian Studer]

- Better readability when possible
- Fixed typing
- Simplified some parts using `getattr` instead of
`hasattr` when possible

* [stix2 import] Fixed duplicate property that was causing issues with the parent class property. [Christian Studer]

* [tests] Fixed the remaining latest datetime/timestamp values that were possibly missing. [Christian Studer]

- Testing `datetime` values - i.e from the
`datetime` python library - instead of str

* [stix2 import] Fixed Marking definition objects handling. [Christian Studer]

- There are still some Marking definition we don't
parse yet - the ones with no `definition_type`
value - and we now properly handle the exception
that appear when we try to look at the ones that
are not loaded

* [stix2 import] Fixed wrong variable name. [Christian Studer]

* [stix2 import] Removed unused variable. [Christian Studer]

* [documentation] Fixed datetime/timestamp values in the ampping documentation. [Christian Studer]

* [tests] Fixed unittests on datetime/timestamp fields/values. [Christian Studer]

* [tests] Made sure all the datetime/timestamp fields/values are properly set in test samples. [Christian Studer]

* [stix2 export] Properly exporting datetime/timestamp fields/values. [Christian Studer]

* [tests] Made some datetime values UTC. [Christian Studer]

* [stix2 export] Fix naive timestamp. [Tomas Lima]

* [tests] Fixed relationships tests to match the recent changes on the default relationship types. [Christian Studer]

* [stix2 export] Typo. [Christian Studer]

- Fixes e918f69 and thus 33 for good this time

* [stix2 export] Fixed default relationships used between SDOs. [Christian Studer]

- The `relationship_specs` mapping dictionary now
only conains default relationships that are
unique between 2 SDOs, if there are at least 2
possible default relationships between 2 SDOs,
we do not know which one to choose
- In that case, or in the case there is no
default relationship known between 2 SDOs, we
us the `related-to` common relationship instead
of `has`
- As a result, this should fix 33

* [stix2 export] Variable name typo. [Christian Studer]

* [tests] Fixed tests for `country` galaxies export as STIX 2.1 Location objects. [Christian Studer]

* [stix2 export] Better `country` galaxy clusters parsing. [Christian Studer]

- We use the description (capitalised) to define
the `Location` name field of the country, and
the value (lower case) as a description, which
should fix 34

* [stix2 import] Avoiding warnings about empty object attribute values while converting Observable objects to MISP. [Christian Studer]

* [stix2 import] Fixed the unix extension parsing from User Account patterns. [Christian Studer]

* [stix2 import] Fixed recently renamed unix extension mapping. [Christian Studer]

* [stix2 import] Trying to fix a python 3.7 syntax issue for the remaining time it is still supported. [Christian Studer]

- 3.8 and above don't complain with the
`*(generator)` statement

* [stix2 import] Fixed the `email` object parsing. [Christian Studer]

* [tests] Fixed tests for the datetime attribute in STIX 2.0 File objects imported as `lnk` MISP objects. [Christian Studer]

* [tests] Fixed tests for STIX 2.0 File objects imported as `lnk` objects. [Christian Studer]

* [tests] Made the datetime fields in the File object - to be imported as `lnk` object - acceptable for STIX 2. [Christian Studer]

* [tests] Fixed the internal STIX 2.0 test samples for `lnk` object import. [Christian Studer]

* [stix2 import] Fixed wrongly set `self` variable. [Christian Studer]

* [stix2 import] Better separation of exceptions during observable objects parsing. [Christian Studer]

* [stix2 import] Some clean-up. [Christian Studer]

- Including:
- a wrong function name fixed
- a better naming for some SDOs parsing
- some unused methods removed

* [stix2 import] Fixed imports. [Christian Studer]

* [stix2 import] A very quick fix on observable mapping error message. [Christian Studer]

* [stix2 import] Fixed imports. [Christian Studer]

* [stix2 import] Fixed the Email Address observable object parsing. [Christian Studer]

* [stix2 import] Avoiding issue while parsing IP addresses patterns with empty list of attributes mapped. [Christian Studer]

* [stix2 import] Reusing the `object_marking_refs` fields parsing in a function. [Christian Studer]

* [stix2 import] Fixed the Location object parsing. [Christian Studer]

* [stix2 import] Correctly handling issues with observable object mapping. [Christian Studer]

* [stix2 import] Fixed the Location object parsing. [Christian Studer]

* [stix2 import] Fixed the pattern & observable types extraction. [Christian Studer]

* [stix2 import] Fixed the `MarkingDefinition` objects parsing function. [Christian Studer]

* [stix2 import] Made the MISP Attributes dict creation more generic and including the `object_marking_ref` field parsing. [Christian Studer]

* [stix2 import] Avoiding issues with Marking-Definition objects with no `definition_type` field. [Christian Studer]

* [stix2 import] Avoiding issues with Report or Grouping object that has no `name` field. [Christian Studer]

Other

* Wip: [stix2 import] Parsing Network Traffic objects. [Christian Studer]

* Wip: [stix2 import] Simplified the email observable objects parsing. [Christian Studer]

* Wip: [stix2 import] Parsing Observed Data with domain & ip observable objects. [Christian Studer]

* Wip: [stix2 import] Importing Software observable objects with the `software` object template. [Christian Studer]

* Merge pull request 35 from SYNchroACK/fix/naive-timestamp. [Christian Studer]

Fix naive timestamp

* Wip: [stix2 import] Converting `WindowsRegistryKey` objects as `registry-key` & `registry-key-value` objects or `regkey` attributes. [Christian Studer]

* Wip: [stix2 import] Parsing User Account Observable objects. [Christian Studer]

* Wip: [stix2 import] Parsing X509 Certificate Observable objects. [Christian Studer]

- Reusing some stuff that is similar as the x509
pattern parsing

* Wip: [stix2 import] Parsing Process observable objects. [Christian Studer]

* Wip: [stix2 import] Made the Observable objects parsing more generic. [Christian Studer]

- Reducing the amount of variables by putting all
the observable objects in one single dictionary.
Instead of using multiple dictionaries for
different object types, we use one and added
generic selection methods instead

* Wip: [stix2 import] Updated the File & Directory observable objects parsing to better support the references between objects. [Christian Studer]

* Revert "fix: [stix2 import] Trying to fix a python 3.7 syntax issue for the remaining time it is still supported" [Christian Studer]

This reverts commit 556c433557e3fb6ba997ef0b7c1c8dd922d19e64.

* Wip: [stix2 import] Converting `Directory` observable objects as the recently added `directory` object template. [Christian Studer]

- Also fixed the observable objects mapping to
MISP for `lnk` objects import

* Wip: [stix2 import] Simplifying the Observable objects conversion with fewer function calls. [Christian Studer]

* Wip: [stix2 import] Properly handling filtering on multiple observable object types. [Christian Studer]

* Wip: [stix2 import] Yield-ing observable objects instead of returning them in a list. [Christian Studer]

* Wip: [stix2 import] Importing EmailMessage Observable objects. [Christian Studer]

* Wip: [stix2 import] Importing File Observable objects in the case of a single field value imported as MISP Attribute. [Christian Studer]

* Wip: [stix2 import] Better "attribute or object" determination for File observable objects, searching for the `extensions` field. [Christian Studer]

* Wip: [stix2 import] Importing MISP `file` objects from File Observable objects. [Christian Studer]

- Also includes the modification of some parsing
functions that are used for multiple Observable
objects

* Add: [tests] Added tests for the time fields recently added into the `file` object template. [Christian Studer]

* Add: [stix export] Included the handling of the object attribute recently added to the `file` object template. [Christian Studer]

- Namely the object attributes mentioned here are:
- `acces-time`
- `creation-time`
- `modification-time`

* Revert "wip: [stix2 export] Simplified the Galaxies mapping" [Christian Studer]

This reverts commit 76f4e6f58fa332e3b9170a20151aca762df16dca.

* Update README.md. [Alexandre Dulaunoy]

Fix documentation for generated website

* Merge pull request 26 from coolacid/main. [Alexandre Dulaunoy]

Use MISP event UUID for bundle ID

* Use f-strings like elsewhere, check for _misp_event to pass tests. [Jason Kendall]

* Use MISP event UUID for bundle ID. [Jason Kendall]

* Wip: [stix2 import] Better Observable objects parsing. [Christian Studer]

* Wip: [stix2 import] Cleaner UUID sanitation in some cases. [Christian Studer]

* Wip: [stix2 import] Better Observable objects exceptions handling. [Christian Studer]

* Wip: [stix2 import] Quick Observable objects parsing improvement. [Christian Studer]

* Wip: [stix2 import] Cleaner way to handle Observable objects import & supporting a few more observable object types. [Christian Studer]

* Wip: [stix2 export] Simplified the Galaxies mapping. [Christian Studer]

* Wip: [stix2 import] Parsing `domain-name` observable objects and reusing some generic observable objects parsing code. [Christian Studer]

* Wip: [stix2 import] Started parsing external STIX 2 observable objects. [Christian Studer]

* Wip: [stix2 import] Parsing `object_marking_refs` field from several STIX objects to import tags in object attributes. [Christian Studer]

**Full Changelog**: https://github.com/MISP/misp-stix/compare/v2.4.168...v2.4.169

Changes

* [stix2 import] Reintroduced the ability to import MISP Galaxies as `tag_names` [Christian Studer]

- Using most of the features that were removed
with 43a3a8a & 3b178eb, with improvements
- Using a parameter to define whether the related
STIX objects should be imported as tag_names.
They are parsed as MISP Galaxy objects otherwise
- The reason to import tag names only is to have
at least some information validated by MISP
using the tag names which in fact are the galaxy
cluster names, since MISP is not able for now to
handle all the different cases for new Galaxy
Clusters: is it a new clusters or an update to
an existing one?
We'll be able to give MISP the Galaxies and
Clusters in standard MISP JSON format when it is
able to fully handle it

* [misp-stix] Updated some aspects of the command line script. [Christian Studer]

- Some parameters are required now
- Introducing the import & export difference (it
is still export only for now since we will
add the required content in the import function)

* [package] Bumped version. [Christian Studer]

* [submodules] Bumped latest submodule versions. [Christian Studer]

* [poetry] Bumped latest locak file. [Christian Studer]

* [stix2 import] Differenciating galaxies parsing between external and internal STIX 2 content. [Christian Studer]

* [stix2 import] Removed some additional data structure layer on the loaded STIX objects. [Christian Studer]

* [stix2 export] Added a `meta` dictionary field to the Custom Galaxy object. [Christian Studer]

- We can now export the `meta` field from a custom
cluster, as it is, in the related field within
the custom STIX object

* [tests] Updated tests for STIX 2 objects imported as MISP Galaxies. [Christian Studer]

* [tests] Updated the samples of STIX 2 objects that are converted as MISP galaxies. [Christian Studer]

- Added some fields to extend the tests
- Removed the unrelevant `kill_chain_phases` fields

* [stix2 import] Properly parsing the different galaxy & cluster fields. [Christian Studer]

* [tests] MISP galaxy types are now documented from the mapping itself. [Christian Studer]

* [stix2 export] Making the mapping classes reachable. [Christian Studer]

- And in that case for example also the galaxy types

* [tests] Updated tests for internal STIX 2 import to prepare the apparition of tests for external STIX 2 import. [Christian Studer]

* [stix2 export] Enhanced the MISP Galaxies to STIX 2 conversion. [Christian Studer]

- More `meta` fields are now supported
- The STIX 2 `external_references` field now
supports the url refs in addition to the
external IDs which were already supported

* [stix2 export] Extended the MISP Galaxies to STIX 2 mapping. [Christian Studer]

* [documentation] Regenerated documentation with the recent changes on mappings. [Christian Studer]

* [documentation] Updated mapping documentation. [Christian Studer]

* [documentation] Regenerated documentation with the recent changes on mappings. [Christian Studer]

* [documentation] Updated mapping documentation. [Christian Studer]

* [stix2 export] Added missing `person` object to the mapping of MISP objects export as STIX 2.0 & 2.1. [Christian Studer]

- This object template was supposed to be supported
for a while...
- It is then now not exported as custom object as
it was before

* [stix2 export] Added missing `person` object to the mapping of MISP objects export as STIX 2.0 & 2.1. [Christian Studer]

- This object template was supposed to be supported
for a while...
- It is then now not exported as custom object as
it was before

Fix

* [misp-galaxy] Bumped latest version. [Christian Studer]

* [stix2 import] Fixed wrong `_create_cluster_args` parameters in some cases. [Christian Studer]

* [stix2 import] Fixed the tests for `region` galaxies import from STIX 2.1 `Location` objects. [Christian Studer]

* [stix2 import] Fixed the `region` Galaxy Cluster value conversion. [Christian Studer]

- In MISP, the `region` galaxy cluster values use
the actual UN M49 names with the area codes. The
codes were not supported before in the STIX 2 to
MISP conversion

* [stix2 import] Fixed issues with `meta` fields in clusters. [Christian Studer]

- We were not able to know whether a `meta` field
initially contained a `-` or an `_` since we
have to use underscore for STIX 2 fields in any
case. We now have a list of meta fields which
should have a `-` to avoid the related issues

* [stix2 import] Fixed the `meta` fields parsing to avoid issues with some undefined (and unnecessary) meta fields mappings. [Christian Studer]

* [stix2 import] Fixed the `accuracy-radius` object attribute mapping. [Christian Studer]

* [stix2 import] Added missing STIX 2 to MISP mapping. [Christian Studer]

* [stix2 export] Using the STIX objects adding function instead of dealing with the private variable. [Christian Studer]

* [stix2 import] STIX 2 import mapping classes renames for more clarity. [Christian Studer]

* [tests] Fixed the tags test to go with the recent changes on some galaxy test samples. [Christian Studer]

* [tests] Added specific testing methods for clusters meta fields. [Christian Studer]

* [tests] Fixed tests for MISP galaxies export as STIX 2, following the recent updates and improvements on their parsing. [Christian Studer]

* [stix2 export] Fixed the `kill_chain` parsing in clusters meta fields. [Christian Studer]

* [stix2 export] Fixed one of the missing attack-pattern object creation that was missed and still using the previous creation function. [Christian Studer]

* [stix2 export] Removed no longer necessary argument of some STIX 2 object creation function. [Christian Studer]

- Which also made unnecessary some of thoses
functions being no longer specific to galaxies

* [stix2 import] Avoiding Custom Objects converted as Attributes to be modified while they are parsed. [Christian Studer]

* [stix2 import] Removed unused Galaxies parsing case. [Christian Studer]

* [stix2 import] Some pycodestyle clean-up. [Christian Studer]

* [stix2 export] Tiny improvement to avoid unused variable in the case of STIX 2.1 export with no Event report. [Christian Studer]

- And a few long lines cleaned up

* [stix2 import] Making sure we cover all the cases while checking if an attribute UUID is valid. [Christian Studer]

- This fixes the object attributes handling in the
case of MISP objects exported as Custom STIX
objects, with invalid UUIDs which were not
correctly handled when we convert the content
back to MISP format

* [stix2 import] Better invalid UUIDs parsing for Custom STIX objects converted as MISP objects. [Christian Studer]

* [tests] Fixed tests for STIX 2.0 registry-key objects import. [Christian Studer]

* [stix2 import] Fixed some loading definitions. [Christian Studer]

* [stix2 import] Fixed variable that should not be self. [Christian Studer]

* [tests] Simply avoiding issues with the custom galaxies not exported in STIX 1 (for now at least) [Christian Studer]

* [tests] Added tests to make sure custom galaxies are correctly exported when embedded in attributes or object attributes. [Christian Studer]

* [stix2 export] Added the missing custom galaxies handler for attributes galaxies. [Christian Studer]

* [stix2 export] Reverted some try/catch bypass used for debugging purposes. [Christian Studer]

* [stix2 export] Clarification on some incomplete MISP Galaxies typing. [Christian Studer]

* [stix2 export] Quick fix & improvement on the custom galaxies export. [Christian Studer]

* [stix2 export] Simply a quick clean-up. [Christian Studer]

* [stix2 export] Fixing the `EventReport` references handling. [Christian Studer]

- When there is no actual reference to a MISP
attribute, object or galaxy in the Event report,
the `object_refs` field is empty, which is not
allowed, so we add a reference to the report or
grouping to avoid raising an exception

* [stix2 export] Fixing the `EventReport` references handling. [Christian Studer]

- When there is no actual reference to a MISP
attribute, object or galaxy in the Event report,
the `object_refs` field is empty, which is not
allowed, so we add a reference to the report or
grouping to avoid raising an exception

* [tests] Fixed tests for `registry-key` objects export as STIX 2.0 following the recent mapping change on the `last-modified` attribute. [Christian Studer]

* [stix2 export] Removed unused import. [Christian Studer]

* [stix2 export] Fixed the `registry-key` object mapping regarding the `last-modified` attribute export as STIX 2.0. [Christian Studer]

* [tests] Fixed tests for `registry-key` objects export as STIX 2.0 following the recent mapping change on the `last-modified` attribute. [Christian Studer]

* [stix2 export] Removed unused import. [Christian Studer]

* [stix2 export] Fixed the `registry-key` object mapping regarding the `last-modified` attribute export as STIX 2.0. [Christian Studer]

* [stix2 import] Avoiding issues with identifiers in compiled patterns. [Christian Studer]

- When `[*]` is part of a pattern,the related
identifiers contain a non str element which
used to break the related exception handling

* [stix2 import] Fixed the hash types handling while parsing patterns. [Christian Studer]

* [tests] Removed the `person` object from the tests for custom objects export as STIX 1. [Christian Studer]

- Following changes on the `person` object export
and its removal from the tests samples for
custom objects

* [tests] Added tests for `person` objects export as STIX 2 & fixed tests on object references. [Christian Studer]

* [stix2 export] Added missing `ObjectReference` checking for objects exported as STIX 2 Identity objects. [Christian Studer]

* [tests] Removed the `person` object from the tests for custom objects export as STIX 1. [Christian Studer]

- Following changes on the `person` object export
and its removal from the tests samples for
custom objects

* [tests] Added tests for `person` objects export as STIX 2 & fixed tests on object references. [Christian Studer]

* [stix2 export] Added missing `ObjectReference` checking for objects exported as STIX 2 Identity objects. [Christian Studer]

* [stix2 import] Removed unused import. [Christian Studer]

Other

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Wip: [stix import] Enabling the command line use of the library for STIX -> MISP import feature. [Christian Studer]

- Minimal feature with the ability to load STIX
files, and convert each of them to a MISP event

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Wip: [tests] Samples and tests for `country` & `region` galaxies import from external STIX 2.1 `Location` objects. [Christian Studer]

* Wip: [stix2 import] Importing `country` & `region` galaxies from external STIX 2.1 data. [Christian Studer]

* Wip: [tests] Added tests for `country` and `location` galaxies import from STIX 2.1 `Location` objects. [Christian Studer]

* Wip: [stix2 import] Importing `country` & `region` galaxies from STIX 2.1 'internal' `Location` objects. [Christian Studer]

* Add: [tests] Added tests for `country` & `region` galaxies export as STIX 2.1. [Christian Studer]

* Add: [stix2 export] Parsing the `meta` fields from the `country` and `region` galaxy clusters. [Christian Studer]

* Add: [stix2 export] Exporting `country` & `region` galaxies as STIX 2.1 Location objects. [Christian Studer]

* Wip: [stix2 import] Added note for the vulnerability object import from external STIX 2. [Christian Studer]

* Add: [tests] Added some of the common external STIX 2 import content testing. [Christian Studer]

* Add: [tests] Added samples & tests for galaxies import from external STIX 2. [Christian Studer]

* Wip: [tests] Added tests for internal custom galaxy objects import from STIX 2. [Christian Studer]

* Wip: [stix2 import] Parsing internal Custom galaxy objects from STIX 2. [Christian Studer]

* Wip: [stix2 import] Using the MISP Galaxy & Cluster classes to convert STIX objects meant to be galaxy clusters, and no longer using the tag names. [Christian Studer]

* Wip: [stix2 import] Removed the synonyms to tag_names mapping. [Christian Studer]

- We will now use the PyMISP classses to create
galaxies and clusters attached to the related
containers (Event & Attributes)
- The galaxies checking for existing galaxies and
references will be processed in MISP directly

* Wip: [stix2 import] Introducing a new way of parsing content converted into Galaxies. [Christian Studer]

- Still some pieces of the puzzle to add

* Wip: [stix2 import] Handling invalid UUIDs in MISP attributes creation. [Christian Studer]

* Wip: [tests] Added tests for STIX 2 content with invalid UUIDs import. [Christian Studer]

* Wip: [stix2 import] Deeper investigations on invalid UUIDs handling. [Christian Studer]

* Wip: [stix2 import] Handling non RFC UUIDs. [Christian Studer]

* Wip: [stix2 import] A few fixes including the import of Identity classes. [Christian Studer]

* Wip: [stix2 import] Importing generic `identity` objects. [Christian Studer]

* Add: [tests] Added tests for custom Galaxies export as STIX 2.0 & 2.1. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Add: [documentation] Mapping documentation has been updated automatically with the tests for `identity` objects export as STIX 2. [Christian Studer]

* Add: [tests] Tests for `identity` objects export as STIX 2.0 & 2.1. [Christian Studer]

* Add: [stix2 export] Added the `identity` object to the list of supported templates. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Add: [stix export] Handling custom galaxies & galaxy clusters. [Christian Studer]

- The Galaxy clusters export to STIX 1 remains the
same, with some clearer warning messages handling
- Custom clusters within existing galaxies are
exported into the usual existing STIX 2 objects,
and custom galaxies are exported as Custom objects

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge pull request 25 from LM-CT/main. [Alexandre Dulaunoy]

Ignore pycache

* Ignore pycache. [Lucas Cloud Target]

* Add: [documentation] Mapping documentation has been updated automatically with the tests for `identity` objects export as STIX 2. [Christian Studer]

* Add: [tests] Tests for `identity` objects export as STIX 2.0 & 2.1. [Christian Studer]

* Add: [stix2 export] Added the `identity` object to the list of supported templates. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Parse_misp_event takes a dict not a JSON. [Alexandre Dulaunoy]

parse_misp_event takes a dict not a JSON

* Wip: [stix2 import] Parsing more patterns. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Wip: [stix2 import] New Exception type for unmapped pattern types. [Christian Studer]

* Wip: [stix2 import] Importing a few more pattern types. [Christian Studer]

* Wip: [stix2 import] Handling STIX 2 pattern values to remove the additional `'` characters. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Wip: [stix2 import] We start parsing STIX 2 patterns from external files. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Wip: [stix2 import] Moving the pattern parsing to another function specific to STIX patterns (to come next) [Christian Studer]

* Merge branch 'main' into dev. [Christian Studer]

* Fix; [stix2 import] Importing exceptions from the parent directory instead of importing it from the library. [Christian Studer]

* Wip: [stix2 import] Making the STIX 2 pattern parser available to be imported from the library. [Christian Studer]

* Wip: [stix2 import] Making the STIX 2 patterns parser better. [Christian Studer]

Changes

* [package] New version. [Christian Studer]

Fix

* [stix2 export] Avoiding variables to be referenced before they are declared. [Christian Studer]

* [stix2 export] Fixed the Hash values checking. [Christian Studer]

- STIX 2 allows some custom Hash types so we don't
need to consider invalid a hash that is not
in the list of common supported types

* [stix2 export] Some details fixed on errors handling functions. [Christian Studer]

Other

* Fix; [stix2 export] Added missing check for `data` fields from attachment attributes. [Christian Studer]

* Wip: [stix2 export] Checking Hash values for object attributes. [Christian Studer]

* Wip: [stix2 export] More Hash values checking. [Christian Studer]

- We also check now Hash values in the case of a
conversion as Observable objects

* Wip: [stix2 export] Introducing a hash value checking function to avoid issues with invalid hashes. [Christian Studer]

* Wip: [stix2 import] Added some helpers to parse content in STIX 2 patterns. [Christian Studer]

- Loading patterns for now

Changes

* [package] Updated to latest version to publish. [Christian Studer]

* [stix2 export] Returning warning as a dictionary of lists instead of sets. [Christian Studer]

* [setup, poetry] Aligning with the package features that are actually used on pypi. [Christian Studer]

* [tests] Ported all STIX 1 export tests to support both JSON & MISP inputs. [Christian Studer]

* [stix2 export] Made the timestamp values checking common to all export classes and moved the test is the values are datetime to this common function. [Christian Studer]

* [tests] Duplicated tests for attributes, objects & galaxies export as STIX 2 to support both JSON & MISP input. [Christian Studer]

* [tests] Tests for interoperability & feeds now support both JSON and MISP inputs. [Christian Studer]

* [stix2 export] Added correct typing to functions receiving attributes, objects or events. [Christian Studer]

- When the library is used in a python script, we
can pass directly MISPEvent, MISPAttribute or
MISPObject objects instead of their JSON format
It is already working, here we simply fixed the
functions header with the correct typing

* [doc] add PyPI references. [Alexandre Dulaunoy]

Fix

* [readme] Updated description. [Christian Studer]

* [stix2 export] Added missing use case making available Attributes parsing in some situations while giving the input as file instead of as loaded dict. [Christian Studer]

- It avoids for instance issues with the command
line script when giving a file containing an
attributes collection

* [stix2 export] Fixed edge case when the `send-date` attribute within an `email` object is not a correctly formatted datetime value. [Christian Studer]

* [tests] Fixed tests for composite attributes exported as STIX 2 indicator that received a tiny change. [Christian Studer]

* [stix1 export] Fixed composite attribute values parsing to avoid issues with values not formatted the right way. [Christian Studer]

* [stix2 export] Fixed parsing of composite attributes which require some attribute type handling. [Christian Studer]

- The composite attribute type will indeed always
have the standard `|` as separator

* [stix2 export] Handling composite attribute values when they are not formatted as they should be with a `|` [Christian Studer]

* [stix2 export] Added the missing `interoperability` parameter in the Relationship object arguments. [Christian Studer]

* [stix2 export] Fixed `annotation` object export as STIX 2.1 when there is no object reference. [Christian Studer]

* [clean up] Removed debugging print statements. [Christian Studer]

* [tests] Making the datetime to str utility function common to all STIX testing classes. [Christian Studer]

* [stix1 export] Handling the `data` field while creating an Artifact object. [Christian Studer]

* [stix1 export] Handling some datetime values. [Christian Studer]

* [documentation] Fixed documentation following changes on the lnk objects export to STIX 2.0. [Christian Studer]

* [tests] Fixing some tests triggered by a lot of unit tests to make them work with a MISP input. [Christian Studer]

* [tests] Avoiding issues with the geolocation object & the `to_ids` value of some asn object attributes. [Christian Studer]

* [stix2 export] Added missing import. [Christian Studer]

* [tests] Better handling of timeline value & the `data` field. [Christian Studer]

* [stix2 export] Better `lnk` objects parsing including the timeline attributes export as STIX 2.0 that were missing. [Christian Studer]

* [stix2 export] Correctly handling the timestamp fields and values. [Christian Studer]

* [stix2 export] Handling properly `data` fields in attributes and object attributes. [Christian Studer]

* [stix2 export] Handling some timestamp values depending whether they are datetime or str. [Christian Studer]

* [requirements] Regenerated the requirements files. [Christian Studer]

* [requirements] Fixed requirements regarding the STIX 2 dependency. [Christian Studer]

* [stix2 export] Fixed timestamp handling when they are already datetime. [Christian Studer]

- Happens if we give the STIX Parser a MISPEvent
type input instead of the JSON format that is
the standard case when used in MISP core

* [stix2 export] Fixed pattern validation to avoid sanitisation for strings to be executed on non string values. [Christian Studer]

* [stix2 export] Fixed custom objects parsing for standalone `pe-section` objects parsing. [Christian Studer]

Other

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

Changes

* [package] Updated library version. [Christian Studer]

* [package] Updated some setup information. [Christian Studer]

* [readme] Updated instructions for pip install. [Christian Studer]

* [poetry] Bumped latest lock file. [Christian Studer]

* [poetry] Added a few information and using the stix2 library package instead of the git dependency. [Christian Studer]

* [poetry] Bumped latest lock file. [Christian Studer]

* [poetry] Updated pyproject file. [Christian Studer]

Fix

* [package] Fixed setup. [Christian Studer]

* [stix2 import] Fixed a couple typo issues. [Christian Studer]

* [poetry] Bumped latest lock file. [Christian Studer]

* [stix2 import] Added missing import. [Christian Studer]

* [stix2 import] Fixed the `add_attribute` method that was missing the `**` prefix that is required when you pass a dict directly to it. [Christian Studer]

Other

* Merge pull request 21 from netantho/patch-1. [Christian Studer]

Add setuptools as a build-system dependency

* Add setuptools as a build-system dependency. [Anthony VEREZ]

* Wip: [stix2 import] Better handling of external references from `attack-pattern` objects. [Christian Studer]

- Instead of having a common parsing function for
all STIX 2 attack pattern external references,
we parse those references depending on whether
it is external STIX data or not, to have 1 very
specific parsing function for content we know,
and a more flexible one for external content in
order to avoid issues with that kind of data

Alongside with [MISP latest release](https://github.com/MISP/MISP/commits/v2.4.160), we are pleased to announce that misp-stix comes with a few improvements that are available and used on MISP.

This release also includes a few new features, changes and improvements on the library itself.

Changes immediately available on MISP

The MISP objects conversion mapping to STIX 2 has been updated to support the following templates:
- http-request
- netflow
- sigma (only STIX 2.1, which supports multiple patterning languages such as yara or suricata that are already included)

With the implementation of the conversion for these object templates, we also added the related tests and updated the [documentation](https://github.com/MISP/misp-stix/tree/main/documentation#readme).

Improvements on the STIX -> MISP import feature

The STIX 2 -> MISP import feature has been substantially improved to complete the support of STIX content that is produced with this library.
Then we should now have a STIX 2 -> MISP mapping similar to the MISP -> STIX 2 one and be able to import back to MISP what has been exported as STIX 2.
(documentation will be also available soon)

(WiP) The conversion of STIX 2 content from external sources has been improved and now supports most of the SDOs.
There is nonetheless a not negligible amount of work needed to "fully" support the conversion of STIX patterns and Cyber Observable objects into the appropriate MISP data structure (Attribute, Objects, ...). Soon we will rework and improve the mapping for these STIX features so STIX -> MISP import feature can be used on MISP and replace the old built-in code :crossed_fingers:

Additional features

Single MISP attributes parsing & incremental conversion

A `parse_misp_attribute` method has been added to handle the conversion to STIX of single MISP attributes (this feature is different from the already implemented `parse_misp_attributes` method that is used to convert MISP Attributes collections).

Alongside with the ability to parse single attributes independently, we improved the ability to parse MISP data incrementally and fetch the conversion results.
As a result, we can now use the main parsing functions that handle MISP data as many times as needed and store the converted STIX data in one single Bundle more easily than before.
For example:
python
from misp_stix_converter import MISPtoSTIX21Parser
parser21 = MISPtoSTIX21Parser()
for event in whatever_process_returning_MISP_events():
parser.parse_misp_event(event)

The STIX objects are available then with:
python
parser.stix_objects if you want to simply look the list of objects
OR
parser.fetch_stix_objects to extract the STIX objects you just generated from the conversion of MISP events

If you want to get those objects within a fancy STIX Bundle:
python
parser.bundle extracts the STIX objects like `fetch_stix_object` and puts them in a STIX Bundle

This feature works with all the supported MISP data structures conversion (Events, Attributes, ...) and does not interfere with the collections handling features that do the same work for you in a single callable function.

This feature has been initiated from a request in 16 by mavam

---
**Changelog** available here: https://github.com/MISP/misp-stix/commits/v2.4.160