Misp-stix

Changes

* [poetry] Bumped lock file with latest dependencies version. [Christian Studer]

* [package] Bumped new version. [Christian Studer]

* [stix2 import] Changed the relationships content storing. [Christian Studer]

- We do not need a dictionary with keys defining
which value is the referenced uuid or the
relationship type, as a tuple with the sorted 2
values makes the job
- It also allows us to use a set to store the
references to avoid storing multiple times the
same relationship to the same target
- Both previous points will help handling the
opposite relationships

* [stix2 import] More accurate relationship type between a sample and the malware it is the sample of. [Christian Studer]

* [poetry] Bumped latest pymisp version. [Christian Studer]

Fix

* [tests] Quick fix on embedded galaxies in attributes tests, as the opposite references handling creates uncertainty in relationships order. [Christian Studer]

* [stix2 import] Handling opposite relationships. [Christian Studer]

- This is usefull for instance when an Indicator
is imported to MISP as an Attribute, and has a
relationship with another SDO imported as a MISP
Object, in which case the relationship used to
be lost because for now, an attribute does not
support references in MISP.
Now we use the opposite reference to keep the
link between the converted MISP Object and
Attribute

* [stix2 import] Added missing relationship parsing. [Christian Studer]

- References between MISP objects and attribute or
object were handled only when the Galaxies are
parsed in their MISP standard format form. They
were missing when Galaxies are imported as tag
names, which shouldn't change object references

* [stix2 import] Reusing code which removed also a typo. [Christian Studer]

* [stix2 import] Added missing `continue` to avoid additional handling for observable objects already handled. [Christian Studer]

* [stix2 export] Remove attack pattern ID from name attribute. [Tomas Lima]

Other

* Add: [poetry] Added `stix-edh` dependency for STIX 1 Markings. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'parser_feature' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'parser_feature' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'main' of github.com:misp/misp-stix into parser_feature. [Christian Studer]

* Wip: [stix2 import] Handling standalone Observable objects. [Christian Studer]

- We started changing the Observable objects converters
in order to start parsing those which are standalone
and not referenced by SDOs
- A lot more Observable object types to be added

* Merge branch 'main' of github.com:misp/misp-stix into parser_feature. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge pull request 38 from SYNchroACK/fix/attack-pattern-name. [Alexandre Dulaunoy]

Remove attack pattern ID from name attribute

What's Changed
* Remove attack pattern ID from name attribute by SYNchroACK in https://github.com/MISP/misp-stix/pull/38


**Full Changelog**: https://github.com/MISP/misp-stix/compare/v2.4.175...v2.4.176

Changes

* [poetry] Updated lock file. [Christian Studer]

* [poetry] Updated lock file. [Christian Studer]

* [package] Set new version. [Christian Studer]

* [__init__] Clearer classes & methods import as well as `noqa` added to imports. [Christian Studer]

* [poetry] Bumped latest lock file. [Christian Studer]

* [package] Bumped version (& pymisp) [Christian Studer]

Fix

* [workflow] Testing both internal & external STIX content to import to MISP. [Christian Studer]

* [tests] Fixed test samples for external Malware objects converted as Galaxies. [Christian Studer]

* [stix2 import] Some clean-up - Removed unused stuff & Added missing stuff. [Christian Studer]

* [stix2 import] Fixed failing message. [Christian Studer]

* [stix2 import] Some pycodestyle clean-up. [Christian Studer]

* [stix2 import] A few typing and unused methods fixed. [Christian Studer]

* [stix2 import] Fixed debugging messages handling in the command-line feature. [Christian Studer]

* [stix2 import] Removed unused UUID extraction method & made the method to populate object attributes common to all converters. [Christian Studer]

* [stix2 import] Fixed reverse malware handling depending on the `is_family` flag. [Christian Studer]

* [stix2 import] Added the missing object attributes populating method. [Christian Studer]

* [stix2 import] Removed the UUID handling methods in the parsers directory to keep using the original ones from `importparser` as a MISP event also need some of those methods. [Christian Studer]

* [tests] Fixed STIX 2.0 test method names. [Christian Studer]

* [stix2 export] Some more pycodestyle to make the mapping cleaner. [Christian Studer]

* [stix2 import] Some quick pycodestyle to make the mapping cleaner. [Christian Studer]

* [stix2 import] Fixed debugging messages handling in the command-line feature. [Christian Studer]

Other

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'parser_feature' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'main' of github.com:misp/misp-stix into parser_feature. [Christian Studer]

* Merge branch 'dev' of github.com:misp/misp-stix into parser_feature. [Christian Studer]

* Wip: [stix2 import] Properly handling Observable. [Christian Studer]

- We moved the InternalSTIX2toMISPParser back to
its previous state regarding observable objects
handling because we do not generate standalone
observable objects with the MISP to STIX feature
- We fixed some bad observable handling in the
External parser to avoid issues with the `used`
flag which was not handled correctly in some
cases

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'parser_feature' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Fix; [stix2 import] Avoiding issues with missing `time` import. [Christian Studer]

* Merge branch 'parser_feature' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'parser_feature' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Wip: [stix2 import] Better Observable objects handling. [Christian Studer]

- Objects referenced by malware & malware-analysis
SDOs are now handled with no duplication issue

* Wip: [stix2 import] Better parsing for some malware-analysis reference fields. [Christian Studer]

* Wip: [stix2 import] Storing observable objects differently. [Christian Studer]

- Preparing for their parsing as standalone
objects or with multiple references from
different SDOs to the same Observable

* Fix; [stix2 import] Fixed Malware conversion as MISP Object. [Christian Studer]

- In the case we do not need to return the converted
MISP objects, we should not yield the objects,
as an iterator needs to be consumed, which we
do only when we convert the Malware as a Galaxy
Cluster too and add it to the appropriate
attributes of the MISP object

* Merge branch 'main' of github.com:misp/misp-stix into parser_feature. [Christian Studer]

* Wip: [stix2 import] Handling the Malware Analysis objects in the main parsing classes. [Christian Studer]

* Wip: [stix2 import] Parsing & Converting STIX 2.1 Malware Analysis objects. [Christian Studer]

- We need to add the parsing mechanisms in the
main parsers
- Some more love is required to handle some of the
fields referenced by the malware analysis object

* Merge branch 'main' of github.com:misp/misp-stix into parser_feature. [Christian Studer]

* Wip: [stix2 import] Parsing specific cases where a STIX 2 Malware object is converted as both an object and a galaxy. [Christian Studer]

- We're adding the galaxy to the attributes with
an ids flag in all the MISP objects that are
generated from the conversion of the Malware
object - there are sometimes software, file or
artifact objects too coming from the different
references the Malware object has

* Wip: [stix2 import] Added missing galaxy as tag names parsing methods & properly handling the galaxy conversion case. [Christian Studer]

* Wip: [stix2 import] Added pluggable Observable objects conversion class to handle observable objects references by malware objects. [Christian Studer]

* Wip: [stix2 import] Calling the already existing converters. [Christian Studer]

- We keep the parsing methods in the parser scripts
as they are for now, in order to avoid breaking
the whole parsing mechanism for the other STIX
objects which conversion methods are not
implemented in the conversion directory yet

* Wip: [stix2 import] Properly converting STIX 2.1 Malware objects. [Christian Studer]

* Wip: [stix2 import] Clarified class names, script names, and improved malware objects parsing. [Christian Studer]

* Wip: [stix2 import] Converting `script` objects from STIX 2 Malware objects. [Christian Studer]

* Fix; [stix2 import] Fixed Malware galaxies meta fields parsing. [Christian Studer]

* Wip: [stix2 import] Porting the conversion capacity with the mappings into the parsers sub-directory. [Christian Studer]

* Wip: [stix2 import] Externalising conversion capacity to specific parsers. [Christian Studer]

- Starting with Attack Pattern & Malware (WiP) objects

**Full Changelog**: https://github.com/MISP/misp-stix/compare/v2.4.174...v2.4.175

Changes

* [poetry] Bumped lock file. [Christian Studer]

* [stix import] Simplified data path. [Christian Studer]

* [tests] Updated tests for sightings import. [Christian Studer]

- Also changed some sample to have different order
with Identity objects in order to test properly
our recent changes on loading and converting the
sightings, which purpose was to avoid issues
with the STIX objects order

* [stix2 import] Updated the External STIX 2 Identity objects mapping to MISP `organization` objects following recent updates on that template. [Christian Studer]

* [readme] Updated MISP collections to STIX 1 export example. [Christian Studer]

* [readme] Updated usage documentation following recent changes on the command-line feature and some helper methods. [Christian Studer]

* [poetry] Bumped latest dependencies. [Christian Studer]

Fix

* [stix2 import] Differenciating between internal and external stix content regarding the external references handling. [Christian Studer]

* [tests] Fixed tests for vulnerability cluster import following recent changes on the meta fields. [Christian Studer]

* [stix2 import] Fixed vulnerability cluster meta fields parsing. [Christian Studer]

* [tests] Fixed the vulnerability clusters meta tests according to the latest changes on the external id (cve) field parsing. [Christian Studer]

* [stix2 export] Fixed vulnerability clusters meta fields parsing. [Christian Studer]

* [stix2 export] Fixed `malware_types` fields & added missing method for `threat_actor_types` parsing. [Christian Studer]

* [stix2 import] Added missing `annotation` object metadata parsing. [Christian Studer]

* [tests] Removed some results writing in files which were used at some point for debugging purposes and forgotten in the code vastness. [Christian Studer]

* [stix2 import] Shorter obervable types extraction while still including the recent fix to avoid issues with observables that are of `dict` type. [Christian Studer]

* [stix1 export] Fixed backward compatibility with old object templates. [Christian Studer]

* [stix2 export] Fixed backward compatibility with old object templates. [Christian Studer]

* [stix import] Sanitised the import variables declaration to avoid issue with wrong value format. [Christian Studer]

* [command-line] Fixed results message. [Christian Studer]

* Observable type access for dict type. [Sura De Silva]

* [stix2 import] Typo within the Opinion objects loading method. [Christian Studer]

* [stix2 import] Better handling of MISP Sightings import. [Christian Studer]

- Storing `Sighting` & `Opinion` objects instead
of converting them to MISP Sightings while
loading them, because in some cases we need the
information of the related org, which is not
always already loaded when the need its info
- We convert the STIX objects to Sightings at the
end while we loop over the different references

* [stix2 import] Better `Identity` object's identity class field handling. [Christian Studer]

* [stix2 import] Using the Galaxy Cluster adding method to add cluster instead of appending it the the list of clusters. [Christian Studer]

Other

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:misp/misp-stix. [Christian Studer]

* Merge pull request 45 from SYNchroACK/fix/wrong-import. [Christian Studer]

Fix wrong stix observables import

* Fix wrong stix observables import. [Tomas Lima]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Wip: [stix2 import] Updated the import conversion of internal STIX 2.x Identity objects to better support recent changes on the `organization` template. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Merge pull request 46 from dragsu/fix-observable-access-dict-type. [Christian Studer]

fix: `type` access for dict type Observables

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Add: [stix2 import] Importing Identity objects with `identity_class` set to organization as `organization` object. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Add: [stix2 import] Adding relations between galaxy clusters. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

Contributions
* fix: `type` access for dict type Observables by dragsu in https://github.com/MISP/misp-stix/pull/46
* Fix wrong stix observables import by SYNchroACK in https://github.com/MISP/misp-stix/pull/45

New Contributors
* dragsu made their first contribution in https://github.com/MISP/misp-stix/pull/46

**Full Changelog**: https://github.com/MISP/misp-stix/compare/v2.4.172...v2.4.174

Changes

* [poetry] Updated dependencies. [Christian Studer]

* [package] Bumped version. [Christian Studer]

* [misp_stix_converter] Changes on the command line feature. [Christian Studer]

- Cleaner separation between the 2 main features,
export & import, as well as cleaner arguments
in general
- Better handling of the messages returned by the
helper methods that are call by the command
line feature

* [tests] Updated STIX 1 export sample result files. [Christian Studer]

* [tests] Added new tests and changes on the collections export as STIX 2. [Christian Studer]

* [stix2 import] Added the `debug` argument to the `stix_2_to_misp` helper. [Christian Studer]

- We return the error and warning messages only
when the `debug` flag is set

* [stix2 import] Added more result details and arguments to the `stix_2_to_misp` helper that converts a STIX file to MISP format. [Christian Studer]

- We added all the arguments needed in both the
declaration of the STIX 2 to MISP parser and the
stix bundle parsing call
- We have a more detailed return message that
gives not only a success message, but also the
errors and warnings

* [stix1 export] We don't instantiate the MISP to STIX 1 Mappings anymore and use the classmethods directly. [Christian Studer]

* [stix1 export] Turned the MISP to STIX 1 Mapping properties into classmethods and made it usable in an uninstantiated way. [Christian Studer]

* [stix2 export] Using the uninstantiated MISP to STIX 2 mappings classmethods. [Christian Studer]

* [stix2 export] Converted STIX 2 Mappings methods into classmethods. [Christian Studer]

* [stix2 import] Improved the internal STIX 2 to MISP mapping. [Christian Studer]

- The `InternalSTIX2toMISPMapping` class is no
longer instantiated
- We're using the different classmethod helpers
- The mapping is then a bit cleaner than before

* [stix2 import] Internal STIX 2 to MISP mapping improved. [Christian Studer]

- Changes on the pattern & observable objects
mapping names
- Reusing mappings that are contained in other ones

* [stix2 import] Changed mapping to not be forced to instantiate them. [Christian Studer]

* [stix export & import] Made the parent parser classes abstract. [Christian Studer]

- As the children classes should be called anyways

* [poetry] Changed pymisp dependency back to the pypi version. [Christian Studer]

* [misp-galaxy] Bumped latest version. [Christian Studer]

* [package] Latest version aligned with MISP. [Christian Studer]

* [poetry] Updated dependencies. [Christian Studer]

* [misp-galaxy] Bumped latest version. [Christian Studer]

* [stix2 import] Changed the `Marking Definition` loading process. [Christian Studer]

Fix

* [import] added missing import. [iglocska]

* [tests] Removed unused imports. [Christian Studer]

* [tests] Fixed STIX 1 export result samples. [Christian Studer]

* [misp_stix_converter] Fixed helpers import - using the method names recently changed. [Christian Studer]

* [stix export] Fixed arguments to give from the command line feature to the STIX export helpers. [Christian Studer]

* [stix2 export] Fixed footer for collections export as STIX 2. [Christian Studer]

* [tests] Updated tests for STIX 1 export helpers. [Christian Studer]

* [stix1 export] Fixed Package header writting for methods used to replicate the MISP pagination - used with collections export helpers. [Christian Studer]

* [stix1 export] Reusing methods from the framing to generate packages (& handling namespaces) [Christian Studer]

* [stix1 export] Handling cases when there is no STIX header. [Christian Studer]

- In this specific case, the STIX package in XML
format is a single xml tag with the included
`/` closing character... so we remove it
- ( JSON >>>>> XML definitely :) )

* [stix1 export] Added option to generate a Package with no header. [Christian Studer]

* [stix1 export] Fixed the creation process of the STIX package used to serve as container for related packages. [Christian Studer]

* [stix export] Made STIX framing methods more modular. [Christian Studer]

* [stix2 export] Returning the result files in a traceback message as list. [Christian Studer]

* [stix2 export] Fixed some statements in the MISP collections export to STIX 2 helper. [Christian Studer]

- Including fixes on:
- the single file handling (regarding the single
file name)
- the default directory for collections export
results
- the input files argument of the function

* [stix1 export] Fixed arguments passed to the MISP collections export to STIX 1. [Christian Studer]

* [stix1 export] Added a use case to support the use of the events collection export even with a single file. [Christian Studer]

* [stix1 export] Fixed name for the result STIX 1 event collections export & added a missing traceback. [Christian Studer]

* [stix1 export] Making sure we avoid exceptions with the fails catching on traceback messages. [Christian Studer]

* [stix2 import] Better handling of the `single_event` variable inside of the STIX 2 to MISP parser. [Christian Studer]

* [stix2 import] Fixed external STIX 2 `email-message` observable & pattern mapping. [Christian Studer]

* [stix2 import] Added missing `campaign` type in the list of STIX object types to look for. [Christian Studer]

* [stix2 import] Fixed the observable registry key values parsing in case of a single key imported as `regkey|value` attribute. [Christian Studer]

* [stix2 import] Catching parsing issues that appear while the STIX file is loaded. [Christian Studer]

* [stix export] Galaxies mapping are now also using the uninstantiated mapping classmethods. [Christian Studer]

* [tests] Using the uninstantiated mapping classes with their classmethods. [Christian Studer]

* [stix2 import] Fixed the `from_misp` test that defines whether a STIX file has been generated with the MISP to STIX conversion feature or not. [Christian Studer]

* [stix2 import] Fixed the email or IP address observable objects from internal STIX content parsing. [Christian Studer]

- Could fail previously with some content generated
from a previous version of the MISP to STIX
conversion feature

* [stix2 import] Fixed marking definition parsing, as we store the tag and not the marking definition object. [Christian Studer]

* [tests] Fixed tests to avoid issues with STIX 2 to MISP mappings, following the recent changes on them. [Christian Studer]

* [stix2 import] Revert change to fix the pattern assertion operator check. [Christian Studer]

- Revert of a part of the code that was staged for
a previous commit while it should not have been
- For now the pattern assertion check will remain
as is even tough there is an ongoing work to
improve it.

* [stix2 import] Fixed missing variable name change. [Christian Studer]

* [stix2 import] Using non instantiated external STIX 2 to MISP mapping. [Christian Studer]

- Same changes as for the internal mapping

* [stix2 import] Removed unused variables & mapping fields. [Christian Studer]

* [stix2 import] Properly transformed the external STIX 2 to MISP mapping methods into classmethods. [Christian Studer]

- Followed the model used in the internal mapping
to have pattern mappings that are waiting for a
field to return the associate value in the
mapping, or observable object mappings that we
loop on in order to check each field

* [stix2 import] Removed unused mapping method. [Christian Studer]

* [stix2 import] Removed unused imports. [Christian Studer]

* [stix2 import] Fixed some mapping dictionary names. [Christian Studer]

* [stix2 export] Fixed fail on copy pasting the generic galaxy mapping update for STIX 2.0. [Christian Studer]

* [stix2 export] Parsing `stix2-pattern` objects. [Christian Studer]

- As they were missing in the export mapping, they
were exported as custom objects, but we simply
have to take the pattern and export it as is,
like we do for sigma or yara patterns for
instance in STIX 2.1
- In this case, it applies to both STIX 2.0 & 2.1

* [stix2 export] Made the `created` & `modified` fields in custom galaxy objects optional. [Christian Studer]

* [stix2 export] Using the property for `identity_id` instead of the 'private' variable. [Christian Studer]

* [stix2 export] Same as the previous commit, for standalone attributes from feeds. [Christian Studer]

* [stix2 export] Fixed the orgc parsing for attributes collections. [Christian Studer]

- The `created_by_ref` values were missing on all
objects because the statement used to wait for
a value where the recent changes made the
related method return nothing anymore

* [stix2 export] Better Orgc & info handling for instance when they are empty. [Christian Studer]

* [stix2 export] Avoiding issues with unset `timestamp` value in MISP Event. [Christian Studer]

* [stix2 export] Checking `Orgc` fields before trying to generate the Identity object which will be used as `created_by_ref` object reference. [Christian Studer]

* [stix2 import, tests] Fixed the galaxy & cluster version. [Christian Studer]

- Forgot that `strip` works only at the beginning
and the end of the string............

* [tests] Removed unused import. [Christian Studer]

* [stix2 import] To avoid any possible issue in MISP with float version, we just made the generic Galaxies & Clusters version int. [Christian Studer]

* [tests] Fixed Galaxies & Clusters tests following all the recent changes on generic conversion from STIX 2.0 & 2.1. [Christian Studer]

* [stix2 import] Fixed the galaxy creation method for external STIX content to avoid issues with `region` and `country` galaxies. [Christian Studer]

* [stix2 import] Fixed the clusters creation method to avoid issues with unassigned cluster value. [Christian Studer]

* [stix2 import] Added missing `self` param in the clusters creation method. [Christian Studer]

* [stix2 import] Syntax fixed in f-string. [Christian Studer]

* [stix2 import] The Galaxy args creation is better and handles some of the formerly missing required field to validate a Galaxy in MISP. [Christian Studer]

* [stix2 import] Quick improvement on a `hasattr` that can be directly replaced by a `getattr` with a default value. [Christian Studer]

* [stix2 import] Fixed the generic info method. [Christian Studer]

- The way it is implemented, it has to be a
property rather than a classmethod in order to
avoid the info field to be null because as a
classmethod, the returned value was a bound
method

Other

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Merge pull request 42 from MISP/dev. [Christian Studer]

A few changes and improvement

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Fix; [stix export] Fixed fail messages as the command line feature wants lists. [Christian Studer]

* Fix; [stix1 export] Fixed the input files argument for the collections export as STIX 1 helpers. [Christian Studer]

* Wip: [stix2 import] Enhanced STIX 2 import helper. [Christian Studer]

* Wip: [stix2 export] Enhanced STIX 2 export helpers. [Christian Studer]

* Wip: [stix1 export] Enhanced the STIX 1 export helper features. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Add: [stix2 export] Added the generic galaxy types to the galaxies export mapping for STIX 2.0 & 2.1. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Add: [stix2 import] Added the `sharing_group_id` field to add this value when the `distribution` level for the event is 4. [Christian Studer]

* Add: [tests] Quickly testing default distribution on events. [Christian Studer]

* Wip: [stix2 import] Adding the MISP Event `distribution` field to the events we generate as result of the conversion from STIX. [Christian Studer]

- For now implemented for STIX 2

* Wip: [stix2 import] Added `namespace` and `icon` value for the Generic galaxies converted from external STIX objects. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Add: [tests] Added unit tests for generic galaxies & clusters - uuids & version are tested. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Wip: [stix2 import] Better Galaxy Clusters creation to include some of the fields required for MISP to validate clusters. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

What's Changed
* A few changes and improvement by chrisr3d in https://github.com/MISP/misp-stix/pull/42


**Full Changelog**: https://github.com/MISP/misp-stix/compare/v2.4.170...v2.4.172

Changes

* [misp_stix_converter] Added quick comments & made the `_from_misp` utility available to import from the library. [Christian Studer]

* [misp_stix_converter] Moved the command line feature to `misp_stix_converter.py` to avoid all the related utility functions to be exposed while importing the python library. [Christian Studer]

* [stix2 import] Using the `from_dict` method as much as possible to populate the different MISP Object or Event fields. [Christian Studer]

- It introduces some changes on the format of the
datetime fields which are now properly defined
as datetime with the right format and the
timezone info

* [stix2 import] Extracted the object case handling to make it callable. [Christian Studer]

* [stix2 import] Better STIX objects as Galaxy import handling. [Christian Studer]

- Instead of testing if we have to import the
tag names or the full Galaxy object each time
we parse a single STIX object, we set a variable
from the beginning to redirect to the related
parsing function

Fix

* [stix2 export] Export the `source` of a sighting as `x_misp_source` as defined in the Custom STIX 2.0 object. [Christian Studer]

- Fixes 28

* [stix2 import] Fixed Galaxy parsing as tag names variable typo. [Christian Studer]

* [misp_stix_converter] Removed unused import. [Christian Studer]

* [misp_stix_converter] Better output names handling. [Christian Studer]

* [misp_stix_converter] Some clean-up. [Christian Studer]

* [stix2 import] Added the missing `entrypoin-address` attribute. [Christian Studer]

* [stix2 import] Making sure we won't have MISP objects rejected for having the same UUID. [Christian Studer]

- `pe` & `pe-section` objects are converted from
the same observable object or pattern as the
`file` object that contains them.
If we create the different MISP objects the same
way we do for the file, they will all have the
same UUID and MISP will reject them

* [tests] Updated tests to handle the recent changes on the datetime values format. [Christian Studer]

* [tests] Fixed tests for internal file with pe & sections objects following recent changes on the related parsing functions. [Christian Studer]

* [stix2 import] Fixed `_add_misp_attribute` function called names. [Christian Studer]

* [stix2 import] Updated the `process` object attributes used to force the MISP content being an object to align with the `requiredOneOf` field of the template. [Christian Studer]

* [stix2 import] Fixed STIX 2 Observable objects to MISP mapping for `Domain Name` with `Network Traffic` objects. [Christian Studer]

* [stix2 import] Fixed wrong object attribute mapping. [Christian Studer]

- The PID attribute is not part of the `Registry Key`
object mapping but `Process`

* [stix2 import] Cleaner `unknown pattern mapping warning` handling. [Christian Studer]

* [stix2 import] Quick clean-up on the error & warning messages handling. [Christian Studer]

* [stix2 import] Quick clean-up. [Christian Studer]

* [stix2 import] Fixed the `x509` import from pattern parsing. [Christian Studer]

* [stix2 import] Fixed the `Identity` object parsing. [Christian Studer]

* [tests] Added the missing `sector` galaxy checking function. [Christian Studer]

* [stix2 import] Fixed the internal STIX 2 objects conversion as MISP Galaxy. [Christian Studer]

- We have to check whether the `description` field
does contain the `|` as separation caracter,
because it is not the case for internal
`Identity` objects with the `identity_class`
field set to 'class' imported as `sector` galaxy

* [tests] Fixed the galaxies export tests to avoid issues with potential missing `description` & `meta` fields within the cluster definition. [Christian Studer]

* [stix2 export] Fixed the `sector` galaxy parsing to avoid issues with the `description` field within the galaxy cluster definition. [Christian Studer]

* [stix2 export] Making the sector galaxy export available for both STIX 2.0 & 2.1. [Christian Studer]

Other

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Wip: [stix2 import] Better `File` and its pe extensions parsing from patterns. [Christian Studer]

* Wip: [stix2 import] Simplified the patterns mapping. [Christian Studer]

* Wip: [stix2 import] Importing directory objects from stix patterns. [Christian Studer]

* Wip: [stix2 import] Network socket parsing improved. [Christian Studer]

* Wip: [stix2 import] Parsing PE optional headers. [Christian Studer]

- Currently only the entry point address supported

* Wip: [stix2 import] Using `from_dict` to update MISPObjects instead of `update` [Christian Studer]

* Wip: [stix2 import] Improved the `Network Traffic` pattern parsing. [Christian Studer]

* Wip: [stix2 import] Replaced more dict `update` by dict merge. [Christian Studer]

* Wip: [stix2 import] Better and more generic Attributes & Objects add handling. [Christian Studer]

- The `confidence` and `object_marking_refs` STIX
fields are properly handled in one place and
added as single Attribute or each object
Attribute tags

* Wip: [stix2 import] Better attributes dictionaries creation. [Christian Studer]

* Wip: [stix2 import] Added `ip-src` & `ip-dst` attribute definition to be reused in different places. [Christian Studer]

* Wip: [stix2 import] Importing Software objects with the `software` object template. [Christian Studer]

* Wip: [stix2 import] Importing `user-account` objects from STIX 2 User Account objects. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Wip: [stix2 import] Converting `Mutex` patterns into `mutex` objects. [Christian Studer]

* Wip: [stix2 import] Handling the exceptions with non existing protocols. [Christian Studer]

* Wip: [stix2 import] Converting `network-traffic` pattern values into `network-connection` objects. [Christian Studer]

- Need to handle the `src` & `dst` refs

* Wip: [stix2 import] Converting pattern with `autonomous-system` values as `asn` object. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Wip: [stix2 import] Better import case handling. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge branch 'dev' of github.com:MISP/misp-stix. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix. [Christian Studer]

* Wip: [documentation] Auto-generated documetation for `sector` galaxies export. [Christian Studer]

* Wip: [stix2 import] Importing `sector` Galaxies from external `Identity` objects with `identity_class` field set to 'class' [Christian Studer]

* Wip: [tests] Tests for STIX 2 Identity objects converson as `sector` galaxies. [Christian Studer]

* Wip: [stix2 import] STIX 2 `Identity` objects conversion as `sector` Galaxy import. [Christian Studer]

* Add: [tests] Tests for `sector` galaxies export to STIX 2.0 & 2.1. [Christian Studer]

* Merge branch 'main' of github.com:MISP/misp-stix into dev. [Christian Studer]

* Merge pull request 36 from SYNchroACK/new/sectors-galaxy. [Christian Studer]

Handle sectors galaxy

* Add: [stix2 export] Handle sectors galaxy. [Tomas Lima]

What's Changed
* Handle sectors galaxy by SYNchroACK in https://github.com/MISP/misp-stix/pull/36


**Full Changelog**: https://github.com/MISP/misp-stix/compare/v2.4.169...v2.4.170

Overview

- Introducing the first version of a more generic way of parsing Observable objects from different sources.
- as the maintenance and extension of the Observable objects conversion to MISP mapping is a continuous task, some content might be missed in the conversion. Feel free to report any issue using the github issues system.
- Merged pull requests:
- Use MISP event UUID for bundle ID by coolacid in 26
- Fix naive timestamp by SYNchroACK in 35
- Work in Progress to be released soon: STIX 2.0 & 2.1 patterning expressions parsing

Detailed changelog

Changes

* [poetry] Bumped latest dependencies versions. [Christian Studer]

* [package] Updated poetry & pymisp requirement. [Christian Studer]

- In order to better support git dependencies, we
updated poetry because it is required in order
to use git dependencies.
- With the change on poetry we can then use the
git dependency for pymisp - for now

* [package] Bumped version. [Christian Studer]

* [submodules] Bumped latest versions. [Christian Studer]

* [documentation] Regenerated the documentation. [Christian Studer]

* [stix2 export] A quick reuse of an existing SDO creation function. [Christian Studer]

* [documentation] Regenerated the MISP -> STIX documentation with the recent mapping updates. [Christian Studer]

* [documentation] Updated mapping documentation following some recent changes. [Christian Studer]

* [github] Enabling github actions on dev branch. [Christian Studer]

* [poetry, package] Updated python & the library versions. [Christian Studer]

* [github, python] Removing support for 3.7 and added 3.11. [Christian Studer]

Fix

* [stix2 import] Fixed missing imports removed by mistake. [Christian Studer]

* [stix2 import] Some cleanup. [Christian Studer]

- Better readability when possible
- Fixed typing
- Simplified some parts using `getattr` instead of
`hasattr` when possible

* [stix2 import] Fixed duplicate property that was causing issues with the parent class property. [Christian Studer]

* [tests] Fixed the remaining latest datetime/timestamp values that were possibly missing. [Christian Studer]

- Testing `datetime` values - i.e from the
`datetime` python library - instead of str

* [stix2 import] Fixed Marking definition objects handling. [Christian Studer]

- There are still some Marking definition we don't
parse yet - the ones with no `definition_type`
value - and we now properly handle the exception
that appear when we try to look at the ones that
are not loaded

* [stix2 import] Fixed wrong variable name. [Christian Studer]

* [stix2 import] Removed unused variable. [Christian Studer]

* [documentation] Fixed datetime/timestamp values in the ampping documentation. [Christian Studer]

* [tests] Fixed unittests on datetime/timestamp fields/values. [Christian Studer]

* [tests] Made sure all the datetime/timestamp fields/values are properly set in test samples. [Christian Studer]

* [stix2 export] Properly exporting datetime/timestamp fields/values. [Christian Studer]

* [tests] Made some datetime values UTC. [Christian Studer]

* [stix2 export] Fix naive timestamp. [Tomas Lima]

* [tests] Fixed relationships tests to match the recent changes on the default relationship types. [Christian Studer]

* [stix2 export] Typo. [Christian Studer]

- Fixes e918f69 and thus 33 for good this time

* [stix2 export] Fixed default relationships used between SDOs. [Christian Studer]

- The `relationship_specs` mapping dictionary now
only conains default relationships that are
unique between 2 SDOs, if there are at least 2
possible default relationships between 2 SDOs,
we do not know which one to choose
- In that case, or in the case there is no
default relationship known between 2 SDOs, we
us the `related-to` common relationship instead
of `has`
- As a result, this should fix 33

* [stix2 export] Variable name typo. [Christian Studer]

* [tests] Fixed tests for `country` galaxies export as STIX 2.1 Location objects. [Christian Studer]

* [stix2 export] Better `country` galaxy clusters parsing. [Christian Studer]

- We use the description (capitalised) to define
the `Location` name field of the country, and
the value (lower case) as a description, which
should fix 34

* [stix2 import] Avoiding warnings about empty object attribute values while converting Observable objects to MISP. [Christian Studer]

* [stix2 import] Fixed the unix extension parsing from User Account patterns. [Christian Studer]

* [stix2 import] Fixed recently renamed unix extension mapping. [Christian Studer]

* [stix2 import] Trying to fix a python 3.7 syntax issue for the remaining time it is still supported. [Christian Studer]

- 3.8 and above don't complain with the
`*(generator)` statement

* [stix2 import] Fixed the `email` object parsing. [Christian Studer]

* [tests] Fixed tests for the datetime attribute in STIX 2.0 File objects imported as `lnk` MISP objects. [Christian Studer]

* [tests] Fixed tests for STIX 2.0 File objects imported as `lnk` objects. [Christian Studer]

* [tests] Made the datetime fields in the File object - to be imported as `lnk` object - acceptable for STIX 2. [Christian Studer]

* [tests] Fixed the internal STIX 2.0 test samples for `lnk` object import. [Christian Studer]

* [stix2 import] Fixed wrongly set `self` variable. [Christian Studer]

* [stix2 import] Better separation of exceptions during observable objects parsing. [Christian Studer]

* [stix2 import] Some clean-up. [Christian Studer]

- Including:
- a wrong function name fixed
- a better naming for some SDOs parsing
- some unused methods removed

* [stix2 import] Fixed imports. [Christian Studer]

* [stix2 import] A very quick fix on observable mapping error message. [Christian Studer]

* [stix2 import] Fixed imports. [Christian Studer]

* [stix2 import] Fixed the Email Address observable object parsing. [Christian Studer]

* [stix2 import] Avoiding issue while parsing IP addresses patterns with empty list of attributes mapped. [Christian Studer]

* [stix2 import] Reusing the `object_marking_refs` fields parsing in a function. [Christian Studer]

* [stix2 import] Fixed the Location object parsing. [Christian Studer]

* [stix2 import] Correctly handling issues with observable object mapping. [Christian Studer]

* [stix2 import] Fixed the Location object parsing. [Christian Studer]

* [stix2 import] Fixed the pattern & observable types extraction. [Christian Studer]

* [stix2 import] Fixed the `MarkingDefinition` objects parsing function. [Christian Studer]

* [stix2 import] Made the MISP Attributes dict creation more generic and including the `object_marking_ref` field parsing. [Christian Studer]

* [stix2 import] Avoiding issues with Marking-Definition objects with no `definition_type` field. [Christian Studer]

* [stix2 import] Avoiding issues with Report or Grouping object that has no `name` field. [Christian Studer]

Other

* Wip: [stix2 import] Parsing Network Traffic objects. [Christian Studer]

* Wip: [stix2 import] Simplified the email observable objects parsing. [Christian Studer]

* Wip: [stix2 import] Parsing Observed Data with domain & ip observable objects. [Christian Studer]

* Wip: [stix2 import] Importing Software observable objects with the `software` object template. [Christian Studer]

* Merge pull request 35 from SYNchroACK/fix/naive-timestamp. [Christian Studer]

Fix naive timestamp

* Wip: [stix2 import] Converting `WindowsRegistryKey` objects as `registry-key` & `registry-key-value` objects or `regkey` attributes. [Christian Studer]

* Wip: [stix2 import] Parsing User Account Observable objects. [Christian Studer]

* Wip: [stix2 import] Parsing X509 Certificate Observable objects. [Christian Studer]

- Reusing some stuff that is similar as the x509
pattern parsing

* Wip: [stix2 import] Parsing Process observable objects. [Christian Studer]

* Wip: [stix2 import] Made the Observable objects parsing more generic. [Christian Studer]

- Reducing the amount of variables by putting all
the observable objects in one single dictionary.
Instead of using multiple dictionaries for
different object types, we use one and added
generic selection methods instead

* Wip: [stix2 import] Updated the File & Directory observable objects parsing to better support the references between objects. [Christian Studer]

* Revert "fix: [stix2 import] Trying to fix a python 3.7 syntax issue for the remaining time it is still supported" [Christian Studer]

This reverts commit 556c433557e3fb6ba997ef0b7c1c8dd922d19e64.

* Wip: [stix2 import] Converting `Directory` observable objects as the recently added `directory` object template. [Christian Studer]

- Also fixed the observable objects mapping to
MISP for `lnk` objects import

* Wip: [stix2 import] Simplifying the Observable objects conversion with fewer function calls. [Christian Studer]

* Wip: [stix2 import] Properly handling filtering on multiple observable object types. [Christian Studer]

* Wip: [stix2 import] Yield-ing observable objects instead of returning them in a list. [Christian Studer]

* Wip: [stix2 import] Importing EmailMessage Observable objects. [Christian Studer]

* Wip: [stix2 import] Importing File Observable objects in the case of a single field value imported as MISP Attribute. [Christian Studer]

* Wip: [stix2 import] Better "attribute or object" determination for File observable objects, searching for the `extensions` field. [Christian Studer]

* Wip: [stix2 import] Importing MISP `file` objects from File Observable objects. [Christian Studer]

- Also includes the modification of some parsing
functions that are used for multiple Observable
objects

* Add: [tests] Added tests for the time fields recently added into the `file` object template. [Christian Studer]

* Add: [stix export] Included the handling of the object attribute recently added to the `file` object template. [Christian Studer]

- Namely the object attributes mentioned here are:
- `acces-time`
- `creation-time`
- `modification-time`

* Revert "wip: [stix2 export] Simplified the Galaxies mapping" [Christian Studer]

This reverts commit 76f4e6f58fa332e3b9170a20151aca762df16dca.

* Update README.md. [Alexandre Dulaunoy]

Fix documentation for generated website

* Merge pull request 26 from coolacid/main. [Alexandre Dulaunoy]

Use MISP event UUID for bundle ID

* Use f-strings like elsewhere, check for _misp_event to pass tests. [Jason Kendall]

* Use MISP event UUID for bundle ID. [Jason Kendall]

* Wip: [stix2 import] Better Observable objects parsing. [Christian Studer]

* Wip: [stix2 import] Cleaner UUID sanitation in some cases. [Christian Studer]

* Wip: [stix2 import] Better Observable objects exceptions handling. [Christian Studer]

* Wip: [stix2 import] Quick Observable objects parsing improvement. [Christian Studer]

* Wip: [stix2 import] Cleaner way to handle Observable objects import & supporting a few more observable object types. [Christian Studer]

* Wip: [stix2 export] Simplified the Galaxies mapping. [Christian Studer]

* Wip: [stix2 import] Parsing `domain-name` observable objects and reusing some generic observable objects parsing code. [Christian Studer]

* Wip: [stix2 import] Started parsing external STIX 2 observable objects. [Christian Studer]

* Wip: [stix2 import] Parsing `object_marking_refs` field from several STIX objects to import tags in object attributes. [Christian Studer]

**Full Changelog**: https://github.com/MISP/misp-stix/compare/v2.4.168...v2.4.169