Kestrel-lang

==================

Added
-----

- Multi-process support for stix-shifter data source interface

- Each native data source query is executed in a subprocess
- A pool of translators are created to pick up translation tasks for each transmitted page/batch
- Ingestion is serialized in main proccess to avoid multi-process execution for SQLite
- Two queues between transmitter/translator and translator/ingestor are used
- Both stix-shifter translation and firepit fast-translation are supported
- With debug flag, the translated results (JSON or DataFrame) will be dump to disk
- Unit tests for the translator subprocess in different modes

- Additional syntax/keywords on singular timeunits

- New variable transformer function ``ADDOBSID``

- Add new syntax and codegen
- Add additional documentatoin

- Unit tests on CLI

- Invoking with ``kestrel x.hf``
- Invoking with ``python -m kestrel x.hf``

Fixed
-----

- No dumped data in stix-shifter interface when debug is enabled
- Multiprocessing conflict with ``runpy``
- STIX-shifter module verification failure due to pypi website update

Removed
-------

- Deprecated functions in ``kestrel/codegen/relations.py``

Changed
-------

- Examples in Kestrel config YAML

==================

Changed
-------

- Kestrel variable definition syntax changed back to ``CNAME`` from ``ECNAME``
- stix-shifter data source profile config changes

- Replace ``result_limit`` with ``retrieval_batch_size``
- Replace ``timeout`` with ``single_batch_timeout``
- Add default values for the configs
- Document updates
- The new fields will be processed by Kestrel before given to stix-shifter

- Replace stix-shifter sync APIs with async APIs

Added
-----

- Scalability end-to-end testing for large query with multiple pages
- Test cases for new stix-shfiter data source configs

Fixed
-----

- Temporary fix of stix-shifter/issues/1493

- Add retry-once logic if server timeout (busy CPU on the client side)
- Nullify the pipelining; need better long-term fix to enable it

- Fixed bugs and reimplement ``transmission_complete()`` in stix-shifter data source interface

==================

Changed
-------

- Upgrade stix-shifter from v4 to v5 in the stix-shifter datasource interface
- Bump stix-shifter version to v5.3.0 to include latest Elastcisearch ECS mappings
- Restrict scopes of Github workflows to eliminate unnecessary executions

Added
-----

- stix-shifter datasource interface query procedure pipelining: a producer-consumer model for transmission and translation/ingestion
- Integration testing with stix-shifter and the first live data source---Elasticsearch
- Raw String implemented in Kestrel
- Documentation on raw String

Fixed
-----

- Logging module reimplemented to fix 334
- asyncio bug in ``tests/test_fast_translate.py``

===================

Fixed
-----

- A bug in firepit v2.3.16 when fast translation is in use; fixed in firepit v2.3.17
- Improved logic on prefetch skipping; fix 322
- Fixing several unit tests with the improved prefetch skipping logic

===================

Added
-----

- Using process UUID for process identification 252 93
- Connector timeout config in stix-shifter data source interface doc

Fixed
-----

- Library deprecation: pkg_resources
- Invalid STIX bundle (missing identity SCO type) yielded by stix-shifter data source interface

Removed
-------

- Python 3.7 support

===================

Fixed
-----

- Typo in pip install suggestion for stix-shifter modules
- Updated github workflows
- Vars created via assign should not lose reference attributes 312