Dl1-data-handler

Numerous bugfixes and improvements.
DLEventSource for MAGIC calibrated files by sahilyadav27.
DLEventSource for MAGIC superstar files.
Expansion of the data format with image parameters by LucaRomanato.
Compatibility with ctapipe v0.9.1.

Major Features And Improvements

* Updated DL1DataWriter to ctapipe v0.9.1 and expand the data format with image parameters.
* Make ctapipe's image_extractor and gain_selector configurable in the config.
* Added multiple parameter tables for different cleaning settings. Use MARS default ones for MAGIC superstar files.
* Added ability to only store selected tels (by tel_id).
* Updated example notebook (also for MAGIC data).
* Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes


Known Issues

* The image parameters can't be read by the current DL1DataReader. This is planned for the next release DL1DH v0.9.1.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

Compatibility with ctlearn v0.5.0. Bugfixes w.r.t. v0.8.2.

Major Features And Improvements

Bug Fixes and Other Changes
* Fix counting of class examples for labels that are sequences. aribrill
* added electrons to the particle transformation. TjarkMiener

Compatibility with ctapipe v0.8.0. Numerous bugfixes and improvements.

DL1DataReader compatible for the file format of DL1DH >= v0.7.4.

Major Features And Improvements

* Updated DL1DataWriter to ctapipe v0.8.0.
* gain_selector in ctapipe now. Redundant unit tests deleted.
* Numerous improvements for the ImageMapper (IM) class.
* ctapipe and ctapipe-extra optional dependencies for DL1DataReader and IM.
* Added and modified Transform functions for the data processor.
* Added telescope pointing for alt/az transformation
* Added 'sorting' transformation for stereo reading.
* Added and modified filters for the data processor in filters.py.
* Updated example notebook.
* Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

* Fixed FlashCam mapping bug with reading pixel positions from the data files.
* Normalization for nearest interpolation in IM.
* Removed rotation back to original orientation option in IM.
* Speeded up the data reading.
* Added parallel reads support.
* Sorting transformation fixed performance drop from the CTLearn cnn-rnn (stereo) model.

Known Issues

* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

v0.8.1-legacy
Release 0.8.1-legacy

"Legacy" release adapting DL1DataReader for the file format of DL1DH <0.6.0.

Major Features And Improvements

* Added `reader_legacy` module containing a DL1DataReader adapted for the legacy file format.

Bug Fixes and Other Changes

* Updated `dl1_data_handler_demo` notebook for legacy reader.
* Updated `transforms` and `utils` modules to use legacy column names in relevant Transforms and filters.
* Added `SortTelescopes` Transform to sort telescopes by trigger or image size.
* Added `event_multiplicity_filter` to filter events based on multiplicity.

Known Issues

* The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead __VERSION__ = "unknown") so it is not being written correctly to the output file headers.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

Integration of DL1DH Reader, Processor, and ImageMapper completed. Numerous bugfixes and improvements for compatibility with CTLearn.

Added option for serial/parallel writing modes to DL1DH Writer. Fixes to setuptools and conda build process.

Major Features And Improvements

* Finished changes to reader and image mapper APIs to be compatible with CTLearn
* Finished incorporating image mapper and DL1DH processor.
* Added example notebook.
* Added six new Transform functions for the data processor and moved to transform.py
* Added single threaded/parallel writes modes as a config option for DL1DataWriter.
* Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

* Fixed output_dir option of DataWriter to correctly write output files to directory.
* Added output_dir option to generate_runlists.py script.
* Moved example config files and runlists to /config folder
* Added separate expected_mc_events config setting for optimizing chunking on the MC_Events table.
* Fixed various issues with setuptools dependencies.
* Fixed FACT camera shape mismatch by upgrading to ctapipe-extra 0.2.17.
* Changed mapping table indices to start at 0 (removed 0 padding).
* Added option for camera rotation for LST, NectarCam, MAGICCam in image mapper.
* Fixed output_dir option in write_data.py script.
* Moved config files to config folder
* Changed indexing convention in image mapper.
* Changed get_image() to return a vector when map method is axial addressing
* Fixed status badge

Known Issues

* The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead __VERSION__ = "unknown") so it is not being written correctly to the output file headers.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

Added mode to write non-triggered MC events using Eventio. Some optimizations/performance improvements.

Note: although the Reader and Processor/ImageMapper have been added, they are still under development. A finalized version of both will be part of the next major release (v0.8.0).

Major Features And Improvements

* Added save_mc_events flag to allow saving of MC event information in a separate table (MC_Events).
* Added single-threaded write data script for easier testing/profiling.

Bug Fixes and Other Changes

* Reduced write time by ~12% per event for triggered events (events with images) by making various optimizations.
* Cleaned up and simplified class definitions/function signatures.

Known Issues

* The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead __VERSION__ = "unknown") so it is not being written correctly to the output file headers.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.