Dl1-data-handler

Compatibility with ctlearn v0.5.0. Bugfixes w.r.t. v0.8.2.

Major Features And Improvements

Bug Fixes and Other Changes
* Fix counting of class examples for labels that are sequences. aribrill
* added electrons to the particle transformation. TjarkMiener

Compatibility with ctapipe v0.8.0. Numerous bugfixes and improvements.

DL1DataReader compatible for the file format of DL1DH >= v0.7.4.

Major Features And Improvements

* Updated DL1DataWriter to ctapipe v0.8.0.
* gain_selector in ctapipe now. Redundant unit tests deleted.
* Numerous improvements for the ImageMapper (IM) class.
* ctapipe and ctapipe-extra optional dependencies for DL1DataReader and IM.
* Added and modified Transform functions for the data processor.
* Added telescope pointing for alt/az transformation
* Added 'sorting' transformation for stereo reading.
* Added and modified filters for the data processor in filters.py.
* Updated example notebook.
* Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

* Fixed FlashCam mapping bug with reading pixel positions from the data files.
* Normalization for nearest interpolation in IM.
* Removed rotation back to original orientation option in IM.
* Speeded up the data reading.
* Added parallel reads support.
* Sorting transformation fixed performance drop from the CTLearn cnn-rnn (stereo) model.

Known Issues

* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

v0.8.1-legacy
Release 0.8.1-legacy

"Legacy" release adapting DL1DataReader for the file format of DL1DH <0.6.0.

Major Features And Improvements

* Added `reader_legacy` module containing a DL1DataReader adapted for the legacy file format.

Bug Fixes and Other Changes

* Updated `dl1_data_handler_demo` notebook for legacy reader.
* Updated `transforms` and `utils` modules to use legacy column names in relevant Transforms and filters.
* Added `SortTelescopes` Transform to sort telescopes by trigger or image size.
* Added `event_multiplicity_filter` to filter events based on multiplicity.

Known Issues

* The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead __VERSION__ = "unknown") so it is not being written correctly to the output file headers.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

Integration of DL1DH Reader, Processor, and ImageMapper completed. Numerous bugfixes and improvements for compatibility with CTLearn.

Added option for serial/parallel writing modes to DL1DH Writer. Fixes to setuptools and conda build process.

Major Features And Improvements

* Finished changes to reader and image mapper APIs to be compatible with CTLearn
* Finished incorporating image mapper and DL1DH processor.
* Added example notebook.
* Added six new Transform functions for the data processor and moved to transform.py
* Added single threaded/parallel writes modes as a config option for DL1DataWriter.
* Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

* Fixed output_dir option of DataWriter to correctly write output files to directory.
* Added output_dir option to generate_runlists.py script.
* Moved example config files and runlists to /config folder
* Added separate expected_mc_events config setting for optimizing chunking on the MC_Events table.
* Fixed various issues with setuptools dependencies.
* Fixed FACT camera shape mismatch by upgrading to ctapipe-extra 0.2.17.
* Changed mapping table indices to start at 0 (removed 0 padding).
* Added option for camera rotation for LST, NectarCam, MAGICCam in image mapper.
* Fixed output_dir option in write_data.py script.
* Moved config files to config folder
* Changed indexing convention in image mapper.
* Changed get_image() to return a vector when map method is axial addressing
* Fixed status badge

Known Issues

* The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead __VERSION__ = "unknown") so it is not being written correctly to the output file headers.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

Added mode to write non-triggered MC events using Eventio. Some optimizations/performance improvements.

Note: although the Reader and Processor/ImageMapper have been added, they are still under development. A finalized version of both will be part of the next major release (v0.8.0).

Major Features And Improvements

* Added save_mc_events flag to allow saving of MC event information in a separate table (MC_Events).
* Added single-threaded write data script for easier testing/profiling.

Bug Fixes and Other Changes

* Reduced write time by ~12% per event for triggered events (events with images) by making various optimizations.
* Cleaned up and simplified class definitions/function signatures.

Known Issues

* The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead __VERSION__ = "unknown") so it is not being written correctly to the output file headers.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.

New release to fix missing gain/channel selection for processing the Prod3 NSB dataset.

Note: although the Reader and Processor have been added, they are still under development. A finalized version of both will be part of the next major release (v0.8.0).

Major Features And Improvements

* Added gain channel selection functionality to DL1DataWriter.
* DL1DataReader and Processor functionality added but still under development.

Bug Fixes and Other Changes

* Fix to filename parsing in generate_runlist.py script.
* Fixes/improvements to installation and dependency management with setuptools and conda.
* Fixed Travis/pytest integration.

Known Issues

* prod_site_array, prod_site_coord, and prod_site_subarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.

This release includes an update to the version in setup.py.

Major Features And Improvements

* None.

Bug Fixes and Other Changes

* Version in setup.py updated to 0.7.2 to avoid version confusion by users.

Known Issues

* prod_site_array, prod_site_coord, and prod_site_subarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.