Dl1-data-handler

Added mode to write non-triggered MC events using Eventio. Some optimizations/performance improvements.

Note: although the Reader and Processor/ImageMapper have been added, they are still under development. A finalized version of both will be part of the next major release (v0.8.0).

Major Features And Improvements

* Added save_mc_events flag to allow saving of MC event information in a separate table (MC_Events).
* Added single-threaded write data script for easier testing/profiling.

Bug Fixes and Other Changes

* Reduced write time by ~12% per event for triggered events (events with images) by making various optimizations.
* Cleaned up and simplified class definitions/function signatures.

Known Issues

* The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead __VERSION__ = "unknown") so it is not being written correctly to the output file headers.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.

New release to fix missing gain/channel selection for processing the Prod3 NSB dataset.

Note: although the Reader and Processor have been added, they are still under development. A finalized version of both will be part of the next major release (v0.8.0).

Major Features And Improvements

* Added gain channel selection functionality to DL1DataWriter.
* DL1DataReader and Processor functionality added but still under development.

Bug Fixes and Other Changes

* Fix to filename parsing in generate_runlist.py script.
* Fixes/improvements to installation and dependency management with setuptools and conda.
* Fixed Travis/pytest integration.

Known Issues

* prod_site_array, prod_site_coord, and prod_site_subarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.

This release includes an update to the version in setup.py.

Major Features And Improvements

* None.

Bug Fixes and Other Changes

* Version in setup.py updated to 0.7.2 to avoid version confusion by users.

Known Issues

* prod_site_array, prod_site_coord, and prod_site_subarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.

This release includes fixes to the pip/setuptools installation as well as a minor fix to the generate_runlist.py script.

Major Features And Improvements

* Setuptools installation process streamlined so it does not depend on manual installation of dependencies from source/pip. Now the entire installation can be done via setup.py/pip. requirements.txt is no longer required and was therefore removed.
* Due to issues with exporting conda environments across systems, conda environment-based installation was removed. A conda package release is planned for the near future.

Bug Fixes and Other Changes

* scripts/generate_runlist.py was modified to handle filenames with underscores in their prod_info fields.

Known Issues

* prod_site_array, prod_site_coord, and prod_site_subarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.
* There appears to be an ongoing issue with processing SCT data. This is being investigated.

This release includes only the writer (raw data to HDF5) portion of the full DL1 Data Handler package. The matching data reader and processor will be included in future releases.

Major Features And Improvements

* Major refactoring of image-extractor into DL1DataWriter class, based on a DL1DataDumper class and the ctapipe EventSource class.
* Now uses on YAML configuration files and run lists for all processing settings
* Python multiprocessing package used to allow parallel processing of runs.
* Image mapping functionality in image.py removed.
* Command line script moved out of main package and into separate write_data.py script.
* generate_runlist.py script added to automatically create run lists from directories of files.
* Telescope type names in output files (table columns and table names) modified from original ctapipe names to be valid Python identifiers (for use when doing selections).
* Telescope optics and camera types added to telescope type table.
* xmax column added to event table.
* Telescope image multiplicity columns added to event table.

Bug Fixes and Other Changes

* .gitignore updated to include build, dist directories.
* TravisCI configuration instructions updated, Python version updated 3.5 -> 3.6.
* AUTHORS list updated.
* run_id renamed to obs_id to match ctapipe container field.
* requirements.txt file reduced by using pipreqs instead of pip freeze.
* Out of date scripts moved to scripts/deprecated.
* Improvements to logging.
* Example config and run list files added.
* Fixed an issue where some tables were not being written correctly before files were closed.

Breaking Changes to the API

* Various name changes to columns and tables (see wiki for full data format specification)
* Telescope type names in output files (table columns and table names) modified from original ctapipe names to be valid Python identifiers (for use when doing selections).
* Telescope optics and camera types added to telescope type table.
* xmax column added to event table
* Telescope image multiplicity columns added to event table

Known Issues

* prod_site_array, prod_site_coord, and prod_site_subarray are included in the output files but defaulted to None, as they are included in the ctapipe MC header container but not supported in event_source as of https://github.com/cta-observatory/ctapipe/commit/5ef63ef2a7a093455f4b0683a68653cc42e84df7.
* Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The call has been removed as it has no effect on the output files.

Major Features And Improvements

* Updated telescope, optics, and camera naming conventions to match ctapipe formatting. Implemented distinction between camera (image)-dependent settings and telescope type settings.
* Added configuration file support and example configuration file for ease of use.
* Added support for compression via PyTables.
* Added support for chunking via PyTables.
* Added support for table column indexing via PyTables.
* ImageExtractor is now available as a conda package at https://anaconda.org/bryankim96/image-extractor.

Bug Fixes and Other Changes

* Updated to internally use new ctapipe.io.EventSource instead of hessioeventsource.
* Updated name of event.r0.run_id to event.r0.obs_id.
* Fixed overflow bug in tel_id column causing duplicated telescope ids.
* Added environment.yml file in addition to requirements.txt for easy Anaconda environment setup.
* On exception/failure while processing, ImageExtractor will now remove the incomplete file.

Breaking Changes to the API

* Type change in tel_id column in array_info table.
* Changes to all telescope names (affects both event_info table rows and the names of image tables).

Known Issues

* Some metadata parameters within the user attributes, specifically those coming from in the simtel history block, are still not filled because they are not yet supported by ctapipe's EventSource. See [this issue](https://github.com/cta-observatory/ctapipe/pull/655) and [this issue](https://github.com/cta-observatory/image-extractor/issues/14)