Dl1-data-handler

Major Features And Improvements

Bug Fixes and Other Changes

* Corrected release version in setup.py.

Breaking Changes to the API

Known Issues

* Due to the new event.mc.shower_primary_id attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install either the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software) or any ctapipe release from 0.5.3 on.
* Some metadata parameters within the user attributes, specifically those coming from in the simtel history block, are not filled. See [this issue](https://github.com/cta-observatory/pyhessio/issues/52).

Major Features And Improvements

* Added compatibility with breaking changes in ctapipe's instrument container implemented in v0.5.3.
* Added process_dataset.py script to merge, calibrate and dump to h5 datasets from CTA MC production "prod3b".

Bug Fixes and Other Changes

Breaking Changes to the API

* Compatibility with ctapipe versions previous to 0.5.3 has not been considered, so the code will break if one of such older versions of ctapipe is used in conjunction with this release.

Known Issues

* Due to the new event.mc.shower_primary_id attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install either the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software) or any ctapipe release from 0.5.3 on.
* Some metadata parameters within the user attributes, specifically those coming from in the simtel history block, are not filled. See [this issue](https://github.com/cta-observatory/pyhessio/issues/52).

Major Features And Improvements

* Added event_index column to image tables (mapping back to the event table) for use when reading data for single-telescope use

Bug Fixes and Other Changes

* Removed LICENCE
* Added .landscape.yml

Breaking Changes to the API

Known Issues

* Due to the new event.mc.shower_primary_id attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).
* Some metadata parameters still not fully implemented

Major Features And Improvements

* Existing telescope types renamed (SCT -> MSTS) and additional telescope types now supported (LST, MSTF, MSTN, SST1, SSTA, SSTC). 2D image mode still only supported for MSTS tel type
* Trace to image conversion for 2D image mode now handled by pre-generating a mapping table
* Now uses telescope num_pixels * optical_foc_len to identify telescope types
* Optical focal length and number of pixels now stored in the telescope table
* Saving raw image vectors (charge and arrival time) now supported in '1D' image mode
* '1D' image mode now saves raw images as float vectors, not uint vectors
* '1D' image mode now saves a PyTables table containing the charge image vector and arrival time image vector as separate columns
* All image tables now store a blank (all-zero) image at index 0. Index 0 is now used in event indices vectors to indicate a non-existant image, instead of index -1.
* Telescope table renamed to Telescope_Info
* Event table renamed to Event_Info
* Image tables renamed to [tel_type/tel_id]
* Now supports varying image scale factors and image dtypes by telescope type
* Units for event-level and telescope-level parameters now saved in the corresponding table attributes
* Removed reconstructed energy parameter from Event Info table
* Gamma_hadron_label parameter renamed to particle_id
* All float parameters (event-level and telescope-level) now saved as 32-bit floats
* Preselection cuts now handled using a dictionary cuts_dict (to be used in ctapipe) or a pre-generated EventDisplay cuts dictionary ED_cuts_dict
* File-level metadata now stored in root group attributes. Identical metadata fields are now enforced for all events in the output file
* particle_id
* zenith
* azimuth
* ImageExtractor version
* ctapipe version
* runlist
* The following metadata fields are included but reading them from the data is not yet implemented
* CORSIKA version
* Simtel version
* prod site array
* prod site subarray
* prod site B field
* prod site altitude
* spectral index
* E min
* E max
* Config file support removed, along with unnecessary options. Example config file removed. All remaining configuration options, value checking, default values now handled through the ImageExtractor class constructor
* 'mode'(gh_class,energy_recon) removed
* 'storage_mode' (all,mapped) -> storage_mode (tel_id,tel_type)
* 'use_pkl_dict' (True,False), energy_bins, preselection_cuts -> ED_cuts_dict,cuts_dict
* image 'mode' (PIXELS_TIMING_3C, etc.) -> img_mode (1D,2D), img_channels, include_timing
* image 'scale_factor' -> img_scale_factors(dict {tel_type: scale_factor})
* image 'dtype' -> img_dtypes(dict {tel_type: dtype})
* image 'dim_order' -> img_dim_order
* telescope 'type_mode' -> tel_type_list(list [tel_types, ...])
* energy_recon removed
* Spliting the data into train/val/test sets now supported
* Shuffling the event data randomly now supported
* '--split' command line flag now takes 3 command line arguments specifying the split fractions
* '--shuffle' command line flag now takes a random seed argument
* '--bins_cuts_dict' command line argument renamed to '--ED_cuts_dict'
* Command line program now takes a text file runlist as an input instead of a wildcard expression
* Added TravisCI and test coverage monitoring. Related dependencies added

Bug Fixes and Other Changes

* Removed unused import statements
* Collected constants
* Some PEP8 fixes (still ongoing)
* Renamed row_descriptors.py to row_types.py, trace_converter.py to image.py
* Debug command line flag now works properly with logger
* Debug output now gives the total number of events in the output file after each new input file processed
* Binning, shuffling, selecting telescopes, writing metadata all moved to ImageExtractor methods
* Some default ImageExtractor constructor arguments modified
* Example pytest tests added

Breaking Changes to the API

* SCT telescope type renamed to MSTS
* Config file removed and config options moved to ImageExtractor constructor. See above for detailed changes
* Images now stored in tables, not earrays
* Default image dtype changed to float32 from uint16. Images now saved without truncation or rounding
* Charge image vectors and arrival time image vectors now stored separately
* All image tables now save a blank image at index 0. Index 0 now used in the event table to indicate a non-existent image/non-triggered camera
* Telescope table renamed to Telescope_Info
* Event table renamed to Event_Info
* Image tables renamed to [tel_type/tel_id]
* Image scale factors and dtypes now determined by telescope type
* Reconstructed energy parameter removed
* Gamma_hadron_label renamed to particle_id
* All float parameters (event-level and telescope-level) now saved as 32-bit floats
* Command line program now takes a text file runlist as an input instead of a wildcard expression

Known Issues

* Due to the new event.mc.shower_primary_id attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).
* Some metadata parameters still not fully implemented

Major Features And Improvements

* Full reorganization of codebase to enhance modularity and adhere to standard development practices. Existing code split into new modules, classes, and methods.
* image_extractor functionality and command line tool restructured into ImageExtractor class + associated class methods and moved to image_extractor.py module. Image_extractor's command line functionality now supports full dataset processing (multiple simtel files) as well as dataset shuffling and splitting (previously handled using external scripts).
* Trace to image/array conversion functionality reorganized into TraceConverter class + associated class methods and moved to trace_converter.py module.
* Custom Pytables row descriptor classes (Event, Tel) moved to row_descriptors.py module.
* Configuration file spec and other config-related command line tools reorganized into config.py module.
* Modules and scripts moved into new directory structure. \_\_init\_\_.py added to image_extractor package.
* Changed gamma_hadron_label to use CORSIKA particle ID codes (gamma = 0, proton = 101, etc.) instead of previous binary labels (gamma = 1, proton = 0).
* Added support for Python package installation with setuptools (setup.py). Package not registered with Conda or PyPl, but can be installed locally using pip.
* Added preliminary support for automatic unit testing and continuous integration with Travis-CI. Installation tests are all passing, but unit tests are not implemented (placeholder created in /tests). Coverage monitoring is also supported through Coveralls.
* Changed requirements.txt to include full dependency list. Can now be used directly with pip install or conda install to install all dependencies.
* Changed default configuration files to use 'mapped' storage mode, 'PIXELS_1C' image mode, and a scale_factor of 1. With the new default configuration file, SCT images will be of shape (120,120,1) and will be stored in separate arrays outside of the event table.
* Removed shuffle_split_events.py and generate_dataset_pytables.sh from /scripts directory, as their functionality is now included in image_extractor.py
* Changed shuffling and splitting functions (now in image_extractor.py) to write shuffled/split events to new tables in existing HDF5 file rather than to a temporary file.
* Other refactoring and reorganization. Global constants unified and moved into appropriate classes/modules.
* Added LICENCE file

Bug Fixes and Other Changes

* Added img_dim_ordering parameter to default config files. Default value set to 'channels_last' to agree with TensorFlow standards.
* Fixed improper assignment to bin_number, reconstructed_energy when not using bins_cuts_dict.
* Changed bins_cuts_dict command line argument to image_extractor.py to be optional. image_extractor code modified to correctly handle case when bins_cuts_dict is missing/None. Will throw exception if config file requires bins_cuts_dict but none is provided.
* Added max_events optional command line argument to image_extractor.py specifying the maximum number of events to read from a given simtel file.
* Added shower core position (x,y), height of first interaction, altitude (zenith) angle, and azimuth angle parameters to event table.
* Added run array direction parameter to telescope table.
* Renamed 'MC_energy' parameter in event table to 'mc_energy'.
* Changed type of 'tel_map' parameter in event table from Int16 to Int32 to allow for large indices (more events) and avoid future overflow issues.

Breaking Changes to the API

* gamma_hadron_label convention changed from (gamma = 1, proton = 0) to (gamma = 0, proton = 101, etc.).
* Naming change from 'MC_energy' to 'mc_energy' for parameter in event table.
* Datatype change to 'tel_map' parameter in event table from Int16 to Int32.
* Major change to default config file behavior to 'mapped' storage mode, 'PIXELS_1C' image mode, and a scale_factor of 1.
* Requirements.txt format now includes full list of dependencies.

Known Issues

* Due to the new event.mc.shower_primary_id attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).

Major Features And Improvements

* Moved config_spec out of image_extractor.py and into config.py. Moved config.py out of /scripts and into main directory.
* Image_extractor now only saves trig_list when in 'all' mode and only stores tel_map when in 'mapped' mode to avoid duplicate information.
* Moved argument parsing, config file validation/reading, and and bins_cuts_dict loading out of image_extractor and into if __name__ == '__main__' block.

Bug Fixes and Other Changes

* Gamma hadron label now determined using particle id from event.mc.shower_primary_id
* Added basic docstrings
* Replaced all print statements for logging and debugging with logger statements
* Various minor naming and style changes to align closer with PEP8 standards
* Removed unused import statements

Breaking Changes to the API

* Image_extractor now only saves trig_list when in 'all' mode and only stores tel_map when in 'mapped' mode to avoid duplicate information.

Known Issues

* Due to the new event.mc.shower_primary_id attribute of the ctapipe MC container being only recently added, the current ctapipe version on anaconda (https://anaconda.org/cta-observatory/ctapipe) does not include it. This functionality will not work with the conda installation of ctapipe. Instead, at the current time it is necessary to install the dev version of ctapipe as described here (https://cta-observatory.github.io/ctapipe/getting_started/index.html#get-the-ctapipe-software).