============================
API Changes
-----------
- ``reference_location`` is now a required argument for ``SubarrayDescription``
[`2402 <https://github.com/cta-observatory/ctapipe/pull/2402>`__]
- ``CameraGeometry.position_to_pix_index`` will now return the minimum integer value for invalid
pixel coordinates instead of -1 due to the danger of using -1 as an index in python accessing
the last element of a data array for invalid pixels.
The function will now also no longer raise an error if the arguments are empty arrays and instead
just return an empty index array.
The function will also no longer log a warning in case of coordinates that do not match a camera pixel.
The function is very low-level and if not finding a pixel at the tested position warrants a warning or
is expected will depend on the calling code. [`2397 <https://github.com/cta-observatory/ctapipe/pull/2397>`__]
- Change the definition of the ``leakage_pixels_width_{1,2}`` image features
to give the ratio of pixels at the border to the pixels after cleaning
instead of to the total number of pixels of the camera. [`2432 <https://github.com/cta-observatory/ctapipe/pull/2432>`__]
- Change how the ``DataWriter`` writes pointing information.
Before, each unique pointing position was written in a table
with the event time as index column into ``dl1/monitoring/telescope/pointing``.
This has two issues: For observed data, each pointing will be unique
in horizontal coordinates due to tracking a fixed ICRS coordinate.
Resulting in a pointing position written for each event, although the
resolution of the monitoring is much lower.
For simulated events, the event time is the timestamp of the simulation
and pointing is fixed in ``AltAz``.
``ctapipe`` was using the closest point in time for simulated events when
reading data back in, however, this is problematic in case of many
simulation runs processed in parallel.
We now store the first received pointing information
in the ``configuration/telescope/pointing`` table per obs id,
only for simulation events. [`2438 <https://github.com/cta-observatory/ctapipe/pull/2438>`__]
- Replace ``n_signal`` and ``n_background`` options in ``ctapipe-train-particle-classifier``
with ``n_events`` and ``signal_fraction``, where ``signal_fraction`` = n_signal / (n_signal + n_background). [`2465 <https://github.com/cta-observatory/ctapipe/pull/2465>`__]
- Move the ``TableLoader`` options from being traitlets to
each ``read_...`` method allowing to load different data with the
same TableLoader-Instance.
In addition the default values for the options have changed. [`2482 <https://github.com/cta-observatory/ctapipe/pull/2482>`__]
- Adding monitoring: MonitoringCameraContainer as keyword argument to
the ``ImageCleaner`` API so cleaning algorithms can now access
relevant information for methods that e.g. require monitoring information. [`2511 <https://github.com/cta-observatory/ctapipe/pull/2511>`__]
- Unified the options for DataWriter and the data level names:
+-------------------------+--------------------------+
| Old | New |
+=========================+==========================+
| ``write_raw_waveforms`` | ``write_r0_waveforms`` |
+-------------------------+--------------------------+
| ``write_waveforms`` | ``write_r1_waveforms`` |
+-------------------------+--------------------------+
| ``write_images`` | ``write_dl1_images`` |
+-------------------------+--------------------------+
| ``write_parameters`` | ``write_dl1_parameters`` |
+-------------------------+--------------------------+
| ``write_showers`` | ``write_dl2`` |
+-------------------------+--------------------------+
This changes requires that existing configuration files are updated
if they use these parameters [`2520 <https://github.com/cta-observatory/ctapipe/pull/2520>`__]
Bug Fixes
---------
- Ensure that ``SubarrayDescription.reference_location`` is always generated by
SimTelEventSource``, even if the metadata is missing. In that case, construct a
dummy location with the correct observatory height and latitude and longitude
equal to zero ("Null Island").
- Fixed the definition of ``h_max``, which was both inconsistent between
`~ctapipe.reco.HillasReconstructor` and `~ctapipe.reco.HillasIntersection`
implementations, and was also incorrect since it was measured from the
observatory elevation rather than from sea level.
The value of ``h_max`` is now defined as the height above sea level of the
shower-max point (in meters), not the distance to that point. Therefore it is
not corrected for the zenith angle of the shower. This is consistent with the
options currently used for *CORSIKA*, where the *SLANT* option is set to false,
meaning heights are actual heights not distances from the impact point, and
``x_max`` is a *depth*, not a *slant depth*. Note that this definition may be
inconsistent with other observatories where slant-depths are used, and also note
that the slant depth or distance to shower max are the more useful quantities
for shower physics. [`2403 <https://github.com/cta-observatory/ctapipe/pull/2403>`__]
- Add the example config for ctapipe-train-disp-reconstructor
to the list of configs generated by ctapipe-quickstart. [`2414 <https://github.com/cta-observatory/ctapipe/pull/2414>`__]
- Do not use a hidden attribute of ``SKLearnReconstructor`` in ``ctapipe-apply-models``. [`2418 <https://github.com/cta-observatory/ctapipe/pull/2418>`__]
- Add docstring for ``ctapipe-train-disp-reconstructor``. [`2420 <https://github.com/cta-observatory/ctapipe/pull/2420>`__]
- Remove warnings about missing R1 or DL0 data when using the CameraCalibrator.
These were previously emitted directly as python warnings and did not use the
component logging system, which they now do.
As we do not actually expect R1 to be present it was also moved down to
debug level. [`2421 <https://github.com/cta-observatory/ctapipe/pull/2421>`__]
- Check that the array pointing is given in horizontal coordinates
before training a ``DispReconstructor``. [`2431 <https://github.com/cta-observatory/ctapipe/pull/2431>`__]
- Fix additional, unwanted columns being written into disp prediction output. [`2440 <https://github.com/cta-observatory/ctapipe/pull/2440>`__]
- Properly transform pixel coordinates between ``CameraFrame``
and ``TelescopeFrame`` in ``MuonIntensityFitter`` taking.
Before, ``MuonIntensityFitter`` always used the equivalent focal
length for transformations, now it is using the focal length
attached to the ``CameraGeometry``, thus respecting the
``focal_length_choice`` options of the event sources. [`2464 <https://github.com/cta-observatory/ctapipe/pull/2464>`__]
- Fix colored logging in case of custom log levels being defined. [`2505 <https://github.com/cta-observatory/ctapipe/pull/2505>`__]
- Fix a possible out-of-bounds array access in the FlashCamExtractor. [`2544 <https://github.com/cta-observatory/ctapipe/pull/2544>`__]
Data Model Changes
------------------
- Remove redundant ``is_valid`` field in ``DispContainer`` and rename the remaining field.
Use the same prefix for both containers filled by ``DispReconstructor``.
Fix default name of ``DispReconstructor`` target column.
Let ``HDF5EventSource`` load ``DispContainer``. [`2443 <https://github.com/cta-observatory/ctapipe/pull/2443>`__]
- Change R1- and DL0-waveforms datamodel shape from (n_pixels, n_samples)
to be always (n_channels, n_pixels, n_samples). ``HDF5EventSource`` was adjusted
accordingly to support also older datamodel versions.
Re-introduce also the possibility of running ``ImageExtractor``\s on data
consisting of multiple gain channels. [`2529 <https://github.com/cta-observatory/ctapipe/pull/2529>`__]
New Features
------------
- Large updates to the Image Pixel-wise fit for Atmospheric Cherenkov Telescopes reconstruction method (https://doi.org/10.48550/arXiv.1403.2993)
* ImPACT - General code clean up and optimisation. Now updated to work similarly to other reconstructors using the standardised interface, such that it can be used ctapipe-process. Significant improvements to tests too
* ImPACT_utilities - Created new file to hold general usage functions, numba used in some areas for speedup
* template_network_interpolator - Now works with templates with different zenith and azimuth angles
* unstructured_interpolator - Significant speed improvements
* pixel_likelihood - Constants added back to neg_log_likelihood_approx, these are quite important to obtaining a well normalised goodness of fit.
* hillas_intersection - Fixed bug in core position being incorrectly calculated, fixed tests too [`2305 <https://github.com/cta-observatory/ctapipe/pull/2305>`__]
- Allow passing the matplotlib axes to the ``SubarrayDescription.peek`` function,
fix warnings in case of layout engine being already defined. [`2369 <https://github.com/cta-observatory/ctapipe/pull/2369>`__]
- Add support for interpolating a monitoring pointing table
in ``TableLoader``. The corresponding table is not yet written by ``ctapipe``,
but can be written by external tools.
This is to enable analysis of real observations, where the pointing changes over time in
alt/az. [`2409 <https://github.com/cta-observatory/ctapipe/pull/2409>`__]
- Implement the overburden-to height a.s.l. transformation function in the atmosphere module
and test that round-trip returns original value. [`2422 <https://github.com/cta-observatory/ctapipe/pull/2422>`__]
- In case no configuration is found for a telescope in ``TelescopeParameter``,
it is now checked whether the telescope exists at all to provide a better
error message. [`2429 <https://github.com/cta-observatory/ctapipe/pull/2429>`__]
- Allow setting n_jobs on the command line for the
train_* and apply_models tools using a new ``n_jobs`` flag.
This temporarily overwrites any settings in the (model) config(s). [`2430 <https://github.com/cta-observatory/ctapipe/pull/2430>`__]
- Add support for using ``str`` and ``Path`` objects as input
to ``ctapipe.io.get_hdf5_datalevels``. [`2451 <https://github.com/cta-observatory/ctapipe/pull/2451>`__]
- The recommended citation for ctapipe has been updated to the ICRC 2023 proceeding,
please update. [`2470 <https://github.com/cta-observatory/ctapipe/pull/2470>`__]
- Support astropy 6.0. [`2475 <https://github.com/cta-observatory/ctapipe/pull/2475>`__]
- The ``DispReconstructor`` now computes a score for how certain the prediction of the disp sign is. [`2479 <https://github.com/cta-observatory/ctapipe/pull/2479>`__]
- Also load the new fixed pointing information in ``TableLoader``.
Add option ``keep_order`` to ``ctapipe.io.astropy_helpers.join_allow_empty``
that will keep the original order of rows when performing left or right joins. [`2481 <https://github.com/cta-observatory/ctapipe/pull/2481>`__]
- Add an ``AstroQuantity`` trait which can hold any ``astropy.units.Quantity``. [`2524 <https://github.com/cta-observatory/ctapipe/pull/2524>`__]
- Add function ``ctapipe.coordinates.get_point_on_shower_axis``
that computes a point on the shower axis in alt/az as seen
from a telescope. [`2537 <https://github.com/cta-observatory/ctapipe/pull/2537>`__]
- Update bokeh dependency to version 3.x. [`2549 <https://github.com/cta-observatory/ctapipe/pull/2549>`__]
Maintenance
-----------
- The CI system now reports to the CTA SonarQube instance for code quality tracking [`2214 <https://github.com/cta-observatory/ctapipe/pull/2214>`__]
- Updated some numpy calls to not use deprecated functions. [`2406 <https://github.com/cta-observatory/ctapipe/pull/2406>`__]
- The ``ctapipe`` source code repository now uses the ``src/``-based layout.
This fixes the editable installation of ctapipe. [`2459 <https://github.com/cta-observatory/ctapipe/pull/2459>`__]
- Fix headings in docs. Change occurrences of ``API Reference`` to ``Reference/API`` for consistency.
Change capitalization of some headings for consistency. [`2474 <https://github.com/cta-observatory/ctapipe/pull/2474>`__]
- The ``from_name`` methods of instrument description classes now raise a warning
that it is better to access instrument information via a ``SubarrayDescription``.
Also improve documentation in instrument module to explain when not to use the
various ``from_name()`` methods. These are provided for the case when no event
data is available, e.g. for unit testing or demos, but do not guarantee that the
returned instrument information corresponds to a particular set of event data. [`2485 <https://github.com/cta-observatory/ctapipe/pull/2485>`__]
- Support and test on python 3.12. [`2486 <https://github.com/cta-observatory/ctapipe/pull/2486>`__]
- Drop support for python 3.9. [`2526 <https://github.com/cta-observatory/ctapipe/pull/2526>`__]
Refactoring and Optimization
----------------------------
- Load data and apply event and column selection in chunks in ``ctapipe-train-*``
before merging afterwards.
This reduces memory usage. [`2423 <https://github.com/cta-observatory/ctapipe/pull/2423>`__]
- Make default ML config files more readable and add comments. [`2455 <https://github.com/cta-observatory/ctapipe/pull/2455>`__]
- Update and add missing docstrings related to the ML functionalities. [`2456 <https://github.com/cta-observatory/ctapipe/pull/2456>`__]
- Add ``true_impact_distance`` to the output of ``CrossValidator``. [`2468 <https://github.com/cta-observatory/ctapipe/pull/2468>`__]
- Add ``cache=True`` to some numba-compiled functions which were missing it. [`2477 <https://github.com/cta-observatory/ctapipe/pull/2477>`__]
- Write cross validation results for each model out immediately after validation to free up memory earlier. [`2483 <https://github.com/cta-observatory/ctapipe/pull/2483>`__]
- Compute deconvolution parameters in FlashCamExtractor only as needed. [`2545 <https://github.com/cta-observatory/ctapipe/pull/2545>`__]