Dkist

=================

Backwards Incompatible Changes
------------------------------

- A major refactor of the ``FileManager`` code has taken place, the following subpackages have been renamed:

* ``dkist.io.loaders`` is now ``dkist.io.dask.loaders``.
* ``dkist.io.dask_utils`` is now ``dkist.io.dask.utils``.
* ``dkist.io.file_manager`` is now ``dkist.io.striped_array`` (apart from DKIST specific functionality, which is still in ``dkist.io.file_manager``). (`487 <https://github.com/DKISTDC/dkist/pull/487>`__)
- ``dkist.io.file_manager.BaseFileManager`` has been renamed ``dkist.io.dask.striped_array.FileManager``.
In addition the DKIST specific functionality that was implemented in ``dkist.io.file_manager.FileManager`` is now in the ``dkist.io.file_manager.DKISTFileManager``, which now instead of being a subclass of ``BaseFileManager`` now encapsulates a ``dkist.io.dask.striped_array.FileManager`` object (or similar object). (`487 <https://github.com/DKISTDC/dkist/pull/487>`__)


Features
--------

- Add support to `dkist.TiledDataset` for mosaic datasets with missing tiles or where tiles are irregularly arranged.
This includes adding a new `dkist.TiledDataset.mask` which is a numpy array, where `True` values are masked. (`487 <https://github.com/DKISTDC/dkist/pull/487>`__)


Bug Fixes
---------

- Fix issue issue causing Globus to intermittently fail after auth. (`533 <https://github.com/DKISTDC/dkist/pull/533>`__)
- Dataset inventory is now refreshed before downloading FITS files with globus to ensure that any data which has been moved at the data center is still downloaded correctly. (`539 <https://github.com/DKISTDC/dkist/pull/539>`__)


Improved Documentation
----------------------

- Added an example of plotting the bounding boxes of VBI datasets on AIA data. (`492 <https://github.com/DKISTDC/dkist/pull/492>`__)


Trivial/Internal Changes
------------------------

- A new version of the ASDF schema and tag for `dkist.TiledDataset`, v1.2.0 is added, along with a new dkist manifest v1.4.0. (`487 <https://github.com/DKISTDC/dkist/pull/487>`__)

===================

Features
--------

- Adds an `overwrite` keyword to `download_all_sample_data()` to force downloading new data. (`523 <https://github.com/DKISTDC/dkist/pull/523>`__)


Improved Documentation
----------------------

- Add a how-to guide explaining how to replace outdated dataset metadata files. (`531 <https://github.com/DKISTDC/dkist/pull/531>`__)


Trivial/Internal Changes
------------------------

- Update hashes for `TiledDataset.plot()` figure tests. (`523 <https://github.com/DKISTDC/dkist/pull/523>`__)

===================

Backwards Incompatible Changes
------------------------------

- This release of ``dkist`` includes both scheduled bumps to our dependencies as well as a special bump of gwcs to version 0.24, and associated dependencies.
The version upgrade to gwcs 0.24 fixes the inverse transform (world to pixel) for VISP WCSs. (`507 <https://github.com/DKISTDC/dkist/pull/507>`__)

* Python >= 3.11
* dask >= 2023.2
* gwcs >= 0.24
* matplotlib >= 3.7
* ndcube >= 2.1
* numpy >= 1.25
* parfive >= 2.1
* sunpy >= 5.0.7
* asdf >= 3.3
* asdf-astropy >= 0.5
* asdf-coordinate-schemas >= 0.3
* asdf-transform-schemas >= 0.5
* asdf-wcs-schemas >= 0.4


Features
--------

- Add a ``figure=`` keyword argument to `TiledDataset.plot` and make it default to the current figure. (`491 <https://github.com/DKISTDC/dkist/pull/491>`__)
- Add `swap_tile_limits` kwarg to `TiledDataset.plot`.
This option allows the user to invert plot limits on either axes to account for WCS values that decrease compared to the pixel axes. (`504 <https://github.com/DKISTDC/dkist/pull/504>`__)
- Update grid orientation of `TiledDataset.plot`.
The grid now has MAXIS1 columns and MAXIS2 rows where MINDEX1 corresponds to column and MINDEX2 corresponds to row.
Additionally, the origin for the grid is now in the lower-left as opposed to the upper-left. (`504 <https://github.com/DKISTDC/dkist/pull/504>`__)
- ``TiledDataset`` now has a ``.meta`` dictionary like that of ``Dataset``. (`513 <https://github.com/DKISTDC/dkist/pull/513>`__)
- History of the ADSF file, such as versions of packages and extensions used when writing it are now exposed ``TiledDataset.meta["history"]`` and ``Dataset.meta["history"]``. (`513 <https://github.com/DKISTDC/dkist/pull/513>`__)


Bug Fixes
---------

- Improve the ASDF detection code so out of date ASDF filenames generated by the DKIST data center are skipped if a newer filename is present. (`503 <https://github.com/DKISTDC/dkist/pull/503>`__)


Improved Documentation
----------------------

- Update the :ref:`dkist:examples:vbi-reproject` example for improvements in ``TiledDataset``. (`491 <https://github.com/DKISTDC/dkist/pull/491>`__)
- Add an examples section for tutorial style self-contained examples. (`491 <https://github.com/DKISTDC/dkist/pull/491>`__)
- Add a how-to guide on redownloading the sample data with Fido / Globus. (`491 <https://github.com/DKISTDC/dkist/pull/491>`__)

==================

Bug Fixes
---------

- Fix some small issues with `Dataset.__repr__`. (`479 <https://github.com/DKISTDC/dkist/pull/479>`__)
- Fix path errors with Globus transfers on windows. (`489 <https://github.com/DKISTDC/dkist/pull/489>`__)


Trivial/Internal Changes
------------------------

- Add tests for cropping datasets by world coords (`310 <https://github.com/DKISTDC/dkist/pull/310>`__)
- Fix small bug which caused `ds.flat` to break if not indexed. (`475 <https://github.com/DKISTDC/dkist/pull/475>`__)


Performance Improvements
------------------------

- Improve the performance of the ``TiledDataset`` ``repr`` and ``str``. (`467 <https://github.com/DKISTDC/dkist/pull/467>`__)

==================

Features
--------

- `dkist.net.transfer_complete_datasets` will now only create one Globus task for all datasets it downloads. (`340 <https://github.com/DKISTDC/dkist/pull/340>`_)
- Add a FileManager property to TiledDataset for tracking files more easily. (`437 <https://github.com/DKISTDC/dkist/pull/437>`_)
- Improve performance of computing arrays when using non-thread or sync dask schedulers, by reducing the amount of data needed to be sent to each worker. (`455 <https://github.com/DKISTDC/dkist/pull/455>`_)


Bug Fixes
---------

- Fix a bug with `dkist.net.transfer_complete_datasets` where a length one ``UnifiedResponse`` would cause an error. (`340 <https://github.com/DKISTDC/dkist/pull/340>`_)
- Fix mark argument parsing in the new pytest hook. (`440 <https://github.com/DKISTDC/dkist/pull/440>`_)
- Minor tweak to correct indexing of >4D datasets. (`453 <https://github.com/DKISTDC/dkist/pull/453>`_)


Trivial/Internal Changes
------------------------

- Add `--ds` and `--tiled-ds` CLI options to allow passing in datasets for use with the test suite.
The dataset given with `--ds` is passed to tests marked with `accept_cli_dataset` and those given with `--tiled-ds` are passed to tests marked with `accept_cli_tiled_dataset`. (`439 <https://github.com/DKISTDC/dkist/pull/439>`_)
- Minor updates to `TiledDataset.plot()` for working with more complex arrangements of tiles. (`441 <https://github.com/DKISTDC/dkist/pull/441>`_)
- Fix failure to fetch CLI options in publish build. (`447 <https://github.com/DKISTDC/dkist/pull/447>`_)

==================

Features
--------

- Add various features for easier inspection of `TiledDataset`:
- `__repr__` method to output basic dataset info;
- `tiles_shape` property to access data array shape for each individual tile;
- `slice_tiles()` method to apply the same slice to all datasets. (`402 <https://github.com/DKISTDC/dkist/pull/402>`_)
- Add `TiledDataset.plot()` quicklook method. (`408 <https://github.com/DKISTDC/dkist/pull/408>`_)
- Add CircleCI config to enable figure comparison testing with pytest_mpl. (`415 <https://github.com/DKISTDC/dkist/pull/415>`_)


Trivial/Internal Changes
------------------------

- Add some more benchmarks to track performance of more parts of the user tools. (`387 <https://github.com/DKISTDC/dkist/pull/387>`_)
- Add benchmarks for slicing and computing datasets. (`394 <https://github.com/DKISTDC/dkist/pull/394>`_)
- Added two partial datasets to `dkist.data.sample` for documentation and testing. (`421 <https://github.com/DKISTDC/dkist/pull/421>`_)
- replace usages of ``copy_arrays`` with ``memmap`` for ``asdf>=3.1.0`` (`422 <https://github.com/DKISTDC/dkist/pull/422>`_)
- Update Dataset representation for better readability. (`431 <https://github.com/DKISTDC/dkist/pull/431>`_)
- SKip a failing test in the release publish build. (`434 <https://github.com/DKISTDC/dkist/pull/434>`_)