Drizzlepac

===================

This version includes all the functionality needed to generate
source catalogs, both point source and extended (segment) source
catalogs, during single-visit mosaic (SVM) processing. In fact,

- Updated code to work with Python >= 3.7
- **GAIAeDR3** catalog now the initial catalog of choice for a posteriori alignment
during standard pipeline processing, as well as for SVM/MVM processing.
- SVM/MVM processing will loop over catalogs, fit methods and fit geometries in
looking for a successful fit, using the first successful fit it computes.

- CATALOGS used: **GAIAeDR3**, **GSC242**, **2MASS** (in this order)
- methods: relative, image-by-image
- geometries: **rscale**, **rshift**, **shift** (each with different minimum cross-matches)

- SVM processing will always generate both point source and extended source catalogs, even
if the catalogs contain no rows of sources and measurements.

- point source catalog will be generated using TinyTim PSF-based detection
- extended source (segment) catalog will only have sources larger
than the PSF kernel deblended.
- catalog columns will closely resemble the Hubble Legacy Archive (HLA) catalogs columns

- Grism/Prism exposures do not get aligned, but instead get the WCS correction from direct images
- Added logic to handle visits where there are only Grism/Prism exposures with no direct images
- ``S_REGION`` keyword:

- added to FLT/FLC file headers
- revised region computation to match closely the actual exposure footprint within mosaic

- Always runs ``updatewcs`` on input files to insure pipeline-default WCSs are always present

- Add ``WCSNAME=OPUS`` if no ``IDCTAB`` WCS was created by ``updatewcs`` (``NGOODPIX=0``, ...).

These changes, and additional significant bug fixes, were implemented using
the following github PRs:

- Implemented deblending of segmentation source catalogs ONLY
for sources larger than the PSF kernel. [1131]

- Insure SVM processing always generates point-source and
segmentation (extended) source catalogs, even if empty. [1129]

- Implemented an efficient single-image identifier of possible
cosmic-rays/defects, and applied it to help make image
alignment more reliable. [1129]

- Update logic for fitting between source lists to minimize/eliminate
use of fitting with less than 4 sources. [1129]

- Implemented model PSF-based point-source identification for SVM
point-source catalog generation. [903, 971, 1127]

- Removed dependence on private photutils functions while enabling
support for all photutils versions >= 1.0.0.
[1127, 1117, 1116, 1096]

- Set values for crowding, biggest source, and source
fraction for use when to use the RickerWavelet kernel and
when to deblend sources when identifying extended sources
using segmentation for the segment catalog. [1115]

- Implemented a more efficient algorithm based on Harris corner
detection for computing the ``S_REGION`` keyword for pipeline
and SVM drizzle products. [1106]

- Fix a memory corruption issue in ``interpolate_bilinear()`` in
``cdrizzleblot.c`` which could result in segfault. [1048]

- Fixed multiprocessing incompatibility with ``Python >= 3.8``. [1101]

- Add support for environment variable switch, ``PIPELINE_RESET_IDCTAB``,
to ``runastrodriz`` which will automatically reset ``IDCTAB``
in FLT/FLC files if different from ``IDCTAB`` in RAW files. [1046]

- Update documentation based on revisions to the code.
[941, 947, 953]

- Update default astrometry catalogs for alignment to try alignment to
the ``GAIA eDR3`` catalog first. [986, 1012]

- Enable user epoch selection when a user requests a GAIA catalog from
the astrometry catalog web service. [1006]

- Insure that ``HDRNAME`` is always valid for updated WCS solutions. [966]

- Revised ``S_REGION`` keyword value to reflect actual outline of chips in
drizzle products. [951]

- Sky Subtraction step will automatically downgrade from ``match`` to ``localmin``,
and from ``globalmin+match`` to ``globalmin`` when sky matching runs into an
Exception. [ 1007]

- Changed to insure that ``EXTNAME`` and ``EXTVER`` are always removed from
simple FITS drizzle product headers. [954]

- Changed to insure that all the distortion keywords (e.g., ``TDD*``, ``D2IM*``,...)
are removed from from the output drizzle product headers [954].

- Set a common active WCS for direct as well as corresponding Grism/Prism images [929, 946]

- Fix a bug in ``tweakback`` that may cause incorrect "updated" WCS to be
picked up from the drizzled image. [913]

- Added ``DRIZPARS`` keyword to final output drizzle product primary header
to document the name of the associated trailer file. [934, 1078]

In addition, numerous changes were made to insure this code stayed
compatible with numpy versions > 1.20 and astropy versions > 4.1.

Updates to the ``STWCS`` package version >= 1.6.0 also translated to
the following changes to the Drizzlepac processing:
- Insure HDRNAME keyword is never empty
- Remove duplicate headerlet extensions when running updatewcs
- Compute new a priori WCS solutions for new IDCTAB not already in astrometry database

***API Changes:***

**imageObject.py:**
- **class imageObject**: Added parameter ``output`` to enable determination
of rootname for use in processing of each detector.

**adrizzle.py:**
- **drizSeparate**: Added optional parameter ``logfile`` for specifying
what file to use for log messages.
- **drizFinal**: Added optional parameter ``logfile`` for specifying
what file to use for log messages.

**wcs_functions.py:**
- Removed ``hdulist`` as parameter from ``get_hstwcs``.

**haputils/analyze.py:**
- **analyze_data**: Added parameter ``type`` to customize logic for SVM
processing.

**haputils/astrometric_utils.py:**
- **retrieve_observation**: Added parameter ``product_type`` to allow for selection of
type of products to be returned; pipeline, HAP, or both.

**haputils/make_poller_files.py:**
- New function ``generate_poller_file`` added to create inputs for SVM processing
from files on disk.

**haputils/processing_utils.py:**
- New function ``find_footprint`` added to determine corners of all chips
in an image for computation of ``S_REGION`` keyword.
- New function ``interpret_sregion`` added to convert ``S_REGION`` keyword
value into list of RA/Dec points for visualization.

===================

- Fix problems with testing code for this package [940]

==================

This version provides the first operational implementation of the single-visit
mosaic processing used to create the single-visit mosaics products.

- revise naming convention for the StaticMask file so that it has a
dataset-specific name instead of a generic common name. [876]

- Update ``runastrodriz`` to work under Windows while adding documentation
to tell the user to run with ``num_cores`` set to 1. [794]

- Fixed a bug in ``TweakReg`` due to which ``TweakReg`` would crash when
``updatehdr`` was set to `False`. [801]

===================

A number of changes have been implemented to either correct problems or
improve the processed results. The most significant of the changes are:

- rscale only used for alignment.
- a minimum of 6 sources now gets used for alignment
- no proper motions used in astrometric (GAIA) catalog when attempting a posteriori fitting
- chip-to-chip alignment errors were corrected


In addition to a few dozen bug fixes, the following updates to the algorithms
were also implemented.

- Simplified the logic in ``tweakreg`` for deciding how to archive primary WCS
resulting in a reduction of duplicate WCSes in image headers. [715]

- Added polynomial look-up table distortion keywords to the list of distortion
keywords used by ``outputimage.deleteDistortionKeywords`` so that
distortions can be removed from ACS images that use ``NPOLFILE``.
This now allows removal of alternate WCS from blotted image headers. [709]

- Added ``rules_file`` parameter to AstroDrizzle to enable use of custom
files in pipeline processing. [674]

- Only apply solutions from the astrometry database which were non-aposteriori
WCS solutions as the PRIMARY WCS. This allows the pipeline to compare the
true apriori WCS solutions (e.g., GSC or HSC WCSs) to aposteriori solutions
computed using the latest distortion-models and alignment algorithms being
used at the time of processing. [669]

- Verification using a similarity index gets reported in the trailer file and
does not get used as a Pass/Fail criteria for alignment. [619]

- If verification fails for either pipeline-default or apriori solution, reset
cosmic-ray(CR) flag (4096) in DQ arrays. This will allow subsequent attempt to
align the images to not be impacted by potentially mis-identified CRs that most
likely blanked out real sources in the field. As a result, the image alignment
process became more robust when computing the aposteriori alignment. [614]

- Fix a crash in ``tweakreg`` when finding sources in very large images
due to a bug in ``scipy.signal.convolve2d``. [670]

- Fix a bug in ``tweakreg`` due to which the number of matched sources needed to be
*strictly* greater than ``minobj``. Now the minimum number of matched sources
maust be *at least* equal or greater than ``minobj``. [604]

- Fix a crash in ``tweakreg`` when ``2dhist`` is enabled and ``numpy``
version is ``1.18.1`` and later. [583, 587]

- Update calibrated (FLC/FLT) files with RMS and NMATCH keywords when it successfully
aligns the data to GAIA using the a posteriori fit. Headerlet files for this fit
which already have these keywords are now retained and provided as the final output
headerlets as well. [555]

- Insure HDRNAME keyword gets added to successfully aligned FLC/FLT files. [580]

- Fix problem with 'tweakback' task when trying to work with updated WCS names. [551]

- Fix problems found in processing data with NGOODPIX==0, DRC files not getting
generated for singletons, alignment trying to use a source too near the chip edge,
catch the case were all inputs have zero exposure time, lazily remove alignment
sub-directories, fixed a bug in overlap computation that showed up in oblong mosaics,
recast an input to histogram2d as int, defined default values for tables when no
sources were found. [593]

- Updated to be compatible with tweakwcs v0.6.0 to correct chip-to-chip alignment issues
in aposteriori WCS solutions. [596]

- Correctly define output drizzle product filename during pipeline processing
for exposures with 'drz' in the rootname. [523]

- Implement multiple levels of verification for the drizzle products generated
during pipeline processing (using runastrodriz); including overlapp difference
computations [520], and magnitude correlation [512].

- Replace alignimages module with O-O based align [512]

- Fix problem with NaNs when looking for sources to use for aligning images [512]

- Fixed code that selected the brightest sources to use for alignment allowing
alignment to work (more often) for images with saturated sources. [512]

- Use logic for defining the PSF extracted from the images to shrink it in each
axis by one-half for images of crowded fields to allow for more sources to be
extracted by daofind-like algorithm. This enables source finding and alignment
to work more reliably on crowded field images. [512]

- Insure all input files, especially those with zero exposure time or grism
images, get updated with the latest pipeline calibration for the distortion. [ 495]

This version also relies on updates in the following packages to get correctly
aligned and combined images with correctly specified WCS keywords:

- TWEAKWCS 0.6.4: This version corrects problems with the chip-to-chip separation
that arose when applying a single fit solution to the entire observation.

- STWCS 1.5.4: This version implements a couple of fixes to insure that use of
headerlets defines the full correct set of keywords from the headerlet for
the PRIMARY WCS in the science exposure without introducing multiple copies of
some keywords.

- Numpy 1.18: Changes in numpy data type definitions affected some of the code used
for computing the offset between images when performing aposteriori alignment
during pipeline processing and when running the 'tweakreg' task.

==================

- Fixed a bug in the ``updatehdr.update_from_shiftfile()`` function that would
crash while reading shift files. [448]

- Migration of the HAP portion of the package to an object-oriented
implemenation. [427]

- Added support for providing HSTWCS object as input to 'final_refimage'
or 'single_refimage' parameter. [426]

- Implementation of grid definition interface to support returning SkyCell
objects that overlap a mosaic footprint. [425]

- Complete rewrite of ``runastrodriz`` for pipeline processing to include
multi-level verification of alignment. [440]

====================

- Removed deprecated parameter ``coords`` from the parameter list of
``pixtopix.tran()`` function. [406]

- Modified the behavior of the ``verbose`` parameter in ``pixtopix.tran()``
to not print coordinates when not run as a script and when ``output``
is `None`. [406]

- Fixed a compatibility issue in ``tweakutils`` that would result in crash in
``skytopix`` when converting coordinates in ``hms`` format. [385]

- Fixed a bug in the ``astrodrizzle.sky`` module due to which sky matching
fails with "Keyword 'MDRIZSKY' not found" error when some of the
input images do not overlap at all with the other images. [380]

- Fixed a bug in the ``util.WithLogging`` decorator due to which incorrect
log file was reported when user-supplied log file name does not have ``.log``
extension. [365]

- Fixed a bug introduced in 364 returning in ``finally`` block. [365]

- Improved ``util.WithLogging`` decorator to handle functions that return
values. [364]

- Fixed a bug in the automatic computation of the IVM weights when IVM
was not provided by the user. [320]

- Fixed a bug in the 2D histogram code used for estimating shifts for
catalog pre-matching. This may result in better matching. [286]

- Now ``tolerance`` (in ``tweakreg``) is no longer ignored when ``use2dhist``
is enabled. [286]

- Fixed VS compiler errors with pointer artithmetic on void pointers. [273]

- Fix logic so that code no longer tries to update headers when no valid fit
could be determined. [241]

- Fixed a bug in the computation of interpolated large scale flat field
for STIS data. The bug was inconsequential in practice.
Removed the dependency on ``stsci.imagemanip`` package. [227]

- Removed the dependency on ``stsci.ndimage`` (using ``scipy`` routines
instead). [225]

- Added ``'Advanced Pipeline Products'`` alignment code to ``drizzlepac``
package. Enhance ``runastrodriz`` to compute and apply absolute astrometric
corrections to GAIA (or related) frame to images where possible.
[200, 213, 216, 223, 234, 235, 244, 248, 249, 250, 251,
259, 260, 268, 271, 283, 294, 302]

- Add computation and reporting of the fit's
`Root-Mean-Square Error (RMSE) <https://en.wikipedia.org/wiki/Root-mean-square_deviation>`_
and `Mean Absolute Error (MAE) <https://en.wikipedia.org/wiki/Mean_absolute_error>`_.
[210]

- Replaced the use of ``WCS._naxis1`` and ``WCS._naxis2`` with
``WCS.pixel_shape`` [207]

- Removed support for Python 2. Only versions >= 3.5 are supported. [207]

- Use a more numerically stable ``numpy.linalg.inv`` instead of own matrix
inversion. [205]

- The intermediate fit match catalog, with the name ``_catalog_fit.match``
generated by ``tweakreg`` now has correct RA and DEC values for the sources
after applying the fit. [200, 202]

- Simplify logic for determining the chip ID for each source. [200]