------------------
General
^^^^^^^
New Features
^^^^^^^^^^^^
- ``AstrometricCorrections`` implemented a simultaneous fit for ``m`` and ``n``
within the uncertainty correction routine, instead of fitting for each
uncertaintiy separately and fitting for m and n after the fact. [67]
- Added ``SNRMagnitudeRelationship`` as a subclass of ``AstrometricCorrections``
to run solely the SNR-magnitude derivation part of the larger astrometric
solutions pipeline, in cases where the astrometry of a catalogue is trustworthy
but we still require the signal-to-noise ratio of sources at a given
brightness. [67]
- B-V reddening calculator added directly through ``SFDEBV``, using Schlegel,
Finkbeiner & Davis (1998), replacing the original NED website lookup call. [67]
- Added chunk post-processing, removing duplicate sources where in the "halo" of
a particular region, if desired. [58]
- ``CrossMatch`` can now generate output csv files during the matching process if
``make_output_csv`` is set to ``True``. [58]
- Added new algorithm, based on the assumption that objects within a photometric
catalogue were fit with PSF photometry in the sky background dominated regime,
where noise is constant, extending Plewa & Sari (2018, MNRAS, 476, 4372). This
can then be combined with the original aperture photometry/photon-noise
dominated case from Wilson & Naylor (2018, MNRAS, 481, 2148) using
signal-to-noise ratio as a measure of the weight to apply to each
algorithm. [50]
- Added ``fit_astrometry`` and ``AstrometricCorrections`` to allow for fitting
well-understood datasets against one another to account for systematic
astrometric uncertainties not present in the photometric catalogues as
given. [50]
- Added ``derive_psf_auf_params`` and ``FitPSFPerturbations`` to calculate the
parameters necessary to fit for the PSF photometry, sky background-dominated
algorithm for perturbation due to unresolved contaminant objects. [50]
- ``csv_to_npy`` now has the option to pre-process astrometric uncertainties
based on ``AstrometricCorrections`` outputs. [50]
- ``npy_to_csv`` now has the option to include within the final output .csv
tables made from cross-match results the pre-processed, updated astrometric
uncertainties that result from ``AstrometricCorrections``. [50]
- Added MPI parallelisation and checkpointing. [49]
- Added option to disable use of memory-mapped files for internal arrays.
Reduces I/O operations at the cost of increased memory consuption. [49]
- Inclusion of galaxy count model, used in the generation of perturbation
AUF components. [41, 44]
- Creation of initial Galactic proper motion model, for inclusion within the
cross-match framework in a future release. [39]
- Added additional data outputs to ``counterpart_pairing``: match separations, as
well as the nearest neighbour non-match for each source within a given island,
and its eta/xi and average contamination derived values. [37]
- Added functionality to convert .csv files to internal files used in the
matching process, and create output .csv files from the resulting merged
datasets created as a result of the matching. [34]
- Added option to include the full perturbation AUF component, based on
simulated Galactic source modelling. [27]
- Added options to photometric routines to create full photometry-based
likelihood and prior, or just use the photometric prior and use the naive
equal-probability likelihood. [25]
Bug Fixes
^^^^^^^^^
- In rare cases ``G`` can be incorrectly negative calculated from
``find_single_island_prob``, and now gets a threshold low-but-positive value
set in these instances. [67]
- Fixed issue reading ``local_N`` when ``compute_local_density`` is used in
combination with no memmapping. [67]
- Fixed "fire extinguisher" priors and likelihoods, used in cases where both c
and f are zero, not cancelling to one in the likelihood ratio. [67]
- Fixed cases where wavelength range of filter response can cause a non-shifted
spectrum to fail due to non-padding in ``create_galaxy_counts``. [67]
- ``create_single_perturb_auf`` raises an error if the number of simulated
sources in a given sightline is insufficient to draw reliable number density
measurements from. [67]
- ``make_perturb_aufs`` checks for ``compute_local_density`` and
``use_memmap_files`` before loading local normalising density binary
files, and otherwise uses pre-computed in-memory array values. [67]
- ``input_npy_folder`` correctly set as ``None`` if passed as such through
the input parameter file. [67]
- If ``use_memmap_files`` is ``False`` but any of the flags for running steps
of the cross-match process are also ``False`` a warning will be raised and
the run flags set to ``True``, since there are no fallback files to load. [67]
- Calls to ``make_tri_counts`` and ``create_galaxy_counts`` changed to use a
grid of extinction vectors within the chosen field of regard to better
handle differential reddening instead of relying on a single Av at a
particular precise set of coordinates. [67]
- ``make_tri_counts`` gains ``brightest_source_mag`` and ``density_mag``
keywords, returning ``num_bright_obj``. [67]
- Convenience function ``min_max_lon`` added, to account for issues where
the minimum and maximum longitude in a given region of space could sit either
side of the 0-360 boundary, and hence the usual x < l < y conditions would
fail. [67]
- ``counterpart_pairing_fortran``'s ``factorial`` function changed from
calculating N! to directly calculating N! / (N-M)! as the previous function
had the potential to overflow unnecessarily. [67]
- Added ``outfolder`` to ``trilegal_webcall`` to avoid a parallelisation race
condition with saving outputs. [67]
- ``mag_h_params`` renamed to ``snr_mag_params`` to ensure commonality of the
reference and parameter without the codebase. [62]
- ``AstrometricCorrections`` makes a correctly multi-magnitude SNR model
array. [59]
- ``npy_to_csv`` expected contamination probability arrays to be transposed from
their ``CrossMatch`` output shape, but they now correctly assume
fortran-ordering. [58]
- Pass ``tri_maglim_bright``, ``tri_maglim_faint``, ``tri_num_bright``, and
``tri_num_faint`` through to ``make_perturb_aufs`` in ``CrossMatch`` call. [56]
- Replaced ``datetime.strptime`` in the ``CrossMatch`` constructor with a
string ``split`` to fix a crash when given walltime is greater than
``24:00:00``. [52]
- Updated ``fit_gal_flag`` keyword as passed through to ``make_perturb_aufs``
incorrectly using ``self.a_fit_gal_flag`` when running catalogue "b" AUF
component generation. [50]
- Corrected issue where ``local_N`` wasn't having entries saved to memmapped
array in ``make_perturb_auf``. [38]
- Updated ``local_N`` to keep the local densities of catalogue in each filter,
instead of overwriting each time. [38]
- Set minimum density of local sources to one source in the region in question,
instead of allowing for a floor of zero density, to avoid issues with AUF
simulations. [38]
- Avoided re-using the same random seed in each density-magnitude combination
during AUF simulations. [38]
- Changed limits on photometric likelihoods and priors to avoid cases where
both field and counterpart posteriors are zero, and hence no matches can be
made in a given island. [38]
- Fixed issue in ``source_pairing`` where incorrect island lengths could be used
for field and counterpart arrays. [38]
- Fixed ordering issue with ``acontamprob`` and ``bcontamprob`` in
``source_pairing``. [38]
- Fix to issue with np.where test in ``test_counterpart_pairing`` causing incorrect
failure to match probabilities. [36]
- Fixes to various minor typos in variables in the cross-match workflow. [32]
- Allow for the non-existence of a TRILEGAL simulation in any folder, and download
new files even if ``tri_download_flag`` was set to ``False``. [32]
- Save local normalising densities to file if ``compute_local_density`` was set
to ``True``, to allow for its non-calculation in the future. [32]
- Overload ``compute_local_density`` if it is set to ``False`` and the file
storing local densities does not exist. [32]
- ``create_single_perturb_auf`` corrected to run on a single filter, as its input
intended, instead of looping through all filters. [32]
- Removed final right-hand bin from consideration when identifying which magnitude
bin each source should be assigned to in ``create_c_and_f``, to avoid an issue
where sources of exactly the last bin are assigned outside the allowed range
of indices. [32]
- Fixed inefficiencies in both group sources and perturbation AUF creation runtime,
significantly improving the speed of those parts of a cross-match. [31]
- Corrected an error in ``tests.generate_random_data``, where only one catalogue
had its source uncertainties simulated. [23]
API Changes
^^^^^^^^^^^
- ``use_photometric_uncertainties`` added as an optional keyword to
``AstrometricCorrections``, allowing for the use of photometric instead of
astrometric uncertainties as a slicing to determine best-fit astrometric
uncertainties. [67]
- ``csv_to_npy``, ``npy_to_csv``, and ``rect_slice_csv`` now expect filenames to
include their extensions. [67]
- ``mn_to_radec`` added to ``csv_to_npy``, to convert any astrometric correction
array coordinates to match catalogue coordinates, with analogous variable
``cat_in_radec``, which now controls the coordinate system of the data. [67]
- Explicitly load save-state data into ``CrossMatch`` and/or ``StageData`` as
appropriate to match ``use_memmap_files`` boolean in both configurations. [67]
- Added ``compute_snr_mag_relation`` as expected keyword into ``CrossMatch``
for each catalogue, calling ``SNRMagnitudeRelationship`` if ``True``. [67]
- Added checks for ``correct_astro_save_folder``, ``csv_cat_file_string``,
``pos_and_err_indices``, ``mag_indices``, and ``mag_unc_indices`` in the case
of ``compute_snr_mag_relation`` as well as ``correct_astrometry``. [67]
- Changed dependencies of ``snr_mag_params_path`` to include the requirement
for just calculating SNR-mag relationships. [67]
- ``csv_cat_file_string``, ``match_out_csv_name``, and ``nonmatch_out_csv_name``
now all explicitly require file extensions, generally ``.csv``. [67]
- Removed ``dens_mag`` as input into ``CrossMatch``, and ``density_mags`` from
``make_perturb_aufs``. [67]
- Changed the requirements of ``al_avs`` in ``make_perturb_aufs`` to not require
``fit_gal_flags``. [67]
- ``gal_al_avs`` is now required if ``include_perturb_auf`` or
``correct_astrometry`` is ``True``, instead of being tied to
``fit_gal_flag``, as all other galaxy-related inputs are. [67]
- ``create_galaxy_counts`` takes ``al_grid`` rather than ``al_inf``, which is
now a list of floats rather than a singular float value, using an average
galaxy count distribution across all extinctions in ``al_grid``. [67]
- ``AstrometricCorrections``'s ``create_densities`` and ``create_distances``
always save binary files instead of checking for their non-existence, due to
the re-structuring of the looping of sightlines and pipeline steps. [67]
- ``check_b_only`` flag added to ``make_ax_coords`` function within
``AstrometricCorrections`` for cases where we only need to run a sub-set
of functions on one catalogue, instead of the two-sided approach for the full
suite of astrometric correction tools. [67]
- ``dens_search_radius`` changed to degrees, instead of arcseconds, in
``AstrometricCorrections``, to match ``CrossMatch`` requirements. [67]
- Removed ``bright_mag`` from input to ``AstrometricCorrections``. [67]
- Added ``AV`` and ``sigma_AV`` as input keywords to
``download_trilegal_simulation`` and ``get_trilegal`` to allow for the manual
passing of specific V-band extinctions to API call. If not passed to it, a
value is still calculated in ``get_trilegal``, and ``AV`` is returned by the
function. [67]
- Added expected area of TRILEGAL simulation as keyword to
``download_trilegal_simulation``. [67]
- ``download_trilegal_simulation`` and ``get_trilegal`` have been re-arranged to
move the try-except loop out of the API call function and into the larger
function. ``get_trilegal`` will therefore either return an API call or fail,
without trying to fetch. [67]
- ``trilegall_webcall`` returns either ``timeout`` or ``good``, allowing for the
re-starting of failed API calls due to e.g. the remote server being busy. [67]
- In ``AstrometricCorrections``, ``triname`` now requires either one or two
``{}`` Python string formats, depending on ``coord_or_chunk``. [62]
- All ``recreate`` flags all removed from ``AstrometricCorrections``, which now
loops on a per-sightline basis instead of using per-step loops. [62]
- Added ``n_pool`` as input to ``CrossMatch`` to control the number of threads used
in ``multiprocessing`` calls. [62]
- Added parameters ``correct_astrometry``, ``best_mag_index``, ``nn_radius``,
``correct_astro_save_folder``, ``csv_cat_file_string``,
``ref_csv_cat_file_string``, ``correct_mag_array``, ``correct_mag_slice``,
``correct_sig_slice``, ``pos_and_err_indices``, ``mag_indices``,
``mag_unc_indices``, ``chunk_overlap_col``, and ``best_mag_index_col`` as
catalogue-level inputs to ``CrossMatch`` to allow for astrometric corrections
through ``AstrometricCorrections`` directly before a cross-match. [62]
- Requirements for ``num_trials``, ``d_mag``, ``run_fw_auf``, ``run_psf_auf``,
``psf_fwhms``, ``dens_mags``, ``snr_mag_params_path``, ``download_tri``,
``tri_set_name``, ``tri_filt_names``, ``tri_filt_num``, ``tri_maglim_faint``,
``tri_num_faint``, ``dens_dist``, ``dd_params_path``, ``l_cut_path``,
``gal_wavs``, ``gal_zmax``, ``gal_nzs``, ``gal_aboffsets``,
``gal_filternames``, and ``gal_al_avs`` inputs to ``CrossMatch`` changed to
either require ``include_perturb_auf`` (and lower-level input criteria) or
``correct_astrometry``. [62]
- Removed expectation of parameters ``tri_num_bright`` and ``tri_maglim_bright`` from
``CrossMatch`` input parameter files. Currently only expect the "faint" versions
due to limits with requesting significant numbers of bright TRILEGAL objects. [61]
- Added ``tri_num_faint`` to ``AstrometricCorrections`` to control the resolution of
TRILEGAL simulations used in fitting for astrometry systematics, and removed
``maglim_b`` from expected keywords, limiting the number of TRILEGAL simulations
to just one across the entire dynamic range, as with ``CrossMatch``. [61]
- Added new keyword ``pregenerate_cutouts`` to ``AstometricCorrections``, indicating
whether sightlines can be assumed to be pre-made or if they should be able to be
made on-the-fly as part of the correction-fitting process. [59]
- ``AstrometricCorrection`` had ``cutout_area`` and ``cutout_height``, as well as
``a_cat_func`` and ``b_cat_func``, made optional keywords. [59]
- ``AstrometricCorrections`` now takes keyword input ``coord_system`` to determine
whether coordinates fed into the class are in equatorial or galactic coordinates,
handling conversions and consistency where necessary. Additionally, keywords were
given more general names reflecting this change and now the class requires
``ax1_mids``, ``ax2_mids``, and ``ax_dimension`` instead of ``lmids``, ``bmids``,
or ``lb_dimension``. [59]
- ``CrossMatch`` now expects ``snr_mag_params_path`` rather than
``mag_h_params_path``, and ``CrossMatch`` loads and ``AstrometricCorrections``
saves ``snr_mag_params.npy`` as the file containing the magnitude-SNR
correlation parameterisation. [59]
- Added new input keywords to ``AstrometricCorrections`` for the indexes of position
and magnitudes and their uncertainties, along with the most complete magnitude to
use in construction of any updates to astrometry of a given catalogue. [59]
- ``AstrometricCorrections`` accepts three new keywords: ``npy_or_csv``,
``coords_or_chunk``, and ``chunks`` which allow for the specification of file
type and structure of small sightlines used to check astrometry of a
catalogue. [59]
- ``npy_to_csv`` always requires two nested lists when using ``extra_col_*_lists``,
rather than allowing a singular ``None``. The default is now ``[None, None]`` for
the passing of no extra columns to be propagated to the output csv file. [58]
- ``tri_maglim_bright``, ``tri_maglim_faint``, ``tri_num_bright``, and
``tri_num_faint`` are only required if ``tri_download_flag`` is ``True``. [56]
- ``tri_filt_num``, ``tri_set_name``, and ``auf_region_frame`` updated to be
necessary inputs into ``make_perturb_aufs`` even if ``tri_download_flag``
is not set. [56]
- Added ``run_fw_auf``, ``run_psf_auf``, ``mag_h_params_path``,
``tri_maglim_bright``, ``tri_maglim_faint``, ``tri_num_bright``, and
``tri_num_faint`` as required input parameters to ``CrossMatch`` if
``include_perturb_auf`` is ``True``. [50]
- Added ``tri_maglim_bright``, ``tri_maglim_faint``, ``tri_num_bright``,
``tri_num_faint``, ``run_fw``, ``run_psf``, ``dd_params``, ``l_cut``, and
``mag_h_params`` as optional inputs to ``make_perturb_aufs``. [50]
- Added ``dd_params_path`` and ``l_cut_path`` as required input parameters if
``include_perturb_auf`` and ``run_psf_auf`` are both ``True``. [50]
- Removed ``dm_max`` as an input to ``CrossMatch``, now being calculated based
on secondary perturber flux vs primary noise and chance of zero perturbers
in ``_calculate_magnitude_offsets``. Also removed as input to
``make_perturb_aufs``. [50]
- ``csv_to_npy`` has ``process_uncerts``, ``astro_sig_fits_filepath``, and
``cat_in_radec`` as optional input parameters. [50]
- ``npy_to_csv`` added ``input_npy_folders`` as an input parameter. [50]
- Removed ``joint_file_path``, ``cat_a_file_path`` and ``cat_b_file_path``
from ``CrossMatch`` constructor and added ``chunks_folder_path``,
``use_memmap_files``, ``resume_file_path``, ``walltime``, ``end_within``,
and ``polling_rate``. [49]
- Added ``use_memmap_files`` as input parameter to relevant functions. [49]
- Added ``StageData`` class to ``misc_functions``. [49]
- Added ``npool`` as input parameter to ``make_island_groupings``. [38]
- Removed ``npool`` as input parameter to ``source_pairing``. [38]
- Added extra columns derived in ``counterpart_pairing`` to output datafiles in
``npy_to_csv``. [37]
- ``npy_to_csv`` now has ``extra_col_name_lists``, allowing for the inclusion of
extra columns from the original catalogue .csv file to be passed through to the
output merged datafiles. [37]
- Moved several functions (``_load_single_sky_slice``, ``_load_rectangular_slice``,
``_lon_cut``, ``_lat_cut``) out of individual Python scripts into
``misc_functions`` to generalise their use in the codebase. [27]
- Removed ``norm_scale_laws`` as an input to catalogue configuration files. [27]
- Added ``dens_mags``, ``num_trials``, ``dm_max``, ``d_mag``, and
``compute_local_density`` as inputs to the joint and catalogue-specific
configuration files [27]
- Added ``int_fracs`` as an input to the joint configuration file for a
cross-match. [25]
Other Changes
^^^^^^^^^^^^^
- Updated documentation to reflect previous improvements to codebase, and add further
introductory and explanatory material. [54]
- Changed ``_make_chunk_queue`` to return a queue ordered by file size in bytes
and improve load balancing in MPI parallelised jobs. [52]
- Added ``matplotlib`` as a dependency, and explictly defined ``pytest-cov`` as a
test dependency. [50]
- Added ``mpi4py`` as a dependency [49]
- Added ``skypy`` and ``speclite`` as dependencies. [41]
- Improved github actions matrix testing coverage. [40]
- Added ``pandas`` as a dependency. [34]
- Updates to documentation to reflect the relaxing of photometric likelihood and
perturbation AUF component options. Other minor changes to documentation
layout. [30]
- GitHub Actions will only run remote data dependent tests (those marked with
``pytest.mark.remote_data``) on a pull request merge. [27]
- Added ``astropy`` as a dependency. [27]