Sherpa

=============

This release of Sherpa is based on the CIAO 4.12 release and includes additional
bug fixes and improvements.

This version of Sherpa has been tested with Python 3.5, 3.6, and 3.7.

This release also provides support for XSPEC 12.10.1 (patch 'a' or later) in
addition to versions 12.10.0 (included in CIAO) and version 12.9.1.

Details
-------

Documentation and infrastructure fixes are not shown.

388 - Address indexing DeprecationWarning/IndexError from NumPy for PHA data with filter + ignore_bad (fix 361)

602 - Reduce integration tol
Changed integration routine tolerance from double epsilon to float epsilon
since opt routines hove tolerance of ~ 1.0e-7

606 - stop using numpy.typeNA which is now deprecated
A numpy deprecation warning was fixed by removing the usage of typeNA, which
was not documented and will be removed in a future release of numpy. The
print_fields function has been changed to include a default mapping using
the current typeNA implementation.

608 - update ciao default plotter to matplotlib
The default Sherpa plotting package has been changed to matplotlib for
CIAO users as well (it had been the default for standalone users for
several years). The update will be applied to new users when the
`sherpa` command is run. Users of previous versions would need to
edit/regenerate the file.

616 - enable run-time-option to test cache (default=True)
Capability to allow the user to turn off caching at runtime for testing
purpose.

622 - Do not create warnings about error bars in plots if no error bars are to be shown (fix 621)
Remove the The displayed errorbars have been supplied with the data or
calculated using chi2xspecvar... warnings that appear for data,
residual/ratio, and fit plots when the user has explicitly turned off
the display of errorbars (by setting the yerrorbars plot-preference
setting to False).

626 - add plot_fit_ratio and plot_bkg_fit_ratio functions
Add the plot_fit_ratio function to the sherpa.ui layer (to match
plot_fit_resid and plot_fit_delchi) and plot_bkg_fit_ratio to the
sherpa.astro.ui layer. Added tests for a number of plot types.

640 - fix 'tuple index out of range' error in the fit.est_errors method
Sherpa's Confidence class is confused if a parameter is frozen then
thawed after a fit resulting with an exception of the form: IndexError:
tuple index out of range. The error does not show up if conf is called
from the ui level

647 - fix numpy hist warnings
When plotting a PDF Sherpa now calls numpy.histogram with density=True |
False rather than normed=True|False. There is no change visible to the
user other than a warning that will not be issued anymore. normed=True
was broken for non uniform bins, but our code always produces uniform
bins, so we should never hit the problematic case.

658 - Handle files with MJD_OBS or MJD-OBS keywords (datastack module)
The sherpa.astro.datastack module has been updated to deal with files
that use the MJD-OBS keyword instead of MJD_OBS. This is only used by
the show_stack routine, which prints out several keywords from each
file in the stack.

665 - Remove python 2.7 support (fix 498)
Remove support for Python 2.7 from Sherpa. The metadata used by pip now
requires python 3.5 or higher (but not version 4),

667 - improve matplotlib plot_fit plots (order of data and model)
Changes to Zorder of plot objects drawn by matplotlib to be more readible

674 - Add link/unlink parameter tests
This PR adds three tests for sherpa's un/link commands.

677 - XSPEC: build against 12.10.1m by removing support for models
Remove support for XSrelline, XSrelline_lp, and XSrelline_lp_ext models.
These were added in 12.10.1 but removed in 12.10.1m. It does not seem
worth supporting just those versions where they were available.

680 - Remove the unused parameter in sherpa.astro.optical.LogEmission model (fix 219)
Remove the unused parameter in the sherpa.astro.optical.LogEmission
model. The LogEmission model only has four parameters (fwhm, pos, flux
and skew) consistent within the calc method. The hidden parameter limit
and it doc were removed.

682 - jointplot now respects the ratio argument (top plot is larger) with matplotlib backend
The matplotlib back end now makes the main (top) plot taller than the
secondary (bottom) plot when using the sherpa.ui.plot_fit_resid and
sherpa.ui.plot_fit_delchi routines (this also holds for the
sherpa.astro.ui variants).

683 - Remove chips from the documentation
Remove mention of ChIPS from the documentation (both on read the docs
and in the sherpa.astro.ui and sherpa.ui modules), replacing with
matplotlib where appropriate.

684 - Add default config options
Whenever an option is read from the configuration file (~/.sherpa.rc)
provide a default option in case the configuration file is missing
(or is missing this option).

685 - raise ModelErr when model using Data1DInt have overlapping bins
Adds a check to regrid make sure the integrated bins do not overlap.
The PR is a fix for issue 569

688 - Improve documentation and testing for get_source_plot when sent lo/hi arguments
Improve the documentation for the sherpa.astro.ui.get_source_plot routine,
describing how touse the result when the lo or hi arguments are sent.

691 - add warning when chips is selected but cannot be imported (rebase of 648)
Sherpa now warns the user if Chips is selected as a backend but it is
not available in the installation

692 - Allow users to override plot preferences when creating a plot
Change to allow kwargs to be specified to change plot preferences at
creation time

698 - Add warnings in documentation about masked arrays
Updates docstring to indicate that sherpa doesn't support numpy masked
arrays

701 - a fix for /data/lenin2/Test/CIAO4.11/SherpaRegressionBeta1/45
Enables the testers to turn on (or off) caching at runtime to test 444

704 - change calc_ftest from delta dof to dof (similar to XSPEC) - fix for 641
Update to make sherpa's calc_ftest provide same results as XSPEC's
ftest (fix to 641)

705 - modelCacher1d needs to have a couple of deep instead of shallow copy()
Update to make deep copy of cached function values (fix to 673)

710 - Add warning about masked array to sherpa.astro.ui.load_arrays
Updates docstring to contain warning consistent with 698

712 Minor typo in docs
Corrects a typo "is is" in the documentation.

716 Fix for plot_cdf: plot() got an unexpected keyword argument 'clearwindpw'
Recent changes broke plot_cdf()due to a typo 'clearwindpw'. This update
corrects the typo.

721 Docs: update installation notes for conda environment
Add a warning about the need to set PYTHON_LDFLAGS to ' ' on macOS when
building within an anaconda environment. This is a documentation-only
change.

725 convert a list to np.array to avoid warning messages
Fixes issue 723 (a lot of warnings while running 'resample_data()' in
CIAO4.12 Sherpa)

728 a fix cause using load_multi_arfs causes caching error
Fixes issue 717 (unable to fit if script uses load_multi_arfs/rmfs)

=============

This release of Sherpa introduces several functional improvements and bug fixes,
in particular Sherpa now has support for:
* asymmetric error bars
* PSFs with better pixel resolution than the data
* running optimization in parallel

Details
-------

630 Fix "get_int_proj does not work when recalc=True" (543)
get_int_proj did not work when recalc=True on Python 3. This has now been fixed.

615 Asymmetric Errors
Sherpa now supports asymmetric error bars. Errors can be read through a new
`load_ascii_with_errors` high level function, or through the new
`Data1DAsymmetricErrs` class. Sherpa uses bootstrap for estimating the uncertainties.

585 plot pvalue
Updates to utilize the appropriate response files (ARF and RMF) for X-ray spectra
and changes to the p_value output to 1/(number of simulations) when p_value is 0
and the number of simulations in not large enough.

596 Run optimization algorithms over multiple cores
This PR enables the user to run the optimization algorithms (DifEvo, LevMar,
and NelderMead) on multi-cores.

607 PSF rebinning (fix 43)
Sherpa now supports using a PSF with a finer resolution than 2D images. If Sherpa
detects that the PSF has a smaller pixel size than the data, it will evaluate the
model on a "PSF Space" that has the same resolution as the PSF and the same footprint as
the data, then rebin the evaluated model back to the data space for calculating the statistic.

614 Refactor data classes (fix 563, 627, 628)
Sherpa's basic data classes have been refactored and cleaned up to help
facilitating fixing bugs and implementing new features. New tests were added to reduce
the chances of introducing regressions in the future.

612 Fix 609 User Model: unable to change parameter values
A regression introduced in Sherpa 4.10.0 presented users from change
user-model parameter values through direct access. This issue has been
fixed. Several tests were added to reduce the chance of regressions in
the future.

Fix 514 - the command line tool `sherpa_smoke` in the conda package now correctly honors
command line arguments.

=============

Optimization routines and statistic methods were made more robust with several
bug fixes and improvements.

This version of Sherpa has been tested with Python 2.7, 3.5, 3.6, and 3.7.
Support for Python 2.7 is being deprecated and may be dropped in future
releases.

This release also provides support for XSPEC 12.10.1 (patch 'a' or later) in
addition to versions 12.10.0 (included in CIAO) and version 12.9.1.

Details
-------

Documentation and infrastructure fixes are not shown.

444 Improve caching
When fitting multiple datasets simultaneously Sherpa now tries to cache
model evaluations to improve performance.

465 Support XSPEC 12.10.0 (fix 436)
XSPEC 12.10.0e is now supported

508 Ensure fields are initialized before use (pileup code)
Resolved an issue where some fields in the C++ extension could have been
uninitialized in a corner case.

523 Ensure the ui modules have a unique list of exported symbols (fix 502)
Remove the `sherpa.utils.erf` symbol from the two `ui` modules as it is
"over-written" by the ModelWrapper-created symbol when applied to
`sherpa.models.basic.Erf`.

524 Be explicit about 'invalid escape characters' in strings
Convert strings that contain "invalid" escape characters, which newer
versions of Python complain about, to raw strings by preceding them with a 'r'
character.

525 Fixed based line radpro_dm test and changed double --> real (template)

530 Ensure XSPEC 12.10.0 uses ATOMDB 3.0.9 by default
Ensure that the default AtomDB version is 3.0.9 when using XSPEC 12.10.0.
This is to fix an issue with the models-only XSPEC installation in XSPEC

==========

This release fixes a regression introduced in Sherpa 4.10.1 related to PSF convolution. As part of the long term PSF rebinning improvements,
Sherpa 4.10.1 introduced a check to validate that the data pixel size and the PSF pixel size match. If they don't match, then a warning is issued.
The change did not account for an edge error case. Sherpa 4.10.2 solves this problem.
The regression introduced a downstream regression in the gammapy project.

Details
---------

551 Fix gammapy/gammapy1905, catch errors with psf pixel size.
Sherpa did not properly catch errors when checking the PSF pixel size matches the image pixel size. This has now been fixed.

Caveats
-----------
There is a potential issue for macOS conda users with python > 3.6 and matplotlib v3.0.1, when pyqt5 is not installed and selected as a matplotlib
backend. With this specific combination of platform and dependencies matplotlib can fail to work properly. The issue can be replicated without
importing Sherpa, so this is not a Sherpa bug.

Several workarounds exist, the simplest of which are:
* install pyqt5 (`conda install pyqt`).
* downgrade matplotlib to v2 or v3.0.0 (`conda install matplotlib=3.0.0`)

=============

This release fixes several bugs and introduces a few new features, notably the ability to evaluate model components on arbitrary grids and generate user-defined ARFs and RMFs. Also, as of this release Sherpa will no longer rely on any Fortran code. See the following section for details.

It is now possible to build the Sherpa documentation using Sphinx. Additionally, the Sphinx documentation is
automatically built and hosted on ReadTheDocs: https://sherpa.readthedocs.io/

Details
-------

407 Ensure 0-length array is an error in filter_resp (fix 405)
Add an explicit check in the C++ filter_resp code to error out if the noticed channels array
is empty.

422 Improve error message with wrong xspec version
Improve the handling of XSPEC versions mismatch.

466 Fix bounding box out-of-bounds memory read
Avoid an out-of-bounds memory read when calling pad_bounding_box (when the data is not
matching the expected conditions for this call).

469 Evaluate model on finer grid
Sherpa users can now define arbitrary grids, called evaluation spaces, on which to
evaluate individual model components, both in 1D and 2D. This can be useful in a number
of cases, for instance when it is desirable to evaluate models on a finer grid than the one
defined by the data, or in convolution models where information outside of the data range
can be used to reduce boundary effects or to inform the evaluation inside the data space.
Also, when plotting individual 1D model components (plot_source_component), if a specific
evaluation space was attached to the model then the plot will be of the model evaluated
over that evaluation space, not the data space. Other plotting commands should be
unaffected by this change.

470 Order for np.ones() should be a one length string not a bool
Fixed a `DeprecationWarning` in the optimization module.

471 Using overwrite rather than clobber of astropy.io.fits
`clobber` in astropy has been deprecated for a while now in favour of overwrite and thus
issues a `DeprecationWarning`. Sherpa now uses `overwrite` instead.

475 Fix unecessary runtime warning (fix 402)
A condition check in the optimization code was modified so as not to produce a warning
when the value is a NaN.

481 Remove Fortran code
Sherpa does not rely on any Fortran routines anymore, except those compiled in XSPEC.
The existing Fortran code, which was used mainly in some of the optimization routines, has
been replaced by equivalent C++ code. This means that gfortran is no longer required for
building Sherpa, although a version of libgfortran compatible with the one used to link the
XSPEC libraries would still be needed at runtime if the XSPEC extension is
built.

482 ARF/RMF creation functions
The `create_rmf` and `create_arf` functions now allow users to easily generate user
defined response objects.

486 `ZeroDivisionError` in `calc_stat_info` (fix 476)
Sherpa did not catch divisions by zero in the calc_stat_info command. That has been fixed.

484 Equivalent Width errors
Several optional arguments `(params=None, error=False, otherids=(), niter=1000)`, were
added to `eqwidth`. If `error = True`, then get_draws shall be run if the fit stat is one of the
following: (Cash, CStat, WStat) otherwise a multivariate normal distribution shall be run to
generate the samples. The optional niter parameter is used to generate the number of
samples. The optional parameter otherids is only used if get_draws is used internally when
multiple data sets were used to fit. Alternatively, the user can enter the samples via the
`params` option where the samples must be a `numpy.ndarray` of dimension `(num, npar)`.

487 PSF bin size warning
Sherpa assumes that the PSF image and the data have the same pixel size. When this is
not true Sherpa ignores the difference, which results in a larger PSF being applied. From
now on Sherpa will issue a warning when the PSF image
pixel size and the data image pixel size are different.

Testing code now works with pip 10 as well, despite a change in the pip 10 API. Also, the README is now properly rendered as markdown in PyPI.

438 Change from error to warning for OGIP violations
Given that users can not easily change a response file, and previous versions of Sherpa would allow the responses to be used, this commit changes some of the errors recently introduced (in PR 383) into warnings. The errors for the first bin edge being <= 0 are still left in because users of the sherpa.astro.ui module will find that these files are auto-corrected for them (by PR 383).

430 Update XSPEC parameters
XSPEC model parameter default values, limits, and properties were reviewed and updated to reflect changes in or mismatches with the model.dat file shipped with XSPEC 12.9.1n.

428 handle function name changes in XSPEC 12.9.1
Sherpa now supports multiple versions of the XSPEC models from the 12.9.0 and 12.9.1 series. Some models recommend using the C-style interface over the older FORTRAN one, which may also resolve some memory access issues. For CIAO 4.10 users this means the interfaces to XSPEC models have been updated to the 12.9.1n versions. For Standalone Sherpa users this means they can build and run Sherpa against a larger range of XSPEC versions, and Sherpa will pick the XSPEC low level model function accordingly. Note that Sherpa has been tested against XSPEC 12.9.0i, 12.9.0o, and 12.9.1n.
The 14 models that have been changed are: apec, bapec, bvapec, bvvapec, gaussian, lorentz, meka, mekal, raymond, vapec, vmeka, vmekal, vraymond, and vvapec.

427 Add support for XSPEC models added in 12.9.1 (fix 331)
Add support for models added in XSPEC 12.9.1: bvtapec, bvvtapec, tapec, vtapec, vvtapec, carbatm, hatm, ismabs, slimbh, snapec, TBfeo, TBgas, TBpcf, TBrel, voigt, xscat.
Version 12.9.0 of XSPEC can still be used, in which case the models can be generated - e.g. a user can say xstapec.mdl to create a Python object - but it will error out when evaluated.
The Si and S elemental parameters for the ismabs model have been renamed from the XSPEC versions since they are not unique using the caseinsensitive matching used by Sherpa: so SI, SII, SIII and SiI, SiII, SiIII have been renamed S_I, S_II, S_III and Si_I, Si_II, and Si_III respectively.
Low level support for the following convolution models from XSPEC 12.9.1 has also been added, but there is no Python model support for these: clumin, rfxconv, vashift, vmshift, xilconv.

412 support ROSAT PSPC (and similar) RMF files with AstroPy
Add explicit tests for reading in, and using, ROSAT PSPC PHA and RMF files. Fix a bug in the AstroPy back-end handling of the ROSAT RMF file.

409 EmissionGaussian model when skew parameter is not unity (fix 403)
The `sherpa.astro.optical.EmissionGaussian` code has been fixed for the case when the skew parameter is not unity. The documentation has also been updated.

399 Python 3 fixes and new tests for pha2 datasets
Fixed several problems when using Sherpa with Python 3: failures when calling `list_bkg_ids` and `list_response_ids` and the parameter querying after `paramprompt(True)` has been called.
There is also a change to avoid a `FutureWarning` being raised in the astropy backend when reading in PHA2 data files.
Tests have been added to test the reading of a PHA2 format dataset, and new files added to the `sherpa-test-data` repository for this purpose.

398 Check that list_samplers returns a list (fix 397)
The `list_samplers` function now returns a list in Python 3 as well.

396 Fix `set_xlog` and related commands using Python 3 (fix 393)
Fix use of commands that set the plot state using the `sherpa.astro.ui` module, such as `set_xlog` and `set_xlinear`, when using Python 3.

395 Add xspec optional models
Sherpa has new infrastructure for supporting multiple versions of XSPEC at the same time, with models that are built and enabled conditionally depending on the version of XSPEC being linked. Models are assumed to have the same parameters and low level functions across supported versions.

394 XSPEC build and functionality improvements
Add support for reading and changing the path to the XSPEC "manager" directory, and reading the current XSPEC "model" directory (`get_xspath_manager`, `set_xspath_manager`, and `get_xspath_model`). These are intended for the power user and so require an explicit import from `sherpa.astro.xspec`.
There are several improvements to the build and interface to the XSPEC model library: these have no user-visible changes.

390 Add nlapec model
The `nlapec` XSpec model has been added. Note that XSPEC 12.9.0 is now required.

385 DS9 calls return string in Python 2 and 3 (fix 319)
Some low lever `xpa` calls were returning byte strings rather than strings in Python 3. This had particular impact on the `image_getregion` function. This has now been fixed.

383 Replace 0-energy bins in ARF and RMFs (fix 332)
Sherpa now performs some validation of the energy ranges of ARF and RMF files against the OGIP standard. If the ranges are inconsistent then the code will return with an error. If one energy bound is 0 the bound is replaced with a small number to avoid numerical issues like division by zero. A new configuration option `minimum_energy`, if present, allows users to override Sherpa's default behavior.

379 Use a line not span to draw horizontal line (fix 378)
Ensure that the line at y=0 (residual) or y=1 (ratio) is drawn for residual or ratio plots when using matplotlib 2.0.

373 Make sure ds9 properly works with Python 2.7 (fix 368)
For Python versions prior to 3.2 add in explicit code to make _Popen work as a context manager.

352 Add low-level support for the XSPEC rgsxsrc convolution model
Add low-level support for the rgsxsrc convolution model in XSPEC. This is intended for R&D into fully supporting XSPEC convolution models in Sherpa.

255 Add aliases for parameter names so to support new xspec names (fix 74)
The XSPEC models have been updated to use parameter names that match the new XSPEC naming scheme introduced in XSPEC 12.9.0. The old parameter names can still be used as aliases for the new name, but are deprecated and may be removed in a next major release. Note this allows any models to define aliases for their parameters.