Sherpa

Allow the tests to pass when run with pytest 8.0.0.

1959 - Support logarithmic axes for 1D and 2D confidence plots
The projection and uncertainty plots, for both interval (1D) and region (2D),
now correctly create a logarithmic scale when requested. Fix 1561.

1962 - Pick up latest Python micro version in conda workflow
This resolves the missing crypt.h issue being seen in the conda test workflow
by picking up the latest micro version of the python packages for the conda
test build workflows.

1965 - Improve the error message when a plot backend is not available.
Improve the error message when a plot backend is not available. Fix 1964

1968 - Remove Conda build 3.25 pin from deployments
Removes the conda version specification (3.25) to avoid compatibility issues

1969 - Tests: group the DS9 tests so they can be run with one worker process
Ensure those tests that use the requires_ds9 decorator all have the same
group marker, so that they will be run within a single worker process when
using pytest-xdist to run tests in parallel. There is no change if
pytest-xdist is not installed.

1970 - Bump the minimum pytest version
Pytest occasionally changes behavior and it does not seem worth our time to
maintain backwards compatibility with old versions of pytest - see 1960

1972 - Remove excess brackets in model names
Remove excess brackets from model and parameter expressions. This is purely
a cosmetic change, but hopefully makes complicated model expressions easier
to read. Fix 780.

1973 - Fix the requires_pylab decorator
Change the testing code to better-handle those tests that want to check
plotting when using the pylab backend. Fix 1971.

1974 - Allow linked parameters to be fit without including them in the model expression
Treat linked parameters as part of the model expression (via the new lpars
attribute and get_thawed_pars routine) so that they can be included in a
fit without including the linked model as part of the model expression.
Fix 777

1976 - Fix bug in multi-panel plotting in bokeh
Fix bug in multi-panel plotting in bokeh

1977 - model: improve examples in docstrings
Allow model.py, parameter.py, and regrid.py to be included in the
docstring pytest run.

1984 - Rework x errorbar support to better-match community expectations
Improve the "x errorbar" handling for histogram-style data (including PHA),
including support for the wavelength setting (fix 1985), to better match
community expectations (e.g. 1817).

1994 - Tests: improve coverage of parameter module
Improve the test coverage of the parameter module.

1996 - docs: update examples to match current behavior
Update the ReadTheDocs documentation to match the current code.

1998 - Cleanup XSPEC interface
A small number of XSPEC model parameter values have been changed to match
the default frozen state of XSPEC version 12.13.1.

2000 - CM-481: update codecov action to v4, add upload token
Updates version of codecov being used

2003 - Minor plot work
Internal clean up of the plotting code in preparation for future changes.
The ARFPlot class will now generate an IOErr rather than PlotErr if sent
a non-PHA dataset (to better match other calls). The DataHistogramPlot
class now treats xerr as a property that can not be changed, and is fixed
to be the half-width of the X bins, although note that this field is not
really used and may be removed at some point in the future.

2004 - Minor improvements to the RMF plot capability
Improve the RMF plot display so that it recognizes the current units
setting and allows the choice of energies to be over-ridden by the user.

2006 - Use a separate context when handling multiprocessing calls
Ensure that Sherpa does not change the global multiprocessing state but
instead uses the (new) sherpa.utils.parallel.context attribute. Fix 1015.

2011 - Specialize the residual-style plots for histogram data
Improve the display of residual-style plots for integrated datasets.
Fix 1817.

2012 - [CI] Update ci-conda-workflow.yml to pin back to macos-12
Pin back to macos-12. This version is specific to macOS-intel. 13 doesn't
work as conda is removed from it.

2019 - update clone link and versions and DOI
This is a documentation update to fix the incorrect link in the source
install example. I also updated the version and DOI link in the description
of the source installation.

2020 - CI: bump the checkout actions versions
Internal change to how the GitHub actions are run for CI.

2021 - Update Conda Deployment Workflow Action Versions
Updates internal actions in the deployment workflow

=============

This release of Sherpa includes various enhancements, documentation updates, bug fixes, and infrastructure changes.

* enhancements:
* add calc_model and calc_source functions to return an evaluated model/source array
* added wstat to plot_pvalue for the likelihood ratio test
* changed XSpec interface to use FunctionUtility C++ API instead of XSFortran API
* improved support for PHA data starting at channel 0 and handling of STAT_ERR and SYS_ERR PHA columns which are set to 0
* improved guess for complex models
* improved filtering handling
* several updates to enhance plotting capabilities and layout
* documentation changes:
* added paper citations to front page of Sherpa Read the Docs documentation
* cleaned up various typos and URL references
* added examples such as use of set_x/y_label
* infrastructure and testing:
* improved test coverage
* many updates to CI
* drop support for Python 3.9 and numpy <1.24
* initial/experimental support for Python 3.12
* bug fixes:
* fixed an issue with plotting 1D data with asymmetric errs after filter
* include the default identifier in save_all output if it has been changed

Details
-------

1757 - Allow the wstat statistic to be used in the likelihood-ratio test
Allow the Likelihood Ratio Test (as used by plot_value) to be run when
using the WStat statistic. Fix 1745.

1793 - XSPEC move from the xsFortran API to the FunctionUtility API (where possible)
Internal change to the XSPEC interface to use the FunctionUtility C++ API
rather than the XSFortran API. Fix 1225.

1921 - Clean up of the IO code
Rework the astro data input and output code so that they take advantage of
the new data representation in sherpa.astro.io.types

1949 - Initial support for Python 3.12
Initial support for building with Python 3.12

=============

This release of Sherpa includes various enhancements, documentation updates, bug fixes, and infrastructure changes.

* enhancements:
* minor plotting changes; add support for splitting model expression into
additive components and plot the results; support of log scale axes for
confidence plots; improved error messages for unavailable plot backends
* improved RMF plot display to allow choice of energy units
* documentation changes:
* updates to fake_pha documentation
* updates to install.rst to fix incorrect links and outdated version references
* updated read the docs documentation to match current code
* bug fixes:
* fixed multi-panel plot issue with Bokeh backend

Details
-------

1608 - Remove the test setup.py option
Add the test configuration so that we can ensure the necessary testing
packages can be installed when the code is built. Support for running the
tests with 'python setup.py test' has been removed to match changes in the
Python packaging ecosystem.

1679 - Automatically split a model expression into additive components and plot the results
Add plot_source_components and plot_model_components calls that will split
up a model like gal * (pl + line) into two lines: one for gal * pl and one
for gal * line. There are also corresponding get_source_components_plot and
get_model_components_plot calls, and the "source_components" and
"model_components" arguments can be used with the plot call.

1684 -Fix fake_pha docs
Fix the documentation and some corner cases of sherpa.astro.fake.fake_pha.
There are several times when the simulated PHA output would not be correct
if the PHA contains at least one background component or a pileup model
was used.

1926 - Let users know they should use load_xstable_model not load_table_model
Support for XSPEC table models in load_table_model was deprecated in the

This release of Sherpa includes various enhancements, documentation updates, bug fixes, and infrastructure changes.

* enhancements:
* the grouping commands like group_counts and group_snr now default to only grouping within the noticed range of channels, which is a change in behaviour
* new plotting backend: Users can now choose between matplotlib and bokeh (experimental) support use of arbitrary python functions when linking model parameters
* updates to fake_pha, save_all, allowing to write out RMF/ARF files
* get_plot_prefs and get_contour_prefs call to simplify access to the plot and contour preferences
* implementation of RMFPlot and DataIMGPlot classes and associated UI functions (plot_rmf, get_rmf_plot)
* update support for XSPEC to include version 12.13.1, allow XSPEC table models which include the ESCALE parameter, and provide experimental support for writing out XSPEC table models.
* documentation changes:
* added doc strings for sherpa data classes
* several updates to in code and RTD documentation
* Infrastructure changes:
* revamp of plotting backends from modules to classes and adding support for multiple backends
* changes to use the NumPy random generator API
* bug fixes:
* PHA source plot Y axis scaling (1825)
* fix model display for grouped data (1779, 1784)
* Change in the ordering of operations when grouping background PHA datasets. (1881)

Details
-------

1382 - Plot changes: Step 1 A- Draft API for backend classes
Plotting infrastructure changes that make the plotting backends classes
instead of modules.

1617 - Pass keyword arguments to models
Allow keyword arguments to be passed to models and ensure these arguments
are respected by the model cache.

1629 - Note when a XSPEC model is changed to F77 form (double precision)
Update a script used when updating the XSPEC support to point out models
that use the double-precision FORTRAN support. There is no functional
change to the code.

1653 - Allow arbitrary functions for linking parameters
Provide a way to use any Python function to be used for linking
parameters. Fix 1652

1730 - Update xspec model creation
Update the code created by the sherpa.astro.utils.xspec routines to
account for changes in the XSPEC interface in Sherpa. This only affects
users who call these routines directly.

1734 - Unify the fake and fake_pha calls
The fake_pha call now accepts a method argument to match the fake call.

1735 - Allow use of the NumPy random generator API
Take advantage of the NumPy random-number generator classes to provide
random numbers, and move away from using the global state provided by
numpy.random.seed and random.seed routines.

1738 - Return more-useful information when an XSPEC model fails
Include the model name and parameter values in the error message of a failed XSPEC model.

1750 - Replace pkg_resources as it is deprecated
Remove the use of the deprecated pkg_resources module and replace with use
of importlib.resources. Fix 1723

1763 - Allow load_image to change the coordinate system
Ensure that the coordinate setting used in load_image calls is properly
handled (when not set to the default value of "logical"). Fix 1762

1770 - Drop python2 ism in the fit code
Internal clean up of the fit code to replace a Python 2 method with a
Python 3 version and to use f-strings.

1771 - utils: make sure OutOfBoundErr is an actual exception
Ensure that OutOfBoundErr is an exception (this is an internal feature).

1772 - Tweak resample data code
Internal changes to the ReSampleData class to better handle invalid input.

1773 - Internal clean up of estmethods
Code clean up of sherpa.estmethods.

1775 - Change a single test to use explicit rather than implicit random state
from NumPy
Simplify a single test. There is no functional change.

1776 - Move the parallel code into a separate module
Move the parallal_map code from sherpa.utils to the new sherpa.utils.
parallel module (the symbols are still accessible from the original
location but code should be updated to use the new location, including the
renames of _ncpus and _multi to ncpus and multi).1777 - Change get_sample
to require named arguments

1778 - Optmethods cleanup
Move testing code out of the optmethods modules and into a separate test,
along with a number of code clean ups. There is no functional change.

1780 - Fix model plot display when factor is set for grouped PHA data (issue
1779)
Ensure that model plots can be created for PHA data when set_analysis
routine is used to set factor > 0 and the data is grouped. Fixes 1779 and
1784

1781 - utils: remove the numpy_convolve routine
Remove the sherpa.utils.numpy_convolve routine.

1785 - Add docs to Sherpa data classes
Add docstring to most Sherpa dataclasses.

1791 - Tests: ensure repeatable test of get_draws
Ensure tests of get_draws use a fixed seed to make the results
reproducible. There is no functional changes in this commit.

1794 - docs: note requirement on ar for building Sherpa
Add ar to the list of requirements for building Sherpa. Fix 1792

1795 - Add docs to describe the inputs for the ``calc`` function of a user
model
Add docs that describe the arguments of the calc function for user models.

1797 - model: explicitly send parameter argument to calc
Ensure that the calc method for models begins with the parameter array
(explicitly given) rather than extracting it from the named arguments sent
to the routine. This should not change any user code.

1800 - utils: minor code clean up
Internal clean up of the sherpa.ui.utils module.

1805 - Clean up display of dataset ids during fits
Ensure dataset ids are listed without leading brackets in the fit and
staterror string outputs (remove an accidental change made in 1770). Fix
1804

1806 - Activate sphinx directive to make plots
Use the Sphinx plot directive to generate (some of) the plots in the docs.

1807 - NumPy 1.25 support
Remove deprecation warnings seen when using NumPy 1.25.0.

1808 - Use Sherpa install instead of code only to build docs
Switches from building the documentations from the source directory, to
building it from an installed version of Sherpa.

1809 - Update the hardcoded Zenodo releases to include the 4.14/4.15 releases
Update the stored Zenodo releases to include the 4.14.* and 4.15.* Sherpa
releases. Users should see no difference (other than avoiding the need for
an http call to retrieve this information) when calling sherpa.citation().

1810 - RTD: fix script name for XSPEC updates
Improve the documentation on how to update to a new XSPEC version by
ensuring the correct helper script is run in an example.

1811 - ui: use the default pickle version when dumping the state
Use the default version when pickling data (used by the UI save call)
rather than an old version. Fix 1243

1812 - Autoupdate copyright year in sphinx docs
Autoupdate copyright year in sphinx docs.

1813 - Fix ReadTheDocs warnings and then make RTD fail on warnings in CI
Fix ReadTheDocs warnings and then make RTD fail on warnings in CI.

1819 - CI: hide NumPy 1.25 array ndim>0 deprecation warnings
Allow the tests to run with NumPy 1.25 and with Crates (from CIAO 4.15) as
the I/O backend. There are no functional changes.

1821 - parallel: tweak ordering of checks for parallelization
Internal change to the decision of whether to use the multiprocessing
module to run code or not.

1823 - sherpa_test: tweak how pytest/plugins are installed
Internal changes to how we check for test requirements used by the
sherpa_test script.

1824 - remove non-existent datastack plotting backend initialization attribute
Clean up error condition by removing the sherpa.plot.backend.
initialize_backend attribute as no longer exists.

1828 - Improve PHA source plots: support type=counts and factor above 1
Source plots for PHA data now follow the rate and factor analysis
settings. Prior to this the rate setting was ignored and the factor
setting produced incorrect values for factor=1 and factor=2). Fix 1825

1829 - Limit precision in the comparison of string outputs
Update the tests to support running on ARM/AARCH64 platforms where numeric
differences can complicate string comparisons. Fix 1815.

1832 - astropy: rework RMF input (fix 1344)
Allow the AstroPy backend to read in HRC-I RMF data. Fixes 1344.

=============

This release of Sherpa includes various enhancements, documentation updates, bug fixes, and infrastructure changes.

* enhancements:
* further improvements to filtering/grouping including reporting a filter change in the UI
* fake_pha can be called with a list of ARF/RMF names
* added linewidth option for line and histogram plots
* documentation changes:
* fixed broken URLs
* improved documentation for templates, plot_pvalue
* added documentation testing with doctestplus
* Infrastructure changes:
* dropped support for Python 3.8
* experimental support of Python 3.11
* supported versions of Xspec are 12.12.0 - 12.13.0
* bug fixes:
* various updates to notice/ignore and group/ungroup code
* fixed issue with show_bkg
* fixed issue when binning values into a 1D histogram
* fixed cache errors with the TableModel class

Details
-------

1396 - XSPEC minimum version is now 12.10.1
Bump the minimum-supported version of XSPEC from 12.9.0 to 12.10.1.
(superseded by 1609)

1472 - tests: remove work-around for 1334 from 1336
Remove a test work around for our CI tests failing (issue 1334)

1486 - Improve filtering/grouping behavior
Changing the grouping array of a PHA dataset (with set_grouping from the UI layer
or by setting the grouping attribute of a DataPHA object) will now re-create the
filter, to fix 1160.
It is suggested that the filter expression (e.g. as returned by the get_filter
routine for uses of the UI layer) is checked to ensure the result is still sensible.

1502 - Cleanup template code
Improve the documentation and testing of the template-model code. There is no
significant change in behavior, but some use cases will generate more-appropriate
errors than they used to.

1591 - Document params and return values for root finding functions
Add docstrings for root finding functions

1607 - Fix an issue when binning values into a 1D histogram
Fix a problem when binning data into a histogram (issue 1605).

1609 - XSPEC minimum version is now 12.12.0
Bump the minimum supported XSPEC version to version 12.12.0.

1611 - Post 4.15.0 updates
Add the 4.15.0 DOI to the README.md.

1613 - Allow testing of the documentation
Adds the ability to run doctests in py and rst files. Those tests will run
automatically, if doctestplus is installed (not on CI in the current setup) and
the file is not explicitly blacklisted for doctestplus.

1621 - Fix a cache error with TableModel
Fix cache errors with the TableModel class (1622), add validation of input
values, and minor additions to the documentation.

=============

This release of Sherpa includes various enhancements, documentation updates, bug fixes, and infrastructure changes.

* enhancements:
* Improved validation of arguments when creating Data objects:
* arrays sent to Data objects are now converted to ndarrays
* the independent axis is now made read-only
* the size of a data object is now fixed.
* Filter setting with notice/ignore are reported to the screen for the users of the UI layer.
* Increased test coverage for plotting
* documentation changes:
* updated readthedocs to use pip and pytest instead of setup.py
* several updates to documentation, including updates to fake_pha, calc_ftest, calc_mlr
* Infrastructure changes:
* Drop support for Python 3.7
* Updates to start creating Python 3.10 Conda packages.
* Use Numpy 1.20 for Python 3.8/3.9 and Numpy 1.21 for Python 3.10.
* Moves toward PEP-517 with some distutils cleanup and more configuration moved from setup.py to setup.cfg
* Various improvements to the GitHub Actions and GitLab workflows
* bug fixes:
* Ensure chi2xspecvar errors match XSPEC when 0 counts are present during background subtraction
* Remove model instances from the global symbol table when clean is called
* Addresses new warnings in the tests for Matplotlib 3.6.0 and AstroPy 5.1
* Minor copy and paste error in fake_pha docstring
* Test issues in test_fake_pha.py due to randomness

Details
-------

1329 - Build updates (move towards PEP-517 support)
Move the Sherpa build system towards a more static configuration (PEP 517)

1412 - XSPEC: initialize XSPEC library at load time
Simplify how and when the XSPEC model library is initialized. Fixes 1388

1477 - Add data validation
Improve the validation of arguments when creating Data objects- Arrays sent to
Data objects are now converted to ndarrays, the independent axis is now made
read-only, the size of a data object is now fixed.

1504 - CI: rework setup
Update the GitHub workflows for Continuous Integration to better take advantage
of the workflow language.

1505 - RTD use pip/pytest rather than setup.py for install/develop/test
Change the RTD documentation to use pip and pytest rather than setup.py when
building and testing Sherpa

1507 - tests: ensure DS9 window is closed after the tests
Ensure that any DS9 process started by the Sherpa tests is closed when the tests
are finished.

1510 - Tests: improve test coverage of plot code
Improve the testing coverage for several plot routines.

1513 - Minor UI tweaks
Added validation to the set_xlog/ylog/xlinear/ylinear and updated save_model,
save_source, save_resid, and save_delchi commands from sherpa.ui to not error
out if not given an explicit identifier.

1516 - Consolidate plot and set_xlog handling
Use the same labels for the plot and set_xlog/set_ylog/set_xlinear/set_ylinear
functions. As a result, a number of names can no-longer be used as an identifier
for a dataset (bkg_chisqr, bkg_delchi, bkg_fit, bkg_model, bkg_ratio, bkg_resid,
bkg_source).

1519 - Use direct html links instead of using `[1]` and a reference section
This replaces the use of a "References" section with direct URL links in
particular in cases where the reference is just a plain old HTML website and
not a scholarly publication.

1523 - Ensure the region-lib code is re-generated when needed
Fix a build issue with clang where the region-library code caused an error
message

1529 - Add arguments to -i in sed calls in config script
Update sed command in config script for improved portability

1532 - Add plot related tests
Add more tests to cover corner case scenarios (mostly plotting related).

1534 - docs: fix calc_ftest/mlr documentation
Note that calc_ftest and calc_mlr can return an array or scalar value, depending
on the input.

1536 - Tests: add more tests of get_order_plot/plot_order
Add a test for multiple orders and plot_order.

1537 - Remove model instances from the global symbol table when clean is called
Ensure that models are removed when clean() is called.

1538 - Ensure chi2xspecvar errors match XSPEC when 0 counts are present during background subtraction
When using the chi2xspecvar statistic for estimating PHA errors, ensure that the
errors match those calculated by XSPEC when the background is subtracted and the
source or background group contains 0 counts

1540 - Tests: fix the random seed for a fake_pha test
Remove randomness from a test.

1547 - Fix CI failures from AstroPy 5.1 warning changes
Allow the macOS tests to run on CI with AstroPy 5.1.

1554 - Install New libxcb Dependencies for Qt
For Linux GitHub Actions, we now install libxcb dependencies required for
Conda's Qt package.

1555 - Bump the ci-pip-arch workflow to ubuntu-20.04
Update to use ubuntu 20 as GutHub Actions is deprecating ubuntu 18.04

1562 - report the change in the filter because of calls to notice and ignore
When a notice or ignore call - including the variants like notice_id and
ignore2d - then the change in the filter is reported to the screen (for users
of the ui layer).

1563 - Add Python 3.10 Conda Builds, Drop Python 3.7 and Numpy <1.20, Switch to Pip for Conda Builds
Drop official support for Python 3.7 and start creating Python 3.10 Conda
packages. Our Conda builds now use Numpy 1.20 for Python 3.8/3.9 and
Numpy 1.21 for Python 3.10.

1566 - GitLab: Add version info to deploy test, update test OS, and allow interrupt.
Updates to gitlab to support automated tests, save version info, set default to
interruptible to allow canceling pipelines, and switched to source conda instead
of adding it to the path

1567 - Fix minor copy and paste error in fake_pha docstring
Make UI layer fake_pha work for XMM/RGS by accepting input of arf=None when an
RMF file is given.

1569 - Minor changes to the fake_pha test added in 1567
Internal changes to the test added for fake_pha in 1567

1572 - Tests: fix a logical test error
Fix a logical error in a plotting test.

1574 - Avoid occasional test errors from Tcl_AsyncDelete
Switch the matplotlib tests to use the Agg backend to avoid possible
Tcl_AsyncDelete errors (issue 1509)

1578 - Gitlab: Reduce artifact retention period
reduces artifact retention from 2 weeks to 3 days due to new gitlab 5gb
restrictions

1579 - Add back --python removed in 1566
reinserts --python flag that was accidentally removed in 1566

1580 -add a versionchanged note for the fake_pha command
Update the documentation to note the change to fake_pha in 1567

1585 - Tests: work around matplotlib 3.6.0 warning
Allow the tests to pass when run with matplotlib version 3.6.0.

1589 - Set the language when building the documentation
Ensure the language setting is defined for recent versions of Sphinx

1594 - gitlab-ci: Set build num to 0 if not set
Set the default Sherpa build number to 0 in our GitLab workflow to know which
package to run deploy tests on for official release builds.

1600 - Change the histogram1d and 2d routines so that they do not sort the input arguments
Address some failures in histogram1d and histogram2d routines by making sure
they copy the input arguments before sorting them.

Caveats
-----------

* There are known issues (1602, 1605, 1606) in the histogram1d/histogram2d functions
leading to failures which were not fully addressed in this release (see the failed
case in the second histogram1d example). This is not the Sherpa core functionality
and numpy.histogram can be used if needed.