Pysat

--------------------
* Maintenance
* Added temporary version limit to xarray due to change in indexing

--------------------
* New Features
* Added the property `empty_partial` to the Constellation class
* Added the option to apply custom functions at the Constellation or
Instrument level within the Constellation class
* Added option to load Constellation for registered Instruments using lists
of platforms, names, tags, and/or inst_ids, which are new attributes
* Added hidden Constellation methods to determine unique attribute elements
and set Instrument attributes across all instruments
* Added hidden Constellation method to sequentially call Instrument methods
* Extended Constellation unit tests
* Added more quality checks to the Constellation initialization
* Standardized Instrument instantiation to always define `inst_module`
* Extended testing options for `pysat.utils.testing` functions
* Added `start_time` keyword for test instruments
* Added `max_latitude` keyword for non-model test instruments
* Added `_test_download_ci` as a standard attribute for `pysat.Instrument`
* Added a testing model similar to TIEGCM to
`pysat.instruments.pysat_testmodel` as tag='pressure_levels'.
* Added the capability to test loading with optional kwargs through
`_test_load_opt` instrument attribute
* Added `rename` and `add_epoch_metadata` methods to the Meta class
* Added I/O utilities to the utility module, extracting existing methods
from the Instrument class and core utility module.
* Added a warning evaluation and directory creation functions to
`pysat.utils.testing`.
* Added the ability to only download new files if remote file listing
capabilities are not available for the Instrument.
* Added kwargs for epoch units and origin in `pysat.utils.io.load_netcdf`.
* Vectorized `Meta.var_case_name` and `Meta.attr_case_name` to support
list of str as input as well as str.
* Added a time function to calculate decimal year from datetime.
* Allow `Instrument.rename` to take a fuction or mapping dict as input,
after adapting routine to use `Meta.rename`
* Added an update method and type evaluation method to MetaLabels.
* Added a new MetaHeader class for holding global metadata.
* Added a pysat Instrument for loading pysat-written NetCDF4 files.
* Allowed MetaLabels to be expanded through setting new Meta data values.
* Added support for user variables when parsing template filenames
in `pysat.utils.files`.
* Improved robustness of parsing delimited files.
* Updated `pysat.Instrument.pandas_format` so that changes to setting
are immediately reflected by changes to Instrument object.
* Added a test function for bad class/function/method input to reduce
duplicate code and improve message test combliance
* Added support for filtering basic netCDF4 metadata when loading via
`pysat.utils.io.load_netcdf`.
* Added support for user developed functions to filter metadata as it
loaded from disk but before assignment to a `pysat.Meta` instance.
* Added `meta.to_dict` to support creating more compliant
netCDF files.
* Added `meta.default_to_netcdf_translation_table`,
`default_from_netcdf_translation_table` and
`apply_table_translation` to `pysat.utils.io` to support
improved compatability of pysat netcdf files without user code
changes.
* Added `drop_meta_labels` kwarg to `pysat.utils.io.load_netcdf` to
support easy removal of unwanted metadata during loading.
* Added support for `meta_processor` in `pysat.utils.io.load_netcdf`
and `pysat.utils.io.inst_to_netcdf` enabling developers to easily modify
metadata before it is loaded from/written to storage.
* Intermediate missing directories are now created as needed when writing
files using `pysat.utils.io.inst_to_netcdf`.
* Condensed code present in both pandas and xarray into
`pysat.utils.io.meta_array_expander`.
* Non-default Meta.labels are now retained by Instrument if defined by
underlying support module.
* Made test data more consistent across pysat testing instruments.
* Added `pysat.Instrument.vars_no_time` function to return data variables
excluding the main time index.
* Added support for additional metdata and other file handling options
to general 'pysat', 'netcdf' instrument.
* Added keyword `decode_times` to flag using xarray or pysat processing
of times when loading a file via `pysat.utils.io.load_netcdf`.
* Added `pysat.utils.io.xarray_all_vars` which returns a list of all variables,
coordinates, and dimensions.
* Added more volume units to `pysat.utils.scale_units`
* Added `assert_isinstance` and `assert_hasattr` to `pysat.utils.testing`
* Deprecations
* Removed `freq` as a standard kwarg for `pysat.Instruments.download`
* Deprecated `_test_download_travis` as a standard attribute for
`pysat.Instrument`. The function is replaced by `_test_download_ci`
* Deprecated
`pysat.tests.instrument_test_class.initialize_test_inst_and_date`,
which is moved to `pysat.tests.classes.cls_instrument_library`.
* Deprecated `pysat.tests.instrument_test_class.InstTestClass`, which is
replaced by `pysat.tests.classes.cls_instrument_library.InstLibTests`.
* Deprecated `pysat.Instrument._filter_netcdf4_metadata` and replaced it
with `pysat.utils.io.filter_netcdf4_metadata`.
* Changed `fname` from a kwarg to an arg in `pysat.Instruments.to_netcdf4`
* Deprecated support for 2D pandas datasets
* Deprecated `pysat.instruments.pysat_testing_xarray`
* Deprecated usage of higher order metadata
* Deprecated `pysat.Instrument.generic_meta_translator` which is replaced
by `pysat.utils.io.apply_table_translation_to_file`.
* Deprecated the general instrument method, `convert_timestamp_to_datetime`,
which is replaced by new functionality in `load_netcdf`.
* Documentation
* Moved logo to 'docs\images'
* Improved consistency of headers throughout documentation
* Added more reference labels to documentation
* Implemented use of Python formatting in documentation examples
* Fixed bad section labels in docstrings
* Fixed noindex warning
* Fixed redirecting links
* Made changelog style and line length consistent
* Added a paragraph describing how to access Instrument sub-module
docstrings from an instantiated Instrument in an interactive session
* Updated examples in docstrings
* Improved docstring style and consistency
* Added docstring linting via flake-docstrings and hacking to CI tests
* Added clarification to the tutorials
* Added limitations to the `listify` docstring
* Updated image link in `README.md` to display in more environments.
* Updated `tag` and `inst_id` keywords and docstrings to new standards
* Added example for version checking when writing tests for packages that
rely on pysat
* Bug Fix
* Fixed default MetaLabel specification in `pysat.utils.load_netcdf4`
* Fixed `parse_delimited_filename` output consistency and ability to handle
leading and trailing non-parsed text in filenames (e.g., file extensions)
* Added `decode_timedelta=False` for loading xarray from netcdf4 (823)
* Closed links to open files when loading data through xarray (887)
* Fixed an issue in generating filenames for `pysat.Instrument._iter_list`
* Allow `tag` and `inst_id` to be specified as None (892)
* Fixed a bug in `pysat.utils.time.create_datetime_index` (906)
* Fixed a bug in `pysat.utils.scale_units` and ensured '/cc' compatibility.
* Added a warning if `inst_module` and `platform`/`name` are used to
instantiate an instrument (850). In case of this, `inst_module` takes
priority.
* Fixed a bug when instantiating empty `pysat.Instrument()` (851)
* Fixed IndexError encountered when loading a multi-day file by filename with
no data for that day.
* Added logging output when metadata not assigned due to a lack of
instrument data (924)
* Fixed a bug where empty check for xarray instruments fail when time not
present. (922)
* Improved feedback when users try to set meta with an array.
* Fixed a bug that expected special treatment by `Instrument.data` type in
`utils.coords.update_longitude`
* Fixed pysat_testmodel Instrument longitude range
* Corrected link to testing badge in README.md
* `from_os` now always returns a sorted Series.
* Moved function call for Instrument modules init function to end of
pysat.Instrument instantiation to ensure Instrument is complete when passed
to init.
* Refactored `pysat.Instrument.generic_meta_translator` to use
`pysat.Meta.to_dict` and the latest metadata label standards.
* Prevent stale data paths stored by pysat from being reassigned if
path is no longer in `pysat.params['data_dirs']`.
* Added missing metadata in testing instruments.
* Corrected identification of string data for xarray in
`pysat.Instrument._get_data_info`.
* Maintenance
* Added unit tests for deprecation warnings related to io_utils reorg.
* Added missing unit tests for `pysat.utils.time`
* Added missing unit tests for `pysat.utils.file.parse_delimited_filename`
* Streamlined unit tests for `test_orbits`
* Moved metadata generation for test instruments to `methods.testing`
* Added integration tests for test instrument kwargs
* Updated class declaration to be consistent with python 3 standards
* Update usage of caplog and capsys in unit tests
* Reorganized tests for the `pysat.Instrument` class into multiple files
* Updated unit tests for `pysat.Instrument` with pytest.mark.parametrize
* Update minimum numpy in CI tests to 1.19 following NEP29
* Made `InstLibTests` more portable to streamline user implementation of the
standard end-to-end instrument tests.
* Moved test classes to `pysat.tests.classes`
* Reorganized the core test_instrument.py as a series of classes
* Separated MetaLabels tests from Meta test class
* Organized and reduced duplication in the Meta test class
* Added CI reports for supported data products
* Added a cap on coveralls to ensure success of continuous integration
* Updated tests in `test_meta` to search all warnings, not just the first
* Updated pandas syntax to be compatible with pandas 2.0 (pandas 1.4
deprecations)
* Cleaned up excess variables upon import
* Removed `data_path` check from `pysat.instruments.methods.general.list_files`
* Compatible with netCDF v1.6.0
* Updated default labels in load_netcdf routines to match core pysat metadata

--------------------
* New Features
* Added a routine for loading CSV files into a pandas DataFrame from a list
of filenames.
* Added check for supported `tag` and `inst_id` at pysat.Instrument
instantiation. (784)
* Expanded Constellation utility by:
* Adding common properties: `empty`, `index`, `date`, `today`,
`yesterday`, `tomorrow`, and `variables` (764)
* Improving the printed output to inform user of the Constellation
contents (764)
* Added methods to download data and create a common time index. (764)
* Added utils.listify, a function that returns a list of whatever is input.
(766)
* Added a warning for download requests that result in an empty date range.
* Deprecations
* Documentation
* Added missing information about date formatting to file listing docstrings.
* Added example for registering external instrument support. (837)
* Bug Fix
* Changed pysat.Instruments.orbits iteration to return a copy of the
Instrument rather than the Instrument itself. Provides robustness against
garbage collection. (770)
* Improved error messages for cases where slice of data may not exist (761)
* Improved windows compatibility (57, 790)
* Fixed Instrument.load bug that prevented use of instrument specific kwargs
* Added pytest as a package requirement (819)
* Fixed pysat.utils.listify for empty list inputs (830)
* Maintenance
* Changed pysat.Instrument from treating all support functions as partial
functions to retaining the original form provided by developer
* Migrates CI testing to Github Actions
* Improved maintenance of documentation build
* Added a check for use of reserved keywords at instantiation
* Tests compatible with pysatSpaceWeather 0.0.4 (782)
* Improved pandas 1.3.0 compliance

--------------------
* New Features
* Added registry module for registering custom external instruments
* Added Meta.mutable flag to control attribute mutability
* Added MetaLabels class to manage metadata labeling
* Unit tests are now pytest compatible, use parametrize, and have improved
messages when failures are encountered
* Added altitudes to test instruments
* New flags added to instruments to streamline unit testing:
`_test_download`, `_test_download_travis`, `_password_req`
* `strict_time_flag` now defaults to True
* Use of start/stop notation in remote_file_list
* Added variable rename method to Instrument object (91)
* Added support for loading more than one day/file (56)
* Added support for iterating over a dataset a with a loaded data width and
stepsize larger than a single day/file
* Added check for inconsistent inputs when loading data via Instrument
* Added file locking for thread-safe behavior (304)
* Allow the Instrument object to be initialized with optional kwargs for any
of the standard methods (not just load).
* Added support for 'cycle' in addition to 'version' and 'revision' for
filename conventions.
* Integrated Custom class directly into Instrument to support mixed use
of Constellation/Instrument objects in code (540)
* Made underlying custom data structures visible (529)
* Updated data_mode method name to custom_attach for the Constellation
object (540)
* Added a display utility for discovering pysat Instrument data sets.
* Added testing utility functions.
* Added support for multiple pysat data directories
* Reorganized .pysat directory to store instrument information under
.pysat/instruments and .pysat/instruments/archive
* Added pysat.params, a central location to store and modify pysat default
parameters. File stored at .pysat/pysat_settings.json
* Added `warn_empty_file_list` to pysat.params to provide a warning if no
Instrument files are found. Default is False.
* Updated default pysat directory organization template
* Added support for dual specification of Instruments to include in a
Constellation
* Added function to transition pysat managed data files to a user
provided template.
* Allow `directory_format` input to `Instrument` to be a function
* Adopted standard for bounds. `stop` is an inclusive bound, `end` is
exclusive
* Added support for SLT calculations outside [0, 24)
* Added support for continuous SLT calculations when loading multiple days
* Instrument support functions now respond to local changes in
Instrument.kwargs
* Added support for pysat.Instrument, Files, and Orbits equality comparisons
* Added .copy function to Instrument, Files, and Orbits classes
* Updated Parameters to ensure paths provided for pysat.params['data_dirs']
are created if they don't already exist
* Added funtion to get file attributes, utils.files.get_file_information
* Deprecations
* Migraged instruments to pysatMadrigal, pysatNASA, pysatSpaceWeather,
pysatIncubator, pysatModels, pysatCDAAC, and pysatMissions
* Migrated file methods to pysat.utils.files (336)
* Renamed `sat_id` Instrument keyword argument to `inst_id`
* Removed ssnl
* Removed utils.stats
* Removed model_utils
* Removed deprecated pandas.Panel
* Import datetime class from datetime module, not pandas
* Import DataFrame and Series classes directly from pandas, not pysat
* Removed coords.scale_units
* Removed time.season_date_range
* Moved methods.nasa_cdaweb.list_files to methods.general
* DeprecationWarning for strict_time_flag only triggered if sloppy data is
found
* Changed the custom function attachment input to allow keyword argument use
when additional function input is required
* Removed python 2.7 syntax
* Removed utils.coords.geodetic_to_geocentric
* Removed utils.coords.geodetic_to_geocentric_horizontal
* Removed utils.coords.spherical_to_cartesian
* Removed utils.coords.global_to_local_cartesian
* Removed utils.coords.local_horizontal_to_global_geo
* Addressed several Warnings raised by incorrect use of dependent packages
* Deprecated use of `inst_id` for number of simulated samples for test
instruments
* Removed writing of custom Meta attributes when producing netCDF4 files
* Removed unneeded description.txt file, using README instead
* Changed `pysat.instruments.methods.general.list_files` kwarg
`fake_monthly_files_from_daily` to `file_cadence`
* Changed name of Instrument method `default` to `preprocess`
* Removed `pysat.data_dir`. Information now at `pysat.params['data_dirs']`.
* Moved `pysat.Instrument._filter_datetime_input` to
`pysat.utils.time.filter_datetime_input`
* Deprecated `pysat.utils.set_data_dir`
* Changed `name` kwarg in Constellation to `const_module`
* Removed unnecessary Instrument attribute `labels`
* Removed unnecessary Instrument kwargs
* Removed the Custom class, incorporating it into Instrument
* Removed deprecated calls to 'modify' type custom functions
* Documentation
* Added info on how to register new instruments
* Fixed description of `tag` and `inst_id` behaviour in testing instruments
* Added a tutorial for developers of instrument libraries for pysat
* Updated instrument templates
* Added .zenodo.json file, to improve specification of authors in citation
* Improved `__str__` and `__repr__` functions for basic classes
* Improved docstring readability and consistency
* Added Travis-CI testing for the documentation
* Added a style guide for developers
* Updated loading data flow figure
* Scrubbed the documentation, ensuring examples, tutorials, and descriptions
match the current implementation and code structure
* Bug Fix
* Updated Instrument.concat_func to behave as described in the docstring
* Fixed custom instrument attribute persistence upon load
* Improved string handling robustness when writing netCDF4 files in Python 3
* Improved pandas 1.1.0 compatibility in tests
* Fixed coupling of two_digit_year_break keyword to underlying method in
methods.general.list_files
* Fixed additional file date range for monthly data with gaps
* Fixed custom Meta attributes removal when transferred to instrument (615)
* Corrected iteration over Instrument within list comprehension
* Removed unused input arguments
* Corrects Instrument today, yesterday, and tomorrow methods by implementing
datetime.datetime.utcnow
* Ensured pysat.Meta instances are immutable at Instrument instantiation
* Removed weak reference back to Instrument within Files class
* Fixed access of xarray data with more than one dimension (471)
* Improved robustness of `eval(inst.__repr__())` (636)
* Fixed `calc_solar_local_time` for data sets with longitude coordinates
* Fixed .copy() when pysat.Instrument instantiated with `inst_module` (728)
* Modified storage of Instrument.kwargs to include all methods so that
`eval(Instrument.__repr__())` works in more cases
* Modified storage of Instrument.kwargs to only include user supplied
keywords
* Improved robustness when working with file dates that aren't centered on
midnight
* Added basic valid path check to `pysat.utils.files.check_and_make_path`
* Maintenance
* nose dependency removed from unit tests
* Specified `dtype` for empty pandas.Series for forward compatibility
* Removed wildcard imports, relative imports
* Include flake8 check as part of testing suites
* Improved unit testing coverage of instrument functions and instrument
object
* Added tests for Instrument attributes `acknowledgements` and `references`
* Removed implicit conversion to integers in
methods.general.convert_timestamp_to_datetime
* Simplified internal logic in Instrument class
* Moved setup metadata to setup.cfg
* Improved instrument tests for files
* Used dt.timedelta instead of pds.DateOffSet where possible
* Reduced code duplication throughout package
* Reduced unused code snippets throughout
* Ensured download start time is used
* Condensed testing support functions into methods/testing.py
* Fixed a bug with usage of numpy.dtype for numpy 1.20 compatibility
* Updated usage of pds.index.to_native_types() to pds.index.astype(str)
for pandas 2.0 compatibility (737)
* Check type as float rather than np.float for future numpy compatibility
(740)
* Verified usage of inst.loc[slice, keyword] will continue to work in
pandas 2.0 (738)

--------------------
* Allow use of new Instrument kwarg, `inst_id` (replaces `sat_id`)
* Allow limited use of new list_files kwarg, `file_cadance`
* Added support for `pysat.params['data_dirs']`, the replacement
for `pysat.data_dir` in pysat 3.0.0
* Deprecation warnings added to:
* Instrument class (old meta labels, `sat_id`, `default`, `multi_file_day`,
`manual_org`, and `_filter_datetime_input`, `data_dir`)
* pysat.instruments.methods.general.list_files kwarg
`fake_montly_files_from_daily`
* pysat.instruments.methods.testing.generate_times kwarg `sat_id`
* Constellation class kwarg `name`
* Custom class
* functions from `_files` class
* functions from `ssnl` module
* Instrument modules:
* the pysatCDAAC: cosmic_gps
* the pysatIncubator: champ_star, demeter_iap, superdarn_grdex,
supermag_magnetometer
* the pysatMadrigal: jro_isr, dmsp_ivm
* the pysatSpaceWeather: sw_dst, sw_f107, sw_kp
* the pysatModels: ucar_tiegcm
* the pysatNASA: cnofs_ivm, cnofs_plp, cnofs_vefi, de2_lang, de2_nacs,
de2_rpa, de2_wats, icon_euv, icon_fuv, icon_ivm, icon_might, iss_fpmu,
omni_hro, rocsat1_ivm, sport_ivm, timed_saber, timed_see
* The usage of a numeric string for `sat_id` to specify number of points
in test instruments
* SpaceWeather, Incubator (DEMETER), Madrigal, CDAWeb, and ICON instrument
methods
* The DE2 and ICON Constellation objects
* pysat.utils.set_data_dir
* Documentation
* Updated docstrings with deprecation notes
* Bug Fix
* Closes files after download in NASA CDAWeb methods
* Updated the ICON constellation object to reflect new instrument standards
* Added version cap for numpy

--------------------
* New Features
* netCDF4 files produced using `to_netcdf4()` now have an unlimited
time dimension
* Documentation
* Updated guidance on numpy version for installation
* Bug Fix
* Updated madrigal methods to simplify compound data types and enable
creation of netCDF4 files using `self.to_netcdf4()`.
* Updated ICON support for latest file organization on SSL servers