Neuroconv

Deprecations
* The usage of `compression_options` directly through the `neuroconv.tools.audio` submodule is now deprecated - users should refer to the new `configure_backend` method for a general approach for setting compression. [PR 939](https://github.com/catalystneuro/neuroconv/pull/939)
* The usage of `compression` and `compression_opts` directly through the `FicTracDataInterface` is now deprecated - users should refer to the new `configure_backend` method for a general approach for setting compression. [PR 941](https://github.com/catalystneuro/neuroconv/pull/941)
* The usage of `compression` directly through the `neuroconv.tools.neo` submodule is now deprecated - users should refer to the new `configure_backend` method for a general approach for setting compression. [PR 943](https://github.com/catalystneuro/neuroconv/pull/943)
* The usage of `compression_options` directly through the `neuroconv.tools.ophys` submodule is now deprecated - users should refer to the new `configure_backend` method for a general approach for setting compression. [PR 940](https://github.com/catalystneuro/neuroconv/pull/940)
* Removed the option of running `interface.run_conversion` without `nwbfile_path` argument . [PR 951](https://github.com/catalystneuro/neuroconv/pull/951)

Features
* Added docker image and tests for an automated Rclone configuration (with file stream passed via an environment variable). [PR 902](https://github.com/catalystneuro/neuroconv/pull/902)

Bug fixes
* Fixed the conversion option schema of a `SpikeGLXConverter` when used inside another `NWBConverter`. [PR 922](https://github.com/catalystneuro/neuroconv/pull/922)
* Fixed a case of the `NeuroScopeSortingExtractor` when the optional `xml_file_path` is not specified. [PR 926](https://github.com/catalystneuro/neuroconv/pull/926)
* Fixed `Can't specify experiment type when converting .abf to .nwb with Neuroconv`. [PR 609](https://github.com/catalystneuro/neuroconv/pull/609)
* Remove assumption that the ports of the Intan acquisition system correspond to electrode groupings in `IntanRecordingInterface` [PR 933](https://github.com/catalystneuro/neuroconv/pull/933)
* Add ValueError for empty metadata in `make_or_load_nwbfile` when an nwbfile needs to be created [PR 948](https://github.com/catalystneuro/neuroconv/pull/948)

Improvements
* Make annotations from the raw format available on `IntanRecordingInterface`. [PR 934](https://github.com/catalystneuro/neuroconv/pull/943)
* Add an option to suppress display the progress bar (tqdm) in `VideoContext` [PR 937](https://github.com/catalystneuro/neuroconv/pull/937)
* Automatic compression of data in the `LightnignPoseDataInterface` has been disabled - users should refer to the new `configure_backend` method for a general approach for setting compression. [PR 942](https://github.com/catalystneuro/neuroconv/pull/942)
* Port over `dlc2nwb` utility functions for ease of maintenance. [PR 946](https://github.com/catalystneuro/neuroconv/pull/946)

Bug fixes
* Added a skip condition in `get_default_dataset_io_configurations` for datasets with any zero-length axis in their `full_shape`. [PR 894](https://github.com/catalystneuro/neuroconv/pull/894)
* Added `packaging` explicitly to minimal requirements. [PR 904](https://github.com/catalystneuro/neuroconv/pull/904)
* Fixed bug when using `make_or_load_nwbfile` with `overwrite=True` on an existing (but corrupt) HDF5 file. [PR 911](https://github.com/catalystneuro/neuroconv/pull/911)
* Change error trigger with warning trigger when adding both `OnePhotonSeries` and `TwoPhotonSeries` to the same file ([Issue 906](https://github.com/catalystneuro/neuroconv/issues/906)). [PR #907](https://github.com/catalystneuro/neuroconv/pull/907)

Improvements
* Propagated `photon_series_type` to `BaseImagingExtractorInterface` init instead of passing it as an argument of `get_metadata()` and `get_metadata_schema()`. [PR 847](https://github.com/catalystneuro/neuroconv/pull/847)
* Converter working with multiple VideoInterface instances [PR 914](https://github.com/catalystneuro/neuroconv/pull/914)

Bug fixes
* Fixed bug causing overwrite of NWB GUIDE watermark. [PR 890](https://github.com/catalystneuro/neuroconv/pull/890)

Deprecations
* Removed `stream_id` as an argument from `IntanRecordingInterface`. [PR 794](https://github.com/catalystneuro/neuroconv/pull/794)
* The usage of `compression` and `compression_opts` directly through the `neuroconv.tools.spikeinterface` submodule are now deprecated - users should refer to the new `configure_backend` method for a general approach for setting compression. [PR 805](https://github.com/catalystneuro/neuroconv/pull/805)
* Dropped the testing of Python 3.8 on the CI. Dropped support for Python 3.8 in setup. [PR 853](https://github.com/catalystneuro/neuroconv/pull/853)
* Deprecated skip_features argument in `add_sorting`. [PR 872](https://github.com/catalystneuro/neuroconv/pull/872)
* Deprecate old (v1) iterator from the ecephys pipeline. [PR 876](https://github.com/catalystneuro/neuroconv/pull/876)

Features
* Added `backend` control to the `make_or_load_nwbfile` helper method in `neuroconv.tools.nwb_helpers`. [PR 800](https://github.com/catalystneuro/neuroconv/pull/800)
* Released the first official Docker images for the package on the GitHub Container Repository (GHCR). [PR 383](https://github.com/catalystneuro/neuroconv/pull/383)
* Support "one-file-per-signal" and "one-file-per-channel" mode with `IntanRecordingInterface`. [PR 791](https://github.com/catalystneuro/neuroconv/pull/791)
* Added `get_default_backend_configuration` method to all `DataInterface` classes. Also added HDF5 `backend` control to all standalone `.run_conversion(...)` methods for those interfaces. [PR 801](https://github.com/catalystneuro/neuroconv/pull/801)
* Added `get_default_backend_configuration` method to all `NWBConverter` classes. Also added HDF5 `backend` control to `.run_conversion(...)`. [PR 804](https://github.com/catalystneuro/neuroconv/pull/804)
* Released the first official Docker images for the package on the GitHub Container Repository (GHCR). [PR 383](https://github.com/catalystneuro/neuroconv/pull/383)
* Added `ScanImageMultiFileImagingInterface` for multi-file (buffered) ScanImage format and changed `ScanImageImagingInterface` to be routing classes for single and multi-plane imaging. [PR 809](https://github.com/catalystneuro/neuroconv/pull/809)
* Added a function to generate ogen timestamps and data from onset times and parameters to `tools.optogenetics`. [PR 832](https://github.com/catalystneuro/neuroconv/pull/832)
* Added `configure_and_write_nwbfile` and optimized imports in `tools.nwb_helpers` module. [PR 848](https://github.com/catalystneuro/neuroconv/pull/848)
* `configure_backend` may now apply a `BackendConfiguration` to equivalent in-memory `pynwb.NWBFile` objects that have different address in RAM. [PR 848](https://github.com/catalystneuro/neuroconv/pull/848)
* Add support for doubled ragged arrays in `add_units_table` [PR 879](https://github.com/catalystneuro/neuroconv/pull/879)
* Add support for doubled ragged arrays in `add_electrodes` [PR 881](https://github.com/catalystneuro/neuroconv/pull/881)
* Propagate `ignore_integrity_checks` from neo to IntanRecordingInterface [PR 887](https://github.com/catalystneuro/neuroconv/pull/887)


Bug fixes
* Remove JSON Schema `definitions` from the `properties` field. [PR 818](https://github.com/catalystneuro/neuroconv/pull/818)
* Fixed writing waveforms directly to file. [PR 799](https://github.com/catalystneuro/neuroconv/pull/799)
* Avoid in-place modification of the metadata in the `VideoInterface` and on neo tools. [PR 814](https://github.com/catalystneuro/neuroconv/pull/814)
* Replaced `waveform_extractor.is_extension` with `waveform_extractor.has_extension`. [PR 799](https://github.com/catalystneuro/neuroconv/pull/799)
* Fixed an issue with `set_aligned_starting_time` for all `SortingInterface`'s that did not have an initial segment start set (and no recording attached). [PR 823](https://github.com/catalystneuro/neuroconv/pull/823)
* Fixed a bug with `parameterized` and `pytest-xdist==3.6.1` in the `ScanImageImagingInterface` tests. [PR 829](https://github.com/catalystneuro/neuroconv/pull/829)
* Added `XX` and `XO` to the base metadata schema. [PR 833](https://github.com/catalystneuro/neuroconv/pull/833)
* `BaseImagingExtractor.add_to_nwbfile()` is fixed in the case where metadata is not supplied. [PR 849](https://github.com/catalystneuro/neuroconv/pull/849)
* Prevent `SpikeGLXConverterPipe` from setting false properties on the sub-`SpikeGLXNIDQInterface`. [PR 860](https://github.com/catalystneuro/neuroconv/pull/860)
* Fixed a bug when adding ragged arrays to the electrode and units table. [PR 870](https://github.com/catalystneuro/neuroconv/pull/870)
* Fixed a bug where `write_recording` will call an empty nwbfile when passing a path. [PR 877](https://github.com/catalystneuro/neuroconv/pull/877)
* Fixed a bug that failed to properly include time alignment information in the output NWB file for objects added from any `RecordingInterface` in combination with `stub_test=True`. [PR 884](https://github.com/catalystneuro/neuroconv/pull/884)
* Fixed a bug that prevented passing `nwbfile=None` and a `backend_configuration` to `NWBConverter.run_conversion`. [PR 885](https://github.com/catalystneuro/neuroconv/pull/885)

Improvements
* Added soft deprecation warning for removing `photon_series_type` from `get_metadata()` and `get_metadata_schema()` (in [PR 847](https://github.com/catalystneuro/neuroconv/pull/847)). [PR #866](https://github.com/catalystneuro/neuroconv/pull/866)
* Fixed docstrings related to backend configurations for various methods. [PR 822](https://github.com/catalystneuro/neuroconv/pull/822)
* Added automatic `backend` detection when a `backend_configuration` is passed to an interface or converter. [PR 840](https://github.com/catalystneuro/neuroconv/pull/840)
* Improve printing of bytes. [PR 831](https://github.com/catalystneuro/neuroconv/pull/831)
* Support for pathlib in source data schema validation. [PR 854](https://github.com/catalystneuro/neuroconv/pull/854)
* Use `ZoneInfo` instead of `dateutil.tz` in the conversion gallery. [PR 858](https://github.com/catalystneuro/neuroconv/pull/858)
* Exposed `progress_bar_class` to ecephys and ophys data iterators. [PR 861](https://github.com/catalystneuro/neuroconv/pull/861)
* Unified the signatures between `add_units`, `add_sorting` and `write_sorting`. [PR 875](https://github.com/catalystneuro/neuroconv/pull/875)
* Improved descriptions of all folder and file paths in the source schema, useful for rendering in the GUIDE. [PR 886](https://github.com/catalystneuro/neuroconv/pull/886)
* Added watermark via `source_script` field of `NWBFile` metadata. `source_script_file_name` is also required to be specified in this case to avoid invalidation. [PR 888](https://github.com/catalystneuro/neuroconv/pull/888)
* Remove parsing xml parsing from the `__init__` of `BrukerTiffSinglePlaneImagingInterface` [PR 895](https://github.com/catalystneuro/neuroconv/pull/895)

Testing
* Add general test for metadata in-place modification by interfaces. [PR 815](https://github.com/catalystneuro/neuroconv/pull/815)

Bug fixes
* Fixed writing the `electrodes` field in `add_electrical_series` when multiple groups are present. [PR 784](https://github.com/catalystneuro/neuroconv/pull/784)

Improvements
* Upgraded Pydantic support to `>v2.0.0`. [PR 767](https://github.com/catalystneuro/neuroconv/pull/767)
* Absorbed the `DatasetInfo` model into the `DatasetIOConfiguration` model. [PR 767](https://github.com/catalystneuro/neuroconv/pull/767)
* Keyword argument `field_name` of the `DatasetIOConfiguration.from_neurodata_object` method has been renamed to `dataset_name` to be more consistent with its usage. This only affects direct initialization of the model; usage via the `BackendConfiguration` constructor and its associated helper functions in `neuroconv.tools.nwb_helpers` is unaffected. [PR 767](https://github.com/catalystneuro/neuroconv/pull/767)
* Manual construction of a `DatasetIOConfiguration` now requires the field `dataset_name`, and will be validated to match the final path of `location_in_file`. Usage via the automated constructors is unchanged. [PR 767](https://github.com/catalystneuro/neuroconv/pull/767)
* Enhance `get_schema_from_method_signature` to extract descriptions from the method docval. [PR 771](https://github.com/catalystneuro/neuroconv/pull/771)
* Avoid writing `channel_to_uV` and `offset_to_uV` in `add_electrodes` [PR 803](https://github.com/catalystneuro/neuroconv/pull/803)
* `BaseSegmentationExtractorInterface` now supports optional background plane segmentations and associated fluorescence traces [PR 783](https://github.com/catalystneuro/neuroconv/pull/783)

Deprecation
* Removed `.get_electrode_table_json()` on the `BaseRecordingExtractorInterface` in favor of GUIDE specific interactions. [PR 431](https://github.com/catalystneuro/neuroconv/pull/431)
* Removed the `SIPickleRecordingInterface` and `SIPickleSortingInterface` interfaces. [PR 757](https://github.com/catalystneuro/neuroconv/pull/757)
* Removed the `SpikeGLXLFPInterface` interface. [PR 757](https://github.com/catalystneuro/neuroconv/pull/757)

Bug fixes
* LocalPathExpander matches only `folder_paths` or `file_paths` if that is indicated in the passed specification. [PR 679](https://github.com/catalystneuro/neuroconv/pull/675) and [PR #675](https://github.com/catalystneuro/neuroconv/pull/679
* Fixed depth consideration in partial chunking pattern for the ROI data buffer. [PR 677](https://github.com/catalystneuro/neuroconv/pull/677)
* Fix mapping between channel names and the electrode table when writing more than one `ElectricalSeries` to the NWBFile. This fixes an issue when the converter pipeline of `SpikeGLXConverterPipe` was writing the electrode table region of the NIDQ stream incorrectly. [PR 678](https://github.com/catalystneuro/neuroconv/pull/678)
* Fix `configure_backend` when applied to `TimeSeries` contents that leverage internal links for `data` or `timestamps`. [PR 732](https://github.com/catalystneuro/neuroconv/pull/732)

Features
* Changed the `Suite2pSegmentationInterface` to support multiple plane segmentation outputs. The interface now has a `plane_name` and `channel_name` arguments to determine which plane output and channel trace add to the NWBFile. [PR 601](https://github.com/catalystneuro/neuroconv/pull/601)
* Added `create_path_template` and corresponding tests [PR 680](https://github.com/catalystneuro/neuroconv/pull/680)
* Added tool function `configure_datasets` for configuring all datasets of an in-memory `NWBFile` to be backend specific. [PR 571](https://github.com/catalystneuro/neuroconv/pull/571)
* Added `LightningPoseConverter` to add pose estimation data and the original and the optional labeled video added as ImageSeries to NWB. [PR 633](https://github.com/catalystneuro/neuroconv/pull/633)
* Added gain as a required `__init__` argument for `TdtRecordingInterface`. [PR 704](https://github.com/catalystneuro/neuroconv/pull/704)
* Extract session_start_time from Plexon `plx` recording file. [PR 723](https://github.com/catalystneuro/neuroconv/pull/723)

Improvements
* `nwbinspector` has been removed as a minimal dependency. It becomes an extra (optional) dependency with `neuroconv[dandi]`. [PR 672](https://github.com/catalystneuro/neuroconv/pull/672)
* Added a `from_nwbfile` class method constructor to all `BackendConfiguration` models. [PR 673](https://github.com/catalystneuro/neuroconv/pull/673)
* Added compression to `FicTracDataInterface`. [PR 678](https://github.com/catalystneuro/neuroconv/pull/678)
* Exposed `block_index` to all OpenEphys interfaces. [PR 695](https://github.com/catalystneuro/neuroconv/pull/695)
* Added support for `DynamicTable` columns in the `configure_backend` tool function. [PR 700](https://github.com/catalystneuro/neuroconv/pull/700)
* Refactored `ScanImagingInterface` to reference ROIExtractors' version of `extract_extra_metadata`. [PR 731](https://github.com/catalystneuro/neuroconv/pull/731)
* Added support for Long NHP probe types for the `SpikeGLXRecorddingInterfacce`. [PR 701](https://github.com/catalystneuro/neuroconv/pull/701)
* Remove unnecessary duplication of probe setting in `SpikeGLXRecordingInterface`. [PR 696](https://github.com/catalystneuro/neuroconv/pull/696)
* Added associated suffixes to all interfaces and converters. [PR 734](https://github.com/catalystneuro/neuroconv/pull/734)
* Added convenience function `get_format_summaries` to `tools.importing` (and exposed at highest level). [PR 734](https://github.com/catalystneuro/neuroconv/pull/734)

Testing
* `RecordingExtractorInterfaceTestMixin` now compares either `group_name`, `group` or a default value of `ElectrodeGroup` to the `group` property in the `NWBRecordingExtractor` instead of comparing `group` to `group` as it was done before [PR 736](https://github.com/catalystneuro/neuroconv/pull/736)
* `TestScanImageImagingInterfaceRecent` now checks metadata against new roiextractors implementation [PR 741](https://github.com/catalystneuro/neuroconv/pull/741).
* Removed editable installs from the CI workflow. [PR 756](https://github.com/catalystneuro/neuroconv/pull/756)