Eoreader

Optimizations

- OPTIM: Try to create `Vision-1` footprint from the preview file instead of from the stack.
- OPTIM: Create footprints for stacked products (i.e. `Maxar`, `SkySat`, `Custom`...) without mask by opening only the first band of the stack
- OPTIM: Create footprints for `Maxar` Products with a resolution 10 times lower.
- OPTIM: Footprints have now maximum 50 vertices in order to avoid pixelized footprints

Bug Fixes

- FIX: Fixing condensed name to avoid duplicates:
- adding the `job_id` for `VHR` products
- adding the polarization channels for `SAR` products
- FIX: Remove import of pystac in `stac_utils`
- FIX: Fix bug for `Vision-1` data looking for non-existing RPC files in case of `ORTP` product type
- FIX: Fix quicklook regex for `Vision-1` data
- FIX: Fix regex for raw bands for extracted `Sentinel-3 OLCI` products
- FIX: Fix `PlanetScope` identifying regex to handle products with a satellite_id containing a letter
- FIX: Force metadata regex for `Maxar` products to look for a file with pattern `{name}.XML` to avoid other misplaced XML to be found in place of the true XML.
- FIX: Fix regression for `Landsat-7` footprint
- FIX: Manage the case with `cloud_cover = -999.0` for `Maxar` products (returns `None`)

Other

- CI: Add new optical products to be tested for end-to-end tests

Breaking Changes

- **BREAKING CHANGES: `Optical` becomes `Spectral` when more appropriate**
- **BREAKING CHANGES: `Platform` and `Sensor` become `Constellation` when more appropriate, to fit STAC vocabulary** ([29](https://github.com/sertit/eoreader/issues/29)):
- `Platform` enum becomes `Constellation`
- `prod.platform` becomes `prod.constellation`
- `prod.sat_id` becomes `prod.constellation_id`
- **BREAKING CHANGES: File `alias` is removed, replaced by `*_bands` files and proper imports in `bands.__init__`**
- **BREAKING CHANGES: Product attribute `band_names` becomes `bands` in order to be STAC compliant ([29](https://github.com/sertit/eoreader/issues/29))**
- **BREAKING CHANGES: Better use of `NIR` and `NARROW_NIR` in the `indices` file (according to the gsd of `Sentinel-2` bands composing the indices)**
- **BREAKING CHANGES: Correcting Landsat product types to better manage processing levels and instrument. Landsat-8/9 condensed name may change!**

Enhancements

- **ENH: Adding the support of `SkySat` (Collect) products** ([20](https://github.com/sertit/eoreader/issues/20))
- **ENH: Bands in mapping are now objects, instead of just IDs** ([29](https://github.com/sertit/eoreader/issues/29)). This allows us to:
- Add band metadata (such as center wavelength, bandwidth...)
- Map spectral bands between STAC spec and EOReader format ([29](https://github.com/sertit/eoreader/issues/29))
- Add a better `__repr__` functions
- **ENH: Handling 8 bands `PlanetScope` data** ([20](https://github.com/sertit/eoreader/issues/20))
- **ENH: Adding the `GREEN1` mapped band, corresponding to PlanetScope `GREEN I` and `Sentinel-3 OLCI` `Oa05` band**
- **ENH: Handle some slightly broken `Sentinel-2` products:**
- when the metadata files are corrupted or when the detfoo vectors are empty ([34](https://github.com/sertit/eoreader/issues/34))
- with missing MSK prefix for QI_DATA files (i.e. `DETFOO` instead of `MSK_DETFOO`)
- **ENH: Handle exception for corrupted bands (in `Sentinel-2` and `utils.read`) ([34](https://github.com/sertit/eoreader/issues/34))**
- **ENH: Add a STAC object that can be used to retrieve STAC Items from every Product (`prod.stac.create_item()`) ([29](https://github.com/sertit/eoreader/issues/29))**
- **ENH: Add a `get_mean_viewing_angles` for Optical Products to fill STAC View Extension ([29](https://github.com/sertit/eoreader/issues/29))**
- **ENH: Extending `get_raw_band_paths` to every product ([31](https://github.com/sertit/eoreader/issues/31))**
- **ENH: Adding a `is_ortho` attribute corresponding to when the product is already orthorectified/geocoded, in order to avoid computing heavy processes without wanting it (i.e. footprint...)**
- **ENH: Adding the instrument name of every constellation, under `prod.instrument`**
- **ENH: Handling `COSMO` product with only the `h5` file in it (if missing XML metadata file)** ([36](https://github.com/sertit/eoreader/issues/36))

Optimizations

- OPTIM: Retrieve name from filename if possible
- OPTIM: Retrieve extent from metadata when possible (for VHR data)

Bug Fixes

- FIX: Fixing the band mapping of `WorldView-2/3 Multi` (8 bands)
- FIX: Retrieval (if possible) of Sentinel-1 [unique ID](https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-1-sar/naming-conventions) (was missing from the product name, as it is not in
the product preview)
- FIX: Fixing PAZ/TDX MTD regex
- FIX: Optical products: Only set cloud cover and radiometry attributes if spectral bands are asked

Other

- INTERNAL: File `spot_6` and `spot_7` are removed, replaced by a unique `spot` file. This shouldn't affect the user.
- INTERNAL: Refactoring Landsat-XX products into `LandsatProduct`, this should be invisible for user.
- INTERNAL: Some refactoring in `VHR` files
- WARNINGS: Filter warnings from `__init__`
- CI: Do not process two times the zipped Sentinel-1 in end-to-end tests and manage when the runner kills SNAP
- DOC: Adding a STAC notebook
- DOC: Various updates

Breaking Changes

- **BREAKING CHANGES: `footprint`, `extent`, `wgs84_extent` and `crs` properties are converted back to methods in order to prevent side effects of expensive computation when displaying the object when
debugging (rollback before version 0.8.0)**
- **BREAKING CHANGES: `get_all_index` becomes `get_all_indices`**
- **BREAKING CHANGES: `acquisition_datetime` becomes `datetime` and `default_resolution`becomes `resolution` for `CustomProduct` in accepted keywords, and the metadata fields have been renamed
according to the `CustomFields` enum**

Enhancements

- **ENH: Adding spectral indices:**
- Shadow Index (`SI`)
- Global Vegetation Moisture Index (`GVMI`)
- Soil Brightness Index (`SBI`), Soil Cuirass Index (`SCI`)
- Panchromatic mocking Index (`PANI`)
- Green-to-Red ratio Index (`GRI`)
- Soil Adjusted Vegetation Index (`SAVI`)
- Optimized Soil Adjusted Vegetation Index (`OSAVI`)
- Visible Atmospherically Resistant Index (Green) (`VARI`)
- Enhanced Vegetation Index (`EVI`)
- Chlorophyll Index RedEdge VRE_3/VRE_2 (`CI1`)
- Chlorophyll Index RedEdge VRE_2/VRE_1 (`CI2`)
- Normalized Difference Moisture Index (with SWIR_21) (`NDMI21`)
- **ENH: Making SAR attribute `snap_filename` public**
- **ENH: Handling `ICEYE` pure SLC products**
- **ENH: Allowing the user to choose if they want the GRD or SLC image for `ICEYE` products**
- **ENH: Add the possibility to directly load the cloud cover for optical data (and add it in the band attributes) ([28](https://github.com/sertit/eoreader/issues/28))**
- **ENH: Add the possibility to retrieve the quicklook path (if existing) and add the `plot` function allowing the user to plot the quicklook (if
existing) ([28](https://github.com/sertit/eoreader/issues/28))**
- **ENH: Add the possibility to retrieve the orbit direction (and add it in the band attributes) ([28](https://github.com/sertit/eoreader/issues/28))**

Bug Fixes

- FIX: Fixing the inversion between `8` and `8A` bands for `Sentinel-2` and `Sentinel-2 Theia` products
- FIX: Loading every optical band in reflectance (fixed for `Sentinel-2 THEIA`, `Maxar`, `Planet` and `Vision-1` data) ([30](https://github.com/sertit/eoreader/issues/30))
- FIX: Fixing `ReferenceError: weakly-referenced object no longer exists` when deleting an object
- FIX: Do not set sea values to nodata when orthorectifying SAR data with SNAP
- FIX: Handle `Sentinel-2` data with PB < 02.07 as `L2Ap` products
- FIX: Fixing nodata and offset for `Sentinel-2` data with PB > 04.00
- FIX: Handle new `ICEYE` metadata name's nomenclature
- FIX: Fixing harmless regex error when searching for B1 path for `Landsat` products
- FIX: Fixing platform for `Sentinel-2 Theia`

Other

- DOC: Creating a real `base` notebook and renaming the old one to `optical`
- DOC: Better type hints (replacing `XDS_TYPE` by `xr.DataArray`)
- CI: Using `sertit.ci.reduce_verbosity` instead of recreating the function

Bug Fixes

- FIX: Handling `Sentinel-2 L2Ap` data
- FIX: Do not use `--no-binary fiona,rasterio` directly in `requirements.txt` (breaks on Windows)
- FIX: Fixing stacking with string bands
- FIX: Better `__repr__` function
- FIX: Read README as UTF-8 in setup.py

Other

- CI: Adding a tag for choosing the runners
- DOC: Fixing cartopy/GEOS conflicts making the documentation build to fail

Enhancements

- **ENH: Adding the support of `Landsat-9` sensor**
- **ENH: Support Sentinel-2 with missing datatake metadata file(sometimes happens with data downloaded from AWS buckets and converted to .SAFE)**

Bug Fixes

- FIX: Using default SAR resolution from
official [Copernicus Data Access Portfolio (2014-2022)](https://spacedata.copernicus.eu/documents/20126/0/DAP+Release+phase2+V2_8.pdf/82297817-2b96-d3de-c397-776292336434?t=1633508426589) (
Sentinel-2 default
resolution goes to 10.0 m !)
- FIX: Use `--no-binary fiona,rasterio` directly in `requirements.txt`
- FIX: Removing useless `outputComplex` line in GPT graphs that is breaking SNAP on Linux
- FIX: Removing the workarounds caused by some bugs of `cloudpathlib` and enabling retrieval of nested SAR products (TSX, TDX, PAZ, RCM) from S3 compatible storage.
- FIX: Do not process nodata for a band already existing
- FIX: Fixing an error when reading `TIR` bands with Landsat-7
- FIX: Fixing an error when additive/multiplicative coefficients are set to `NULL` for Landsat data
- FIX: Returning sun angles always as float (some `Sentinel-3` angles were returned as `np.array`)

Other

- CI: Do not try to process SAR end to end if GPT cannot be found
- CI: Publishing wheel from GitHub instead of Gitlab
- REPO: Setting GitHub as the main repository and using new Gitlab runners

Enhancements

- **ENH: Adding the support of `Pleiades-Neo`, `Vision-1` and `SAOCOM` sensors**
- **ENH: Adding a keyword to allow passing a specific DEM path in `load`/`stack` (for VHR orthorectification and `DEM` bands)**
- **ENH: Adding the name of the DEM in DEM band (i.e. allow to compute the `HILLSHADE` with a DEM and the `SLOPE` with a DTM)**

Bug Fixes

- FIX: `Sentinel-2` Processing Baseline 04.00: `NARROW_NIR` bands are now loaded correctly
- FIX: `Maxar` products (with `Multi` band ID) are now correctly handled
- FIX: Using `COPDEM-30` (`GLO-30`) by default for SNAP as it appears that the retrieval has been fixed.
- FIX: Fixing the default name for cleaned bands for `Sentinel-3 SLSTR` data (was set on `CLEAN` instead of `NODATA`)
- FIX: Fixing default band for Custom stacks
- FIX: Fixing `get_existing_band_paths` behavior for Custom stacks
- FIX: Remove other never covered lines of code (archived `RCM` products, complex `ICEYE` products, others...)
- FIX: Re-enabling loading str bands (regression)
- FIX: Proper check for empty fields when parsing metadata
- FIX: VHR `_get_dem_path` raises `ValueError` instead of `TypeError`
- FIX: Pre-process SAR bands before despeckling if not existing (was OK in most of the cases, but broke in some cases, especially with CI folder activated and S3 compatible storage)
- FIX: Remove warning `invalid escape sequence \.`, `\w`, `\D` and `\s`
- FIX: Do not set `long_name` for `RAW_CLOUDS` arrays
- FIX: Providing a URL DEM on Windows throws a `OSError` instead of a bare `Exception`

Optimizations

- OPTIM: Do not pre-process existing Sentinel-3 geocoded bands
- OPTIM: Do not look for valid metadata further than a given nested level in product's directory (for extracted products)

Other

- CI: Using another (faster) runner
- CI: Add on disk and end-to-end tests
- CI: Do not write tmp files when running on disk tests
- CI: Coverage:
- Get coverage as HTML
- Remove useless lines from coverage
- Combine coverage of S3 and on disk tests
- DOC: Adding a DEM notebook