Eoreader

Breaking Changes

- **BREAKING CHANGES: Refactoring spectral indices management** ([47](https://github.com/sertit/eoreader/issues/47))
- Using [spyndex](https://github.com/awesome-spectral-indices/spyndex) library, allowing to use all spectral indices
listed [here](https://github.com/awesome-spectral-indices/awesome-spectral-indices/blob/main/output/spectral-indices-table.csv)
- SAR products may now compute indices if possible (see [this list](https://awesome-ee-spectral-indices.readthedocs.io/en/latest/list.html#radar))
- Old EOReader indices are still available for legacy purposes, with some changes:
- For Sentinel-2 data, the band `NIR` and `NARROW_NIR` may be interchanged for some index (
see [this discussion](https://github.com/awesome-spectral-indices/awesome-spectral-indices/issues/27))
- OSAVI formula has changed to stick with the original paper definition (see [issue](https://github.com/awesome-spectral-indices/awesome-spectral-indices/issues/12))
- `NDRE2/3` formula are fixed, now using `VRE_2/3` and `NDRE1` corresponds to `NDREI` and uses `VRE_1`
- `CI1` is renamed `CI32` and `CI2` is renamed `CI21` for readability purposes
- `NDWI21` can be written `NDWI2100` for homogeneity purposes
- `RDI` (or `DSI`) uses now `SWIR_1` instead of `SWIR_2` (see [this](https://github.com/awesome-spectral-indices/awesome-spectral-indices/issues/18) issue)
- `PANI` equivalent is now `BITM` and is normalised ! (divided by 3)
- `SBI` is normalized (divided by 2) to fit with `BIXS` definition
- ⚠ *You may need to install the last `spyndex` directly from GitHub latest version to have all available indices*
- **BREAKING CHANGES: Using `pyresample` to geocode Sentinel-3 data** ([55](https://github.com/sertit/eoreader/issues/55))
- Cleaner: better conversion from swath to grid
- Faster: Up to 4 times faster
- Allows code refactoring between OLCI and SLSTR
- **BREAKING CHANGES: For SAR product types that are not available in the Data Access Portfolio, default resolution is now the pixel spacing instead of the rg x az resolution**
- Changes mainly Sentinel-1 default resolutions (except from IW mode)

Enhancements

- **ENH: Adding the support of Harmonized Landsat-Sentinel constellation** ([49](https://github.com/sertit/eoreader/issues/49))
- **ENH: Adding the support of GEOSAT-2 constellation** ([59](https://github.com/sertit/eoreader/issues/59))

Bug Fixes

- FIX: Fixing `CustomProduct` initialization when fields are set to None (instead of not declaring them)
- FIX: SNAP cannot handle float predictors other than 1! Set it to 1 when saving ortho SAR images to disk, in order for SNAP to be able to despeckle
them. See [SNAP issue](https://forum.step.esa.int/t/exception-found-when-reading-compressed-tif/654/7). ([#62](https://github.com/sertit/eoreader/issues/62))
- FIX: Fixing mix in `Sentinel-2` mapping for `B8` (`NIR`, 10m resolution, large spectral bandwidth) and `B8A` (`NARROW_NIR`, 20m resolution, narrow spectral bandwidth)

Other

- DOC: Add FAQ entry concerning SAR constellations extent KML files failing to be read (TLDR: needs `ogr2ogr` in your
PATH)
- DOC: Add Technical Note published in Remote Sensing MDPI in Readme
- DOC: Update optical band mapping graphs (fix regression to 0.15.0 supported constellation)
- DOC: Add information about DEM management in SAR notebook ([61](https://github.com/sertit/eoreader/issues/61))
- DOC: Updating indices paragraphs
- CI: Using actions/checkoutv3
- CI: Updating versions of pre-commit hooks
- LIBS: Updating `requirements.txt` and `setup.py` to add `pyresample` and `zarr`

Enhancements

- **ENH: Adding the support of RapidEye constellation**
- **ENH: Handling Planet data with multiple subdatasets** ([45](https://github.com/sertit/eoreader/issues/45))
- **ENH: Adding the support of Landsat Level-2 products** ([49](https://github.com/sertit/eoreader/issues/49))
- **ENH: Adding the support of Pleiades-Neo SEN and PRJ products** *(needs GDAL 3.5+ or rasterio 1.3.0+)*
- **ENH: Adding the function `bands.is_thermal_band`**
- **ENH: Adding the ability for optical custom stacks to load indices**
- **ENH: Adding [BAIM (MODIS Burned Area Index)](https://www.researchgate.net/publication/248428333_Burnt_Area_Index_BAIM_for_burned_area_discrimination_at_regional_scale_using_MODIS_datafire)
spectral index**
- **ENH: Better management of raw units of the bands of optical products**
- **ENH: Copying files from `tmp_process` when changing product's output**

Bug Fixes

- FIX: Stacks saved as integers on disk keep their original dtype (float32) in Python
- FIX: Stacks with bands loaded "as is" are correctly saved as integers on disk ([52](https://github.com/sertit/eoreader/issues/52))
- FIX: Using stack CRS (if projected) for `DIMAP` products instead of recomputing from lat/lon, solving potential discrepancies between stack and product CRS
- FIX: Workaround for JP2 bug when updating an existing raster (maybe related to [this bug](https://github.com/rasterio/rasterio/issues/2528))
- FIX: Better management of SkySat datetime conversion from JSON to XML (deterministic way)
- FIX: Fixing computation of invalid pixels for `Sentinel-2` and `DIMAP` products (do not remove straylight mask)
- FIX: Fixing reprojection resolution of VHR data
- FIX: Computing Brightness Temperature of `Landsat` TIR bands instead of leaving them as is
- FIX: Better management of Landsat Instrument values
- FIX: Better radiometry attribute (adding `brightness temperature` and `reflectance and brightness temperature` values)
- FIX: Changing `Brilliance Temperature` to the correct `Brightness Temperature`
- FIX: Fixing pandas FutureWarning `The frame.append method is deprecated and will be removed from pandas in a future version.`
- FIX: Fixing DeprecationWarning `invalid escape sequence \.`
- FIX: Manage correctly Planet dubious pixels [(especially for 8 bands products)](https://community.planet.com/planet-s-community-forum-3/planetscope-8-bands-and-udm-mask-245?postid=427#post427)

Optimizations

- OPTIM: Reduce memory usage when updating all the bands attributes
- OPTIM: Reduce memory usage when stacking as integers

Other

- DOC: Add the need of using SNAP 8.0 up-to-date or SNAP 9.0 ([42](https://github.com/sertit/eoreader/issues/42))
- DOC: Add the STAC session in API documentation
- DOC: Add warnings for shifts when orthorectifying DIMAP SEN products (using RPCs) ([53](https://github.com/sertit/eoreader/issues/53))
- DOC: Add limitations to custom stacks
- DEPS: Dropping support of Python 3.7 ([18](https://github.com/sertit/eoreader/issues/18))
- DEPS: Update minimum version of libs *(geopandas 0.11.0+, rasterio 1.3.0+...)*

Bug Fixes

- FIX: Add the missing conversion to reflectance for `Sentinel-3 OLCI`
- FIX: Better condition for the conversion to reflectance for `Sentinel-2 THEIA`
- FIX: Add logs for `SkySat` data that cannot been converted to reflectance and fix the `radiometry` field of its band xarrays
- FIX: Add the correct nodata (when overridden by the user) to stacks saved as uint16

Optimizations

- OPTIM: Reduce memory usage during stacking

Other

- CI: Test reflectance values

Enhancements

- **ENH: Adding the support of SuperView-1 constellation ([21](https://github.com/sertit/eoreader/issues/21))**
- **ENH: Adding the support of SPOT-4/5 constellations ([39](https://github.com/sertit/eoreader/issues/39))**
- **ENH: Allow the possibility to pass a constellation (or a constellation list) to `Reader().open()` to speed up the opening of a product**
- **ENH: Add a quicklook search for `Sentinel-3` products**

Bug Fixes

- FIX: Fix quicklook media type with `JP2` files
- FIX: Fix `Sentinel-3 SLSTR` `F1` bands based on F grid
- FIX: Correct the UTM projection for `Sentinel-3` data
- FIX: Fix handling of zipped `Sentinel-2 L2Ap`
- FIX: Fix zipped `Sentinel-2` with other XML files in GRANULE subdirectories

Other

- Renaming `master` branch to `main`

Optimizations

- OPTIM: Try to create `Vision-1` footprint from the preview file instead of from the stack.
- OPTIM: Create footprints for stacked products (i.e. `Maxar`, `SkySat`, `Custom`...) without mask by opening only the first band of the stack
- OPTIM: Create footprints for `Maxar` Products with a resolution 10 times lower.
- OPTIM: Footprints have now maximum 50 vertices in order to avoid pixelized footprints

Bug Fixes

- FIX: Fixing condensed name to avoid duplicates:
- adding the `job_id` for `VHR` products
- adding the polarization channels for `SAR` products
- FIX: Remove import of pystac in `stac_utils`
- FIX: Fix bug for `Vision-1` data looking for non-existing RPC files in case of `ORTP` product type
- FIX: Fix quicklook regex for `Vision-1` data
- FIX: Fix regex for raw bands for extracted `Sentinel-3 OLCI` products
- FIX: Fix `PlanetScope` identifying regex to handle products with a satellite_id containing a letter
- FIX: Force metadata regex for `Maxar` products to look for a file with pattern `{name}.XML` to avoid other misplaced XML to be found in place of the true XML.
- FIX: Fix regression for `Landsat-7` footprint
- FIX: Manage the case with `cloud_cover = -999.0` for `Maxar` products (returns `None`)

Other

- CI: Add new optical products to be tested for end-to-end tests

Breaking Changes

- **BREAKING CHANGES: `Optical` becomes `Spectral` when more appropriate**
- **BREAKING CHANGES: `Platform` and `Sensor` become `Constellation` when more appropriate, to fit STAC vocabulary** ([29](https://github.com/sertit/eoreader/issues/29)):
- `Platform` enum becomes `Constellation`
- `prod.platform` becomes `prod.constellation`
- `prod.sat_id` becomes `prod.constellation_id`
- **BREAKING CHANGES: File `alias` is removed, replaced by `*_bands` files and proper imports in `bands.__init__`**
- **BREAKING CHANGES: Product attribute `band_names` becomes `bands` in order to be STAC compliant ([29](https://github.com/sertit/eoreader/issues/29))**
- **BREAKING CHANGES: Better use of `NIR` and `NARROW_NIR` in the `indices` file (according to the gsd of `Sentinel-2` bands composing the indices)**
- **BREAKING CHANGES: Correcting Landsat product types to better manage processing levels and instrument. Landsat-8/9 condensed name may change!**

Enhancements

- **ENH: Adding the support of `SkySat` (Collect) products** ([20](https://github.com/sertit/eoreader/issues/20))
- **ENH: Bands in mapping are now objects, instead of just IDs** ([29](https://github.com/sertit/eoreader/issues/29)). This allows us to:
- Add band metadata (such as center wavelength, bandwidth...)
- Map spectral bands between STAC spec and EOReader format ([29](https://github.com/sertit/eoreader/issues/29))
- Add a better `__repr__` functions
- **ENH: Handling 8 bands `PlanetScope` data** ([20](https://github.com/sertit/eoreader/issues/20))
- **ENH: Adding the `GREEN1` mapped band, corresponding to PlanetScope `GREEN I` and `Sentinel-3 OLCI` `Oa05` band**
- **ENH: Handle some slightly broken `Sentinel-2` products:**
- when the metadata files are corrupted or when the detfoo vectors are empty ([34](https://github.com/sertit/eoreader/issues/34))
- with missing MSK prefix for QI_DATA files (i.e. `DETFOO` instead of `MSK_DETFOO`)
- **ENH: Handle exception for corrupted bands (in `Sentinel-2` and `utils.read`) ([34](https://github.com/sertit/eoreader/issues/34))**
- **ENH: Add a STAC object that can be used to retrieve STAC Items from every Product (`prod.stac.create_item()`) ([29](https://github.com/sertit/eoreader/issues/29))**
- **ENH: Add a `get_mean_viewing_angles` for Optical Products to fill STAC View Extension ([29](https://github.com/sertit/eoreader/issues/29))**
- **ENH: Extending `get_raw_band_paths` to every product ([31](https://github.com/sertit/eoreader/issues/31))**
- **ENH: Adding a `is_ortho` attribute corresponding to when the product is already orthorectified/geocoded, in order to avoid computing heavy processes without wanting it (i.e. footprint...)**
- **ENH: Adding the instrument name of every constellation, under `prod.instrument`**
- **ENH: Handling `COSMO` product with only the `h5` file in it (if missing XML metadata file)** ([36](https://github.com/sertit/eoreader/issues/36))

Optimizations

- OPTIM: Retrieve name from filename if possible
- OPTIM: Retrieve extent from metadata when possible (for VHR data)

Bug Fixes

- FIX: Fixing the band mapping of `WorldView-2/3 Multi` (8 bands)
- FIX: Retrieval (if possible) of Sentinel-1 [unique ID](https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-1-sar/naming-conventions) (was missing from the product name, as it is not in
the product preview)
- FIX: Fixing PAZ/TDX MTD regex
- FIX: Optical products: Only set cloud cover and radiometry attributes if spectral bands are asked

Other

- INTERNAL: File `spot_6` and `spot_7` are removed, replaced by a unique `spot` file. This shouldn't affect the user.
- INTERNAL: Refactoring Landsat-XX products into `LandsatProduct`, this should be invisible for user.
- INTERNAL: Some refactoring in `VHR` files
- WARNINGS: Filter warnings from `__init__`
- CI: Do not process two times the zipped Sentinel-1 in end-to-end tests and manage when the runner kills SNAP
- DOC: Adding a STAC notebook
- DOC: Various updates