Mintpy

Cut a release to match with [Stephenson et al. (2022)](https://doi.org/10.1029/2022GL099835).

Technical Improvements

+ automate the `PyPI` publishing with GitHub Actions (827)
+ remove `openmp` from the dependency list (825)
+ bugfixes (825, 826, 829)

Contributors

+ Zhang Yunjun

Cut a release for the [2022 UNAVCO InSAR short course (ISCE+)](https://www.unavco.org/event/2022-short-course-insar-processing-analysis-isce/).

New Features

+ `closure_phase_bias`: add the phase non-closure induced bias estimation for small baseline approaches (Zheng et al., 2022) (765 by Y. Zheng, Z. Yunjun).
+ add `bulk_plate_motion` to estimate the tectonic plate motion in ENU/LOS direction from plate motion models (Stephenson et al., 2022, under review) (769, 808 by Y. K. Liu, Z. Yunjun).
+ `smallbaselineApp`: integrate complex time functions and uncertainty quantification into the routine workflow (795) via:
- mintpy.timeFunc.polynomial
- mintpy.timeFunc.periodic
- mintpy.timeFunc.stepDate
- mintpy.timeFunc.exp/log
- mintpy.timeFunc.uncertaintyQuantification
- mintpy.timeFunc.timeSeriesCovFile
+ `load_data`: load ionosphere stack from `isce2/topsStack` into `ionStack.h5` file with auto-resize (780, 796 by Y. K. Liu, Z. Yunjun) via:
- mintpy.load.ionUnwFile
- mintpy.load.ionCorFile
- mintpy.load.ionConnCompFile
+ `load_data`: support `mean` and `median` interpolation while loading stacks of observations (797 by Y. K. Liu) via:
- mintpy.multilook.x/ystep [renamed from mintpy.load.x/ystep]
- mintpy.multilook.method
+ add the single entry point with sub-commands via `mintpy.__main__` (823 by A. Valentino)
+ add `s1ab_range_bias.py` for the S1A/B range bias correction for range offset (Yunjun et al., 2022) (787).

Technical Improvements

+ auto build the `latest` and `develop` docker image and push to the [GitHub container registry](https://github.com/insarlab/MintPy/pkgs/container/mintpy) (#696 by J. Kennedy).
+ add `mintpy` to the [Debian ecosystem](https://tracker.debian.org/pkg/mintpy) with `apt install` support (#812 by A. Valentino).
+ add `mintpy.objects.ionex` for IONEX file download/read, to replace part of `mintpy.simulation.iono` (794).
+ `tsview`: switch to a new narrower slider and to use `fig.canvas.draw_idel/flush_events()` (764)
+ repo structure change:
- move the `requirements.txt` file to the root level (772)
- move shell scripts to the root level as `scripts` (785 by A. Valentino)
- spin-off `utils.map` module from `utils.plot` for lat/lon labels and scalebar (794)
+ reduced test data size for `SanFranSenDT42` and `WCapeSenAT29` (799)
+ various bugs fix

Contributors

+ Zhang Yunjun
+ Yuan-Kai Liu
+ Yujie Zheng
+ Antonio Valentino
+ Joseph Kennedy
+ Brett Buzzanga
+ Scott Staniewicz
+ Falk Amelung
+ Kang Liang
+ Bryan Marfito

Cut a new release for [Zheng et al. (2022)](https://doi.org/10.1109/TGRS.2022.3167648), [Yunjun et al. (2022)](https://doi.org/10.1109/TGRS.2022.3168509) and [Gregg et al. (2022)](https://doi.org/10.1126/sciadv.abm4261).

New Features:

+ add `closure_phase_bias.py` to mask areas susceptible to the closure phase bias (Zheng et al., 2022), contributed by Yujie Zheng
+ add `iono_tec.py` and `simulation/iono.py` for GIM-based ionospheric correction for range offset (Yunjun et al., 2022)
+ [preliminary] add TS full covariance matrix calculation and propagation, with contributions from Sara Mirzaee
+ File I/O:
- support 4D matrix IO in `readfile/writefile/cluster.py`
- support SLC stack as `FILE_TYPE==timeseries and DATA_TYPE==complex`
- initial `NO_DATA_VALUE` support as a new standard metadata
- `uavsar`: support UAVSAR `.ann/lkv/llh` files reading and solid Earth tides correction
- `prep_fringe`: add “connectComponent” to ifgramStack file if exists, contributed by Brett Buzzanga
+ Utilities:
- `add/diff/image_stitch`: generic multi-dataset file support, with contributions from Yuan-Kai Liu and Ollie Stephenson
- `asc_desc2horz_vert.py`: support decomposition with pixel-wised geometry via `--geom-file` option
- `geocode`: change the default geocoding output lat/lon step from “same row/col number” to “same pixel area“, with contributions from Brett Buzzanga (722, 737)
+ add `mintpy.unwrapError.connCompMinArea` option, contributed by Sara Mirzaee
+ visualization:
- plot_network: color-code network based on temp/perp baseline, contributed by Yuan-Kai Liu
- view: add `--no-data-value` option
- tsview: support `--coastline` via carroty map projection
- tsview: support very long TS files, e.g. TEC and GRACE datasets in h5 format.

Technical Improvements:

+ drop dependencies on `pykdtree`, `xarray` and `zarr` as they are now properly installed by `condo install pyresample`.
+ support `h5py>=3` by fixing the bool array indexing bug, contributed by Brett Buzzanga
+ support `python 3.9` and `python 3.10` via documentation change only
+ fix argv/args bug for conda/pip installed version
+ use numpy values as the standard for the string/object conversion between different packages
+ add the first unit test in `tests/test_asc_desc2horz_vert.py`
+ refactor the following scripts:
- `legacy/load2hdf5.py`
- `prep_snap.py`
- `objects/gps.get_gps_los_obs()`
+ other bugs fix

Contributors:

+ Zhang Yunjun
+ Yujie Zheng
+ Sara Mirzaee
+ Brett Buzzanga
+ Yuan-Kai Liu
+ Ollie Stephenson
+ Zel Hurewitz
+ Marin Govorcin
+ Andrew Johnston

New Features:

+ add `mintpy` to the `conda-forge` channel (https://anaconda.org/conda-forge/mintpy). Contributed by Joseph Kennedy.
+ simplify the installation:
- `setup.py`: use `entry_points/console_scripts` for a fully functioning `pip` installed local version. Contributed by Joseph Kennedy.
- update `docker` image. Contributed by Forrest Williams.
- simplify the installation note and environment files to leverage 1) pip install and 2) pysolid and pyapse on conda-forge, with contributions from Joseph Kennedy, James Rine, Andrew Johnston and Panji Brotoisworo.
+ big-data friendly features:
- improved memory handling for parallel plotting via `mintpy.plot.maxMemory` option
- improved stability for Dask parallelization via better patch splitting and flexible num of workers in percentage
+ `hyp3`: routine workflow works end-to-end in UTM coordinates.
- add `RidgecrestSenDT71` example dataset. Contributed by Jiang Zhu
- geometry and metadata bugs fixed.
+ more `offset` functionalities:
- support `cosicorr` offset products via `prep_cosicorr.py`. Contributed by Forrest Williams
- `load_data`: add offset STD from `ampcor`
- `ifgram_inversion`: add masking based on offset STD
- `ifgram_inversion`: add initial phase / offset STD time series calculation via `--calc-std`
+ `gnss` related:
- check observation time overlap percentage (>=25%) between GNSS and InSAR, to better discard abnormal GNSS sites
- add `--ex-gps` to manually exclude GNSS sites
- add `--mask-gps` option to mask out GNSS sites based on the displayed data matrix. Contributed by Simran Sangha
- support `--gps-comp horz / vert --hz-az` to plot horizontal and vertical components. Contributed Simran Sangha
+ add `utils/s1_utils.py` for Sentinel-1 specific utility functions
+ `save_gdal.py`: support UTM projection. Contributed by Panji Brotoisworo
+ improved visualization:
- `view.py`: add `--mask-dem` option to mask out DEM pixels based on the displayed data matrix. Contributed by Simran Sangha
- support American length unit, e.g. inch, foot, yard. Contributed by Simran Sangha.

Technical Improvements:

+ `readfile / writefile`: generic I/O logic for complex binary files to better handle ISCE-2 files directly
+ drop dependencies on `sklearn` by using numpy for bootstrap resampling. Contributed by Forrest Williams
+ `stack`: Speedup triplet design matrix. Contributed by Scott Staniewicz
+ `add.py`: handles time series with different dates and reference date/point, same as `diff.py`
+ `timeseries2velocity.py`: add `--save-res` to save the fitting residual files
+ new standard mintpy metadata keys: `BANDS` and `INTERLEAVE` for binary files
+ move time function related functions to `utils/time_func.py`
+ rename `test` directory to `tests`
+ bugs fix

Contributors:

+ Zhang Yunjun
+ Joseph Kennedy
+ Forrest Williams
+ Simran Sangha
+ Ollie Stephenson
+ Scott Staniewicz
+ Jiang Zhu
+ James Rine
+ Panji Brotoisworo
+ Yuan-Kai Liu
+ Andrew Johnston

Cut a new release for the 2021 UNAVCO short course.

New Features:
+ big-data friendly features:
- memory-efficient `tsview.py` via downsampling.
- CPU-efficient dask parallelization in `ifgram_inversion.py` and `dem_error.py` by limiting `*NUM_THREADS` env variable to 1.
+ support `hyp3` product from ASF HyP3 service [preliminary]
- load HyP3 products via `prep_hyp3.py` contributed by Forrest Williams, Bryan Marfito and Jiang Zhu.
- support UTM projection in `tropo_pyaps3.py` and `solid_earth_tides.py` contributed by Jiang Zhu and Bryan Marfito.
+ support `gmtsar` geocoded product via `prep_gmtsar.py` with contributions from Xiaohua Xu and Rebecca Bussard
+ add `save_kite.py` to save mintpy displacement for `pyrocko` suite contributed by Marin Govorcin
+ `timeseries2velocity.py`:
- add two time functions: exponential via `--exp` and logarithmic via `--log` contributed by Yuan-Kai Liu
- support STD propagation from time-series to time func
+ `modify_network.py`:
- add "Effective Coherence Ratio" method from Kang et al. (2021) contributed by Forrest Williams
- calc MST network from remaining pairs after all the other criteria applied contributed by Brett Buzzanga
+ utilities contributed by Forrest Williams:
- `spatial_filter.py`: add `double_difference` algorithm from Bekaert et al. (2020)
- `generate_mask.py`: add `--minpixel` option to select pixels based on cluster size
- `generate_mask.py`: add `--vstd` option to select pixels based on signal-to-noise ratio using velocity STD
+ visualizations:
- `save_kmz`: support radar-coded file with point styles
- `tsview.py`: plot & save the time function fitting results with contributions from Yuan-Kai Liu
- `view.py`:
- support `--ref-point` out of the subset box
- support `--ref-lalo` for radar-coded file
- support UTM projections
- add `--math` option for simple operations, e.g. sqrt to display STD from VAR file
- improved save & re-use GPS LOS data in CSV file
- plot colorbar with the unit in the label for each subplot in case of multi-subplots with different units
- add `--mask-vmin/vmax` options to adjust masking thresholds on the fly
- more colormaps

Technical Improvements:
+ translate `plot_smallbaselineApp.sh` from shell script to python, for better cross-platform compatibility with parallelization, contributed by Panji Brotoisworo and Joshua Zahner
+ improvement API support of `isce-2/topsApp` geocoded products contributed by Yuan-Kai Liu
+ improved support of isce-2 data files in the utility scripts (geocode.py, multilook.py and subset.py).
+ Circle CI:
- speedup installation using mamba
- separate datasets into different runs for better navigation
- platform-independent fortran compiler
+ Various bugs and typo fixes.

Contributors:
+ Zhang Yunjun
+ Forrest Williams
+ Yuan-Kai Liu
+ Xiaohua Xu
+ Marin Govorcin
+ Panji Brotoisworo
+ Bryan Marfito
+ Jiang Zhu
+ Brett Buzzanga
+ Joshua Zahner
+ Rebecca Bussard
+ Simran Sangha
+ Oliver Stephenson
+ Sara Mirzaee
+ Russell Grew
+ Andrew Johnston
+ James Rine

New Features

+ big-data friendly features:
- memory efficient `smallbaselineApp.py` (and all scripts it relied on) via block-by-block IO or downsampling controlled via `mintpy.compute.maxMemory` (4GB by default) with contributions from Yuan-Kai Liu
- parallel computing via dask extended to `dem_error.py` controlled via `mintpy.compute.cluster/numWorker/config`
- `mintpy.load.x/ystep` to multilook while loading data to reduce the data size
+ [`ISCE/alosStack`](https://github.com/isce-framework/isce2/blob/main/contrib/stack/alosStack/alosStack_tutorial.txt) products support (require isce2>=2.5) via `mintpy.load.processor = isce` contributed by Cunren Liang
+ `solid_earth_tides.py` for solid Earth tides correction using [`PySolid`](https://github.com/insarlab/PySolid) and integrate into the routine workflow
+ refactor `tropo_gacos.py` to bring back `GACOS` tropospheric correction and integrate into the routine workflow contributed by Sara Mirzaee and Simran Sangha
+ complex time functions in `timeseries2velocity.py` contributed by Yuan-Kai Liu
+ more example datasets and tests:
- `SanFranSenDT42` for `aria` contributed by Heresh Fattahi
- `WCapeSenAT129` for `snap` contributed by Andre Theron
+ diversity updates with contributions from Sara Mirzaee:
- replace oppressive language identified in the community with neutral language
- rename the default branch from `master` to `main`

Technical Improvements
+ support `mintpy.load.autoPath = yes` for `aria`
+ support load interferogram magnitude data via `mintpy.load.magFile`
+ support stitching non-phase files via `image_stitch.py --no-offset` contributed by Robert Zinke
+ improved time series analysis for offset stack
+ use `NCORRLOOKS` for weighting if available
+ improved documentation
+ cleanup sub-directory:
- remove `mintpy/objects/insarStack`
- remove `mintpy/legacy`
- remove `mintpy/dev`
- move `sh` to `mintpy/sh`
- move data/resource files from `docs` into `mintpy` so that the latter is independent from the former.
+ visualization:
- auto discretize colormap if input data has <= 20 unique values
- show colorer for multiple subplots with different units
+ preliminary version of `setup.py`
+ register `mintpy` to PyPI with contributions from Meissam Mehdizadeh
+ support adding missing attributes from template files directly for `load_data.py` and `save_hdfeos5.py`
+ new ASCII logo
+ files/scripts name change:
- match.py -> image_stitch.py
- ifgramStack_coherence_spatialCoh.txt -> coherenceSpatialAvg.txt
- numNonzeroIntClosure.h5 -> numTriNonzeroIntAmbiguity.h5
+ drop `python 3.5` support.
+ bugs fix

Contributors

+ Zhang Yunjun
+ Cunren Liang
+ Yuan-Kai Liu
+ Sara Mirzaee
+ Simran Sangha
+ Heresh Fattahi
+ Andre Theron
+ Xiaoran Lv
+ Robert Zinke
+ Meissam Mehdizadeh
+ Scott Staniewicz
+ Russel Grew
+ Bryan Marfito 
+ Lei Yuan