Libertem

Changelog: https://libertem.github.io/LiberTEM/changelog.html#v0-4-0

The main points of this release are the [Job API deprecation](https://libertem.github.io/LiberTEM/changelog.html#job-deprecation) and restructuring of our packaging, namely [extracting the blobfinder module](https://libertem.github.io/LiberTEM/changelog.html#restructuring-0-4).

New features
* dtype support for UDFs dtype support (549, 550)
* Dismiss error messages via keyboard: allows pressing the escape key to close all currently open error messages (437)
* ROI doesn’t have any effect if in pick mode, so we hide the dropdown in that case (511)
* Make tileshape parameter of HDF5 DataSet optional (578)
* Open browser after starting the server. Enabled by default, can be disabled using –no-browser (81, 580)
* Implement libertem.udf.masks.ApplyMasksUDF as a replacement of ApplyMasksJob (549, 550)
* Implement libertem.udf.raw.PickUDF as a replacement of PickFrameJob (549, 550)

Bug fixes
* Fix FRMS6 in a distributed setting. We now make sure to only do I/O in methods that are running on worker nodes (531).
* Fixed loading of nD HDF5 files. Previously the HDF5 DataSet was hardcoded for 4D data - now, arbitraty dimensions should be supported (574, 567)
* Fix DaskJobExecutor.run_each_host. Need to pass pure=False to ensure multiple runs of the function (528).

Obsolescence
* Because HDFS support is right now not tested (and to my knowledge also not used) and the upstream hdfs3 project is not actively maintained, remove support for HDFS. ClusterDataSet or CachedDataSet should be used instead (38, 534).

Misc
* Depend on distributed>=2.2.0 because of an API change. (577)
* All analyses ported from Job to UDF back-end. The Job-related code remains for now for comparison purposes (549, 550)

Travis CI build log: https://travis-ci.org/LiberTEM/LiberTEM/builds/650034945

Travis CI build log: https://travis-ci.org/LiberTEM/LiberTEM/builds/650023752

Travis CI build log: https://travis-ci.org/LiberTEM/LiberTEM/builds/649867599

Changelog: https://libertem.github.io/LiberTEM/changelog.html#v0-3-0

New features

* Make OOP based composition and subclassing easier for CorrelationUDF (466)
* Introduce plain circular match pattern Circular (469)
* Distributed sharded dataset ClusterDataSet (136, 457)
* Support for caching data sets CachedDataSet from slower storage (NFS, spinning metal) on fast local storage (471)
* Clustering analysis (401, 408 by kruzaeva).
* libertem.io.dataset.dm.DMDataSet implementation based on ncempy (497)
* Adds a new map() executor primitive. Used to concurrently read the metadata for DM3/DM4 files on initialization.
* Note: no support for the web GUI yet, as the naming patterns for DM file series varies wildly. Needs changes in the file dialog.
* Speed up of up to 150x for correlation-based peak refinement in libertem.udf.blobfinder.correlation with a Numba-based pipeline (468)
* Introduce FullFrameCorrelationUDF which correlates a large number (several hundred) of small peaks (10x10) on small frames (256x256) faster than FastCorrelationUDF and SparseCorrelationUDF (468)
* Introduce UDFPreprocessMixin (464)
* Implement iterator over AnalysisResultSet (496)
* Add hologram simulation libertem.utils.generate.hologram_frame() (475)
* Implement Hologram reconstruction UDF libertem.udf.holography.HoloReconstructUDF (475)

Bug fixes
* Improved error and validation handling when opening files with GUI (433, 442)
* Clean-up and improvements of libertem.analysis.fullmatch.FullMatcher (463)
* Ensure that RAW dataset sizes are calculated as int64 to avoid integer overflows (495, 493)
* Resolve shape mismatch issue and simplify dominant order calculation in Radial Fourier Analysis (502)
* Actually pass the enable_direct parameter from web API to the DataSet

Documentation
* Created Authorship policy (460, 483)
* Change management process (443, 451, 453)
* Documentation for Crystallinity map and Clustering analysis (408 by kruzaeva)
* Instructions for profiling slow tests (447, 448)
* Improve API reference on Analysis results (494, 496)
* Restructure and update the API reference for a number of UDFs and other application-specific code (503, 507, 508)

Obsolescence
* The Job interface is planned to be replaced with an implementation based on UDFs in one of the upcoming releases.

Misc
* Split up the bobfinder code between several files to reduce file size (468)

Travis CI build log: https://travis-ci.org/LiberTEM/LiberTEM/builds/624366945

Travis CI build log: https://travis-ci.org/LiberTEM/LiberTEM/builds/624360722

Travis CI build log: https://travis-ci.org/LiberTEM/LiberTEM/builds/624233647