Readfish

What's new
The highlights:
- Updated to use MinKNOW >= 6.0 🥳 There were some `minknow_api` endpoint changes, around the basecaller version. Big one is the switch from using read_id to read_number, and new strand classifications.
> [!WARNING]
Breaking for any version of `MinKNOW <= 6.0.0`
As of `readfish >=2024.3.0` we no longer support guppy.

- Uses `py_basecall_client.pass_reads` to send reads to the base caller for improved performance.
- Bug fixes, most are for the flongle, due to the final column of pores being 6 channels not 10, like the preceding columns.
- Chores - updated to use Node 20 actions which was surprisingly easy over all

What's Changed
* Feature/minknow-6-compatibilyt by Adoni5 in https://github.com/LooseLab/readfish/pull/368
* Feature/pass reads by Adoni5 in https://github.com/LooseLab/readfish/pull/361
* Delete dc918797-33da-4414-9999-eb9132d22971_readfish.tsv by Adoni5 in https://github.com/LooseLab/readfish/pull/380
* Catch some "placeholder" channel positions being 0 on flongle by Adoni5 in https://github.com/LooseLab/readfish/pull/384
* Fix issue340 - live TOML handling by Adoni5 in https://github.com/LooseLab/readfish/pull/383
* bump all node 16 actions to node 20 by Adoni5 in https://github.com/LooseLab/readfish/pull/377
* Minor change to readme to better reflect update to 6.0 and deprecatio… by mattloose in https://github.com/LooseLab/readfish/pull/

**Full Changelog**: https://github.com/LooseLab/readfish/compare/2024.2.0...v2024.3.0

🙏🏼 for Rory's thesis

What's new
Dorado 7.3.9 support
As discussed in 347, due to the move to `ont_pybasecall_client_lib` by ONT with the release of Dorado 7.3.9, readfish was unable to connect to the basecall socket, as it was using `ont_pyguppy_client_lib`.

This release supports both connections to `dorado` 7.3.9 and upwards with the `dorado` plugin, (which also handles the new parameter `sample_rate` on `package_reads` smartly) and `dorado` < 7.3.9 with the `guppy plugin`.

To use the new plugin, simply switch the `caller_settings` table in your toml from `.guppy` to `.dorado`, leaving other parameters the same, like so:
toml
[caller_settings.dorado]


MinKNOW version checking
Readfish also now lists a tested MinKNOW version range, and will raise a warning if the MinKNOW version being connected to is outside the range for the given version of readfish.
However, readfish will still attempt to run adaptive sampling, and will most likely succeed!
As recommended in the logged warning, if a newer version of readfish doesn't exist, we recommend opening an issue on the github.

> [!WARNING]
> The Guppy plugin will likely be deprecated in the next major release of `readfish`.

What's Changed
* Dorado plugin by mattloose and Adoni5 in https://github.com/LooseLab/readfish/pull/344
* Minknow version checking by mattloose and Adoni5 https://github.com/LooseLab/readfish/pull/351

**Full Changelog**: https://github.com/LooseLab/readfish/compare/2024.1.0...2024.2.0

What's new
* Implements some basic **Experimental** Duplex based adaptive sampling.
Currently there are two modes, "Duplex simple" and "duplex", which are accessed under the `--chemistry` flag under the `targets` sub command. They both take targets in the same way that the normal target commands do. We currently do not support mixing simplex and duplex targets.
`duplex_simple` accepts a read if the previous read seen on the channel was stop receiving and the new read isn't, `duplex` checks that the previous reads alignment was on the same contig and opposite strand.
**NB.** Only `duplex_simple` has been tested.
* [This pull request](https://github.com/LooseLab/readfish/pull/324) refactors the code that the check for reads to be over-ridden, and also adds some new reasons for read decisions:
python
: Potential second half of a duplex read
duplex_override = "duplex_override"
: Read sequenced as translocated portion was of unknown length at start of readfish
first_read_override = "first_read_override"


Bug fixes
* Don't throw an error if there are multiple suffixes on the provided `.mmi` file
* `--wait-on-ready` actually works, and is an `int`, so User can specify to wait for the Sequencing state for as long as they want (default now 120s)
* Fixes links in documentation to bulk files and adds a promethion 4KhZ R10 bulk file
* `--unblock-duration` type to float, and manually specifying it was crashing argparsing


What's Changed - Associated PRs
* Issue309 by mattloose in https://github.com/LooseLab/readfish/pull/310 - adds documentation and promethion 4KhZ R10 bulk file
* Closes issue 312 by mattloose in https://github.com/LooseLab/readfish/pull/313 - unblock duration type to float
* Fix changing directory after 60 seconds if no run folder by Adoni5 in https://github.com/LooseLab/readfish/pull/323
* Change type on --wait-for-ready to int by Adoni5 in https://github.com/LooseLab/readfish/pull/327
* Adding additional .mmi extensions to all possible fasta file format l… by mattloose in https://github.com/LooseLab/readfish/pull/330
* Feature/duplex by Adoni5 in https://github.com/LooseLab/readfish/pull/324


**Full Changelog**: https://github.com/LooseLab/readfish/compare/2023.1.1...2024.1.0

A small release, with a bug fix.

For major new features introduced in refactored readfish, please see https://github.com/LooseLab/readfish/releases/tag/2023.1.0.

Bug fixes
* Fixes a bug caught by jamesemery in 289, where regions were required to be present in a TOML file, even when the intent was to use barcoding only.

Other changes
* Adds a [Changelog](https://looselab.github.io/readfish/changelog.html), found in the docs and at the bottom of the README.
* Minor documentation fixes to broken links and problems installing `grpcio` on Apple Silicon.

Major Changes

1. This release is a complete overhaul of the inner workings of readfish. Fundamentally nothing has changed in the way readfish works, but we have dramatically improved the compatibility with nanopore platforms, as well as supporting more compute systems including Apple Silicon.

2. This release is compatible with all flavours of PromethION from the lonely p2solo to the gregarious P48. To do this we include a new rust based wrapper for minimap2 which enables multi-threaded mapping. The release also supports GridION and MinION. It is untested on mk1c and we don't recommend using it.

3. This release is compatible with both guppy and dorado via the `ont-pyguppy-client-lib` and so introduces native support for apple silicon.

4. We also introduce a new readfish stats module which will summarise an experiment given the TOML configuration for readfish and the path to the FASTQs from the run.

5. The code includes all new documentation - see https://looselab.github.io/readfish - and will prevent users from making many common mistakes.

> [!NOTE]
> Important: The TOML file specification has been modified for this version and users should update their TOML files accordingly.

This includes a description of the new plugin framework, developed by alexomics , which will enable users to develop their own custom mappers, squiggle analysis packages and other tools and then integrate them into a common readfish environment.

In principle readfish can be used as a library by other packages.

Full information on new features can be found in the readme and associated documentation. All issues should be reported by the GitHub issue tracker and not by email to the authors.

What's Changed
* Add unblock all script by alexomics
* Add github worklflow action to close stale issues by Adoni5
* Move logger initialisation into check that we have arguments by Adoni5
* Add pre commit and make all files compliant by Adoni5
* validation script by alexomics
* Switching behaviour of validate by mattloose
* Add readfish identifier to appear in guppy server logs by mattloose
* Feature/describe plugin by Adoni5
* Remove --chunk-log option by alexomics
* Implement runsafe and additional logging. by mattloose
* Implementing check on basecalling model in guppy plugin. This will va… by mattloose
* Fixes to describe by Adoni5
* Update _config.py to check barcodes if regions and barcodes are present. by Adoni5
* Enabling changes to the log for the readuntil client by tools which u… by mattloose
* Add `readfish[mappy,guppy]` as default install target by Adoni5
* Split mappy plugins by alexomics
* Feature/summarise-entry-point by Adoni5
* Fix-stats-macos-aarch-test by Adoni5
* Adding a check for the current break_reads_seconds value and testing … by mattloose
* Feature/logging-stats-padding by Adoni5

No take backsies (for real)

This version of ReadFish contains the updated BossRuns code in response to reviewer comments. ReadFish BossRuns should now be compatible with `minknow-core` <=5.1.4 and >= 5.0.0 and `Guppy` >= 6.0.0. The command to run readfish bossruns remains the same, however the guppy address and port are now most likely `ipc:///tmp/guppy` and `5555`.