Chik-blockchain

Added

- The GUI now displays sub blocks as well as transaction blocks on the Full Node page.
- `chik plots check` enforces a minimum of `-n 5` to decrease false negatives. Thanks to eFishCent for these ongoing pull requests!
- Testnets and mainnets will now have an initial period of sub blocks where transactions are blocked.
- Transaction volume testing added to tests and various tests have been sped up.
- We have added connection limits for max_inbound_wallet, max_inbound_farmer, and max_inbound_timelord.

Changed

- On starting full node, the weight proof cache does not attempt to load all sub blocks. Startup times are noticeably improved though there remains a hesitation when validating the mempool. Our klvm Rust implementation, which will likely ship in the next release, will drop example processing times from 180 to 3 seconds.
- Changes to weight proofs and sub block storage and cacheing required a new database schema. This will require a re-sync or obtaining a synced blockchain_v23.db.
- klvm bytecode is now generated and confirmed that the checked-in klvm and ChikLisp code matches the CI compiled code.
- We have removed the '-r' flag from `chik` as it was being overridden in most cases by the `-r` for restart flag to `chik start`. Use `chik --root-path` instead.
- `chik -h` now recommends `chik netspace -d 192` which is approximately one hours worth of sub blocks. Use `-d 1000` to get the same estimate of netspace as the RPC and GUI.
- `chik show -c` now displays in MiB and the GUI has been changed to MiB to match.
- `chik configure` now accepts the shorter `-upnp` and `-log-level` arguments also.
- `chik plots check` now defaults to `-n 30` instead of `-n 1` - HT eFishCent.
- `chik plots create` now enforces a minimum of k=22. As a reminder, anything less than k=32 is just for testing and be careful extrapolating performance of a k less than 30 to a k=32 or larger.
- We have updated development dependencies for setuptools, yarl, idna, multidict, and chardet.
- Updated some copyright dates to 2021.

Fixed

- We upgraded our fork of aiosqlite to version 16.0 which has significant performance improvements. Our fixes to aiosqlite are waiting to be upstreamed.
- The Plots tab in the GUI will no longer show red/error when the node is still syncing.
- Inbound and outbound peer connection limits were not being honored.
- Weight proofs were not correctly extending.
- In some cases when closing a p2p connection to another node, there was an infinite "Closing" loop.
- `chik show -c` was showing upload MiB in the download column and vice versa. pyl and psydafke deserves credit for insisting it was broken and kd637xx for the PR assist.
- `chik show` handles sub block 0 better.

Added

- Node now attempts to pre-validate and cache transactions.
- The harvester will try to not load a plot file that is too small for its k size. This should help keep from partial plots being found when they are copied into a harvester directory. Harvester will check again on the next challenge and find a completed copy of a plot file then.
- `chik plots create -x` skips adding [final dir] to harvester for farming

Changed

- We now use bech32m and have added the bech32m tests from Pieter Wuille (sipa) outlined [here](https://gist.github.com/sipa/14c248c288c3880a3b191f978a34508e) with thanks.
- In the GUI, choosing to parallel plot with a delay now is a delay between the start of the parallel plots started in one session.
- Removed loading plot file names when starting `chik plots create`; decreases plotter time when there are a lot of plots on the machine. Huge thanks to eFishCent for this PR!

Fixed

- Various fixes to improve node's ability to sync. There are still plenty of additional performance improvements coming for node so expect it to get easier to run on less powerful devices.
- Wallet now handles large amounts of coins much better and generally syncs better.
- Thanks to nup002 for the PR to use scientific notation in the logs for address_manager.select_peer timings.
- `chik show -h` now correctly states that you use the first 8 characters of the node id to remove a node on the cli.
- Thank you to wallentx for adding better help for `chik configure --enable-upnp`.
- Pull requests from forks won't have failures on CI.

Added

- The cli now warns if you attempt to create a plot smaller than k=32.
- `chik configure` now lets you enable or disable uPnP.
- If a peer gives a bad weight proof it will now be disconnected.

Changed

- Harvester now only checks every 2 minutes for new files and otherwise caches the plot listing in memory and logs how long it took to load all plot files at INFO level.
- Harvester multithreading is now configureable in config.yaml.
- Websocket heartbeat timeout was increased from 30 seconds to 300 seconds.
- Bumped Colorlog to 4.7.2, and pyinstaller to 4.2.

Fixed

- Weight proofs were failing to verify contributing to a chain stall. This release gets things moving again but nodes are using too much CPU and can pause/lag at times. This may resolve as people upgrade to Beta 21.
- A toxic combination of transaction limits set too high and a non performant klvm kept the chain stalled. A faster rust implementation of klvm is already nearing completion.
- `chik netspace -s` would not correctly look up the start block height by block hash. Additionally netspace now flips to PiB above 1024 TiB. To compare netspace to `chik show` of the GUI use `chik netspace -d 1000` as `chik netspace` defaults to `-d 192` which is one hour.

Added

- Plotting now checks to see if there are MacOS created `._` plot files and ignores them.
- Mnemonics now autocomplete in the GUI.

Changed

- Node sync is now multithreaded and much quicker.
- Peer gossip is faster and smarter. It also will no longer accidentally gossip a private IP address to another peer.
- When syncing in the GUI, estimated time to win just shows syncing until synced.
- If harvester hits an exception it will be caught, logged and skipped. This normally happens if it attempts to harvest a plot file you are still copying in.
- The Rate Limited wallet has been updated to work in new consensus.

Fixed

- There was a bug in block reorg code that would keep a peer with a lower weight chain from validating and syncing to a higher weight chain when the node thought it had a double spend in the other chain. This caused a persistent chain split.
- The Farm page in the GUI should not report just error when initially starting to sync.

Added

- Welcome to the new consensus. This release is an all but a full re-write of the blockchain in under 30 days. There is now only one tip of the blockchain but we went from two chains to three. Block times are now a little under a minute but there are a couple of sub blocks between each transaction block. A block is also itself a special kind of sub block and each sub block rewards the farmer who won it 1 TXCK. Sub blocks come, on average, about every 17 to 18 seconds.
- Starting with this Beta, there are 4608 opportunities per day for a farmer to win 1 TXCK compared to Beta 18 where there were 288 opportunities per day for a farmer to win 16 TXCK.
- There is a lot more information and explanation of the new consensus algorithm in the New Consensus Working Document linked from [chiknetwork.com](https://chiknetwork.com/). Among the improvements this gives the Chik blockchain are a much higher security level against all attacks, more frequent transaction blocks that have less time variation between them and are then buried under confirmations (sub blocks also count towards re-org security) much more quickly.
- New consensus means this is a very hard fork. All of your TXCK from Beta 17/18 will be gone. Your plots and keys will work just fine however. You will have to sync to the new chain.
- You now have to sync 16 times more "blocks" for every 5 minutes of historical time so syncing is slower than it was on the old chain. We're aware of this and will be speeding it up and addressing blockchain database growth in the nest couple of releases.
- Prior to this Beta 19, we had block times that targeted 5 minutes and rewarded 16 TXCK to one farmer. Moving forward we have epoch times that target 10 minutes and reward 32 TXCK to 32 farmers about every 17-18 seconds over that period. This has subtle naming and UI impacts in various places.
- Total transaction throughput is still targeted at 2.1x Bitcoin's throughput per hour but you will get more confirmations on a transaction much faster. This release has the errata that it doesn't limit transaction block size correctly.
- For testing purposes this chain is quickly halving block rewards. By the time you're reading this and using the chain, farmers and pools will be receiving less than 1 TXCK for each block won as if it were 15-20 years from now. Block rewards are given in two components, 7/8's to the pool key and 1/8 to the farmer. The farmer also receives any transaction fees from the block.
- You can now plot in parallel using the GUI. A known limitation is that you can't yet specify that you want 4 sets of two parallel plots. Each parallel plot added starts immediately parallel. We will continue to improve this.
- The GUI now warns if you attempt to create a plot smaller than k=32.
- Added Chinese language localization (zh-cn). A big thank you to goomario for their pull request!
- You can now specify which private key to use for `chik plots create`. After obtaining the fingerprint from `chik keys show`, try `chik plots create -a FINGERPRINT`. Thanks to eFishCent for this pull request!
- We use a faster hash to prime function for chikvdf from the current release of gmp-6.2.1 which we have upgraded chikvdf and blspy to support.
- There is a new cli command - `chik configure`. This allows you to update certain configuration details like log level in config.yaml from the command line. This is particularly useful in containerization and linux automation. Try `chik configure -h`. Note that if chik services are running and you issue this command you will have to restart them for changes to take effect but you can use this command in the venv when no services are running or call it directly by path in the venv without activating the venv. Expect the options for this command to expand.
- We now fully support Python 3.9.

Changed

- The Plot tab on the GUI is now the Plots tab. It starts out with a much more friendly new user wizard and otherwise keeps all of your farming plots listed here. Use the "+ ADD A PLOT" button in the top right to plot your second or later plot.
- The new plots page offers advanced plotting options in the various "Show Advanced Options" fold outs.
- The plotter supports the new bitfield back propagation method and the old method from Beta 17. To choose the old method add a `-e` to the command line or choose "Disable bitfield plotting" in "Show Advanced Options" of the Plots tab. Bitfield back propagation writes about 13% less total writes and can be faster on some slower hard drive temp spaces. For now, SSD temp space will likely plot faster with bitfield back propagation disabled. We will be returning to speed enhancements to the plotter as we approach and pass our mainnet launch.
- The Farm tab in the GUI is significantly enhanced. Here you have a dashboard overview of your farm and your activity in response to challenges blockchain challnegs, how long it will take you - on average - to win a block, and how much TXCK you've won so far. Harvester and Full Node connections have moved to Advanced Options.
- Harvester and farmer will start when the GUI starts instead of waiting for key selection if there are already keys available. This means you will start farming on reboot if you have the Chik application set to launch on start.
- Testnet is now running at the primary port of 59678. Update your routers appropriately. This opens 9678 for mainnet.
- All networking code has been refactored and mostly moved to websockets.
- RPCs and daemon now communicate over TLS with certificates that are generated into `~/.chik/VERSION/config/`
- We have moved to taproot across all of our transactions and smart transactions.
- We have adopted chech32m encoding of keys and addresses in parallel to bitcoin's coming adoption of bech32m.
- The rate limited wallet was updated and re-factored.
- All appropriate Chiklisp smart transactions have been updated to use aggsig_me.
- Full node should be more aggressive about finding other peers.
- Peer disconnect messages are now set to log level INFO down from WARNING.
- chikvdf now allows passing in input to a VDF for new consensus.
- sha256tree has been removed from Chiklisp.
- `chik show -s` has been refactored to support the new consensus.
- `chik netspace` has been refactored for new consensus.
- aiohttp, klvm-tools, colorlog, concurrent-log-handler, keyring, cryptography, and sortedcontainers have been upgraded to their current versions.
- Tests now place a cache of blocks and plots in the ~/.chik/ directory to speed up total testing time.
- Changes were made to chikpos to correctly support the new bitfiled backpropogation on FreeBSD and OpenBSD. With the exception of needing to work around python cryptography as outlined on the wiki, FreeBSD and OpenBSD should be able to compile and run chik-blockchain.
- With the change to new consensus many components of the chain and local database are not yet stored optimally. Startup and sync times may be slower than usual so please be patient. This will improve next release.
- Errata: Coinbase amount is missing from the GUI Block view.
- Eratta: wallet Backup, and Fly-sync on the wallet are currently not working.

Fixed

- There was a regression in Beta 18 where the plotter took 499GiB of temp space for a k32 when it used to only use 332GiB. The plotter should now use just slightly less than it did in Beta 17.
- blspy was bumped to 0.3.1 which now correctly supports the aggsig of no signatures and is built with gmp-6.2.1.
- Fixed a plotter crash after pulling a disk without ejecting it first.
- `sh install.sh` now works properly on Linux Mint.
- `chik show -s` now is less brain dead when a node is initially starting to sync.

Added

- F1 generation in the plotter is now fully parallel for a small speedup.
- We have bitfield optimized phase 2 of plotting. There is only about a 1% increase in speed from this change but there is a 12% decrease in writes with a penalty of 3% more reads. More details in [PR 120](https://github.com/Chik-Network/chikpos/pull/120). Note that some sorts in phase 2 and phase 3 will now appear "out of order" and that is now expected behavior.
- Partial support for Python 3.9. That includes new versions of Chik dependencies like chikbip158.

Changed

- We have moved from using gulrak/filesystem across all platforms to only using it on MacOS. It's required on MacOS as we are still targeting Mojave compatibility. This should resolve Windows path issues.
- We upgraded to cbor 5.2.0 but expect to deprecate cbor in a future release.

Fixed

- A segfault caused by memory leaks in bls-library has been fixed. This should end the random farmer and harvester crashes over time as outlined in [Issue 500](https://github.com/Chik-Network/chik-blockchain/issues/500).
- Plotting could hang up retrying in an "error 0" state due to a bug in table handling in some edge cases.
- CPU utilization as reported in the plotter is now accurate for Windows.
- FreeBSD and OpenBSD should be able to build and install chik-blockchain and its dependencies again.
- Starting with recent setuptools fixes, we can no longer pass an empty string to the linker on Windows when building binary wheels in the sub repos. Thanks jaraco for tracking this down.