Chia-blockchain

Added

- The GUI now displays sub blocks as well as transaction blocks on the Full Node page.
- `chia plots check` enforces a minimum of `-n 5` to decrease false negatives. Thanks to eFishCent for these ongoing pull requests!
- Testnets and mainnets will now have an initial period of sub blocks where transactions are blocked.
- Transaction volume testing added to tests and various tests have been sped up.
- We have added connection limits for max_inbound_wallet, max_inbound_farmer, and max_inbound_timelord.

Changed

- On starting full node, the weight proof cache does not attempt to load all sub blocks. Startup times are noticeably improved though there remains a hesitation when validating the mempool. Our clvm Rust implementation, which will likely ship in the next release, will drop example processing times from 180 to 3 seconds.
- Changes to weight proofs and sub block storage and cacheing required a new database schema. This will require a re-sync or obtaining a synced blockchain_v23.db.
- clvm bytecode is now generated and confirmed that the checked-in clvm and ChiaLisp code matches the CI compiled code.
- We have removed the '-r' flag from `chia` as it was being overridden in most cases by the `-r` for restart flag to `chia start`. Use `chia --root-path` instead.
- `chia -h` now recommends `chia netspace -d 192` which is approximately one hours worth of sub blocks. Use `-d 1000` to get the same estimate of netspace as the RPC and GUI.
- `chia show -c` now displays in MiB and the GUI has been changed to MiB to match.
- `chia configure` now accepts the shorter `-upnp` and `-log-level` arguments also.
- `chia plots check` now defaults to `-n 30` instead of `-n 1` - HT eFishCent.
- `chia plots create` now enforces a minimum of k=22. As a reminder, anything less than k=32 is just for testing and be careful extrapolating performance of a k less than 30 to a k=32 or larger.
- We have updated development dependencies for setuptools, yarl, idna, multidict, and chardet.
- Updated some copyright dates to 2021.

Fixed

- We upgraded our fork of aiosqlite to version 16.0 which has significant performance improvements. Our fixes to aiosqlite are waiting to be upstreamed.
- The Plots tab in the GUI will no longer show red/error when the node is still syncing.
- Inbound and outbound peer connection limits were not being honored.
- Weight proofs were not correctly extending.
- In some cases when closing a p2p connection to another node, there was an infinite "Closing" loop.
- `chia show -c` was showing upload MiB in the download column and vice versa. pyl and psydafke deserves credit for insisting it was broken and kd637xx for the PR assist.
- `chia show` handles sub block 0 better.

Added

- Node now attempts to pre-validate and cache transactions.
- The harvester will try to not load a plot file that is too small for its k size. This should help keep from partial plots being found when they are copied into a harvester directory. Harvester will check again on the next challenge and find a completed copy of a plot file then.
- `chia plots create -x` skips adding [final dir] to harvester for farming

Changed

- We now use bech32m and have added the bech32m tests from Pieter Wuille (sipa) outlined [here](https://gist.github.com/sipa/14c248c288c3880a3b191f978a34508e) with thanks.
- In the GUI, choosing to parallel plot with a delay now is a delay between the start of the parallel plots started in one session.
- Removed loading plot file names when starting `chia plots create`; decreases plotter time when there are a lot of plots on the machine. Huge thanks to eFishCent for this PR!

Fixed

- Various fixes to improve node's ability to sync. There are still plenty of additional performance improvements coming for node so expect it to get easier to run on less powerful devices.
- Wallet now handles large amounts of coins much better and generally syncs better.
- Thanks to nup002 for the PR to use scientific notation in the logs for address_manager.select_peer timings.
- `chia show -h` now correctly states that you use the first 8 characters of the node id to remove a node on the cli.
- Thank you to wallentx for adding better help for `chia configure --enable-upnp`.
- Pull requests from forks won't have failures on CI.

Added

- The cli now warns if you attempt to create a plot smaller than k=32.
- `chia configure` now lets you enable or disable uPnP.
- If a peer gives a bad weight proof it will now be disconnected.

Changed

- Harvester now only checks every 2 minutes for new files and otherwise caches the plot listing in memory and logs how long it took to load all plot files at INFO level.
- Harvester multithreading is now configureable in config.yaml.
- Websocket heartbeat timeout was increased from 30 seconds to 300 seconds.
- Bumped Colorlog to 4.7.2, and pyinstaller to 4.2.

Fixed

- Weight proofs were failing to verify contributing to a chain stall. This release gets things moving again but nodes are using too much CPU and can pause/lag at times. This may resolve as people upgrade to Beta 21.
- A toxic combination of transaction limits set too high and a non performant clvm kept the chain stalled. A faster rust implementation of clvm is already nearing completion.
- `chia netspace -s` would not correctly look up the start block height by block hash. Additionally netspace now flips to PiB above 1024 TiB. To compare netspace to `chia show` of the GUI use `chia netspace -d 1000` as `chia netspace` defaults to `-d 192` which is one hour.

Added

- Plotting now checks to see if there are MacOS created `._` plot files and ignores them.
- Mnemonics now autocomplete in the GUI.

Changed

- Node sync is now multithreaded and much quicker.
- Peer gossip is faster and smarter. It also will no longer accidentally gossip a private IP address to another peer.
- When syncing in the GUI, estimated time to win just shows syncing until synced.
- If harvester hits an exception it will be caught, logged and skipped. This normally happens if it attempts to harvest a plot file you are still copying in.
- The Rate Limited wallet has been updated to work in new consensus.

Fixed

- There was a bug in block reorg code that would keep a peer with a lower weight chain from validating and syncing to a higher weight chain when the node thought it had a double spend in the other chain. This caused a persistent chain split.
- The Farm page in the GUI should not report just error when initially starting to sync.

Added

- Welcome to the new consensus. This release is an all but a full re-write of the blockchain in under 30 days. There is now only one tip of the blockchain but we went from two chains to three. Block times are now a little under a minute but there are a couple of sub blocks between each transaction block. A block is also itself a special kind of sub block and each sub block rewards the farmer who won it 1 TXCH. Sub blocks come, on average, about every 17 to 18 seconds.
- Starting with this Beta, there are 4608 opportunities per day for a farmer to win 1 TXCH compared to Beta 18 where there were 288 opportunities per day for a farmer to win 16 TXCH.
- There is a lot more information and explanation of the new consensus algorithm in the New Consensus Working Document linked from [chia.net](https://chia.net/). Among the improvements this gives the Chia blockchain are a much higher security level against all attacks, more frequent transaction blocks that have less time variation between them and are then buried under confirmations (sub blocks also count towards re-org security) much more quickly.
- New consensus means this is a very hard fork. All of your TXCH from Beta 17/18 will be gone. Your plots and keys will work just fine however. You will have to sync to the new chain.
- You now have to sync 16 times more "blocks" for every 5 minutes of historical time so syncing is slower than it was on the old chain. We're aware of this and will be speeding it up and addressing blockchain database growth in the nest couple of releases.
- Prior to this Beta 19, we had block times that targeted 5 minutes and rewarded 16 TXCH to one farmer. Moving forward we have epoch times that target 10 minutes and reward 32 TXCH to 32 farmers about every 17-18 seconds over that period. This has subtle naming and UI impacts in various places.
- Total transaction throughput is still targeted at 2.1x Bitcoin's throughput per hour but you will get more confirmations on a transaction much faster. This release has the errata that it doesn't limit transaction block size correctly.
- For testing purposes this chain is quickly halving block rewards. By the time you're reading this and using the chain, farmers and pools will be receiving less than 1 TXCH for each block won as if it were 15-20 years from now. Block rewards are given in two components, 7/8's to the pool key and 1/8 to the farmer. The farmer also receives any transaction fees from the block.
- You can now plot in parallel using the GUI. A known limitation is that you can't yet specify that you want 4 sets of two parallel plots. Each parallel plot added starts immediately parallel. We will continue to improve this.
- The GUI now warns if you attempt to create a plot smaller than k=32.
- Added Chinese language localization (zh-cn). A big thank you to goomario for their pull request!
- You can now specify which private key to use for `chia plots create`. After obtaining the fingerprint from `chia keys show`, try `chia plots create -a FINGERPRINT`. Thanks to eFishCent for this pull request!
- We use a faster hash to prime function for chiavdf from the current release of gmp-6.2.1 which we have upgraded chiavdf and blspy to support.
- There is a new cli command - `chia configure`. This allows you to update certain configuration details like log level in config.yaml from the command line. This is particularly useful in containerization and linux automation. Try `chia configure -h`. Note that if chia services are running and you issue this command you will have to restart them for changes to take effect but you can use this command in the venv when no services are running or call it directly by path in the venv without activating the venv. Expect the options for this command to expand.
- We now fully support Python 3.9.

Changed

- The Plot tab on the GUI is now the Plots tab. It starts out with a much more friendly new user wizard and otherwise keeps all of your farming plots listed here. Use the "+ ADD A PLOT" button in the top right to plot your second or later plot.
- The new plots page offers advanced plotting options in the various "Show Advanced Options" fold outs.
- The plotter supports the new bitfield back propagation method and the old method from Beta 17. To choose the old method add a `-e` to the command line or choose "Disable bitfield plotting" in "Show Advanced Options" of the Plots tab. Bitfield back propagation writes about 13% less total writes and can be faster on some slower hard drive temp spaces. For now, SSD temp space will likely plot faster with bitfield back propagation disabled. We will be returning to speed enhancements to the plotter as we approach and pass our mainnet launch.
- The Farm tab in the GUI is significantly enhanced. Here you have a dashboard overview of your farm and your activity in response to challenges blockchain challnegs, how long it will take you - on average - to win a block, and how much TXCH you've won so far. Harvester and Full Node connections have moved to Advanced Options.
- Harvester and farmer will start when the GUI starts instead of waiting for key selection if there are already keys available. This means you will start farming on reboot if you have the Chia application set to launch on start.
- Testnet is now running at the primary port of 58444. Update your routers appropriately. This opens 8444 for mainnet.
- All networking code has been refactored and mostly moved to websockets.
- RPCs and daemon now communicate over TLS with certificates that are generated into `~/.chia/VERSION/config/`
- We have moved to taproot across all of our transactions and smart transactions.
- We have adopted chech32m encoding of keys and addresses in parallel to bitcoin's coming adoption of bech32m.
- The rate limited wallet was updated and re-factored.
- All appropriate Chialisp smart transactions have been updated to use aggsig_me.
- Full node should be more aggressive about finding other peers.
- Peer disconnect messages are now set to log level INFO down from WARNING.
- chiavdf now allows passing in input to a VDF for new consensus.
- sha256tree has been removed from Chialisp.
- `chia show -s` has been refactored to support the new consensus.
- `chia netspace` has been refactored for new consensus.
- aiohttp, clvm-tools, colorlog, concurrent-log-handler, keyring, cryptography, and sortedcontainers have been upgraded to their current versions.
- Tests now place a cache of blocks and plots in the ~/.chia/ directory to speed up total testing time.
- Changes were made to chiapos to correctly support the new bitfiled backpropogation on FreeBSD and OpenBSD. With the exception of needing to work around python cryptography as outlined on the wiki, FreeBSD and OpenBSD should be able to compile and run chia-blockchain.
- With the change to new consensus many components of the chain and local database are not yet stored optimally. Startup and sync times may be slower than usual so please be patient. This will improve next release.
- Errata: Coinbase amount is missing from the GUI Block view.
- Eratta: wallet Backup, and Fly-sync on the wallet are currently not working.

Fixed

- There was a regression in Beta 18 where the plotter took 499GiB of temp space for a k32 when it used to only use 332GiB. The plotter should now use just slightly less than it did in Beta 17.
- blspy was bumped to 0.3.1 which now correctly supports the aggsig of no signatures and is built with gmp-6.2.1.
- Fixed a plotter crash after pulling a disk without ejecting it first.
- `sh install.sh` now works properly on Linux Mint.
- `chia show -s` now is less brain dead when a node is initially starting to sync.

Added

- F1 generation in the plotter is now fully parallel for a small speedup.
- We have bitfield optimized phase 2 of plotting. There is only about a 1% increase in speed from this change but there is a 12% decrease in writes with a penalty of 3% more reads. More details in [PR 120](https://github.com/Chia-Network/chiapos/pull/120). Note that some sorts in phase 2 and phase 3 will now appear "out of order" and that is now expected behavior.
- Partial support for Python 3.9. That includes new versions of Chia dependencies like chiabip158.

Changed

- We have moved from using gulrak/filesystem across all platforms to only using it on MacOS. It's required on MacOS as we are still targeting Mojave compatibility. This should resolve Windows path issues.
- We upgraded to cbor 5.2.0 but expect to deprecate cbor in a future release.

Fixed

- A segfault caused by memory leaks in bls-library has been fixed. This should end the random farmer and harvester crashes over time as outlined in [Issue 500](https://github.com/Chia-Network/chia-blockchain/issues/500).
- Plotting could hang up retrying in an "error 0" state due to a bug in table handling in some edge cases.
- CPU utilization as reported in the plotter is now accurate for Windows.
- FreeBSD and OpenBSD should be able to build and install chia-blockchain and its dependencies again.
- Starting with recent setuptools fixes, we can no longer pass an empty string to the linker on Windows when building binary wheels in the sub repos. Thanks jaraco for tracking this down.