Chik-blockchain

Added

- The Beta 27 chain is a hard fork. All TXCK from previous releases has been reset on this chain. Your keys and plots of k=32 or larger continue to work just fine on this new chain.
- We now use the rust version of klvm, klvm_rs, in preference to validate transactions. We have additionally published binary wheels or klvm_rs for all four platforms and all three supported python versions. The rust version is approximately 50 times faster than the python version used to validate on chain transactions in previous versions.
- We have moved to compressed quadratic forms for VDFs. Using compressed representation of quadratic forms reduces their serialized size from 130 to 100 bytes (for forms with 1024-bit discriminant). This shrinks the size of VDF outputs and VDF proofs, and it's a breaking change as the compressed representation is not compatible with the older uncompressed (a, b) representation. Compressed forms are also used in calls to chikvdf and in timelord's communication with VDF clients. The form compression algorithm is based on ["Trustless Groups of Unknown Order with Hyperelliptic Curves"](https://eprint.iacr.org/2020/196) by Samuel Dobson, Steven D. Galbraith and Benjamin Smith.
- Last Attempted Proof on the Farm tab of the GUI now shows hours:minutes:seconds instead of just hours:minutes. This makes it much easier to see that your farmer is responding to recent challenges at a glance.
- You can now send and receive transactions with the command line. Try `chik wallet -h` to learn more. Also, `chik wallet` now requires a third argument of `show`, therefor you will use `chik wallet show` to see your wallet balance.
- We have added the [Crowdin](https://crowdin.com/) translation platform to [chik blockchain gui](https://crowdin.com/project/chik-blockchain). We are still getting it fully set up, but helping to translate the GUI is going to be much easier.
- Full Node > Connections in the GUI now shows the peak sub block height your connected peers believe they are at. A node syncing from you will not be at the true peak sub block height until it gets into sync.
- `chik init -c [directory]` will create new TLS certificates signed by your CA located in `[directory]`. Use this feature to configure a new remote harvester. Type `chik init -h` to get instructions. Huge thanks to a very efficient eFishCent for this quick and thorough pull request.
- We build both MacOS x86_64 and MacOS universal wheels for chikpos, chikvdf, blpsy, and chikbip158 in Python 3.9. The universal build allows M1 Macs to run these dependencies in ARM64 native mode.
- On first run in the GUI (or when there are no plot directories) there is now an "Add Plot Directories" on the Farm tab also.

Changed

- We are moving away from the terms sub blocks and blocks in our new consensus. What used to be called sub blocks will now just be blocks. Some blocks are now also transaction blocks. This is simpler both in the code and to reason about. Not all the code or UI may have caught up yet.
- This release has the final mainnet rewards schedule. During the first three years, each block winner will win 2 TXCK/XCK per block for a total of 9216 TXCK per day from 4608 challenges per day.
- Smart transactions now use an announcement instead of 'coin consumed' or lock methods.
- The GUI is now in a separate submodule repository from chik-blockchain, [chik-blockchain-gui](https://github.com/Chik-Network/chik-blockchain-gui). The installers and install scripts have been updated and it continues to follow the same install steps. Note that the GUI directory will now be `chik-blockchain-gui`. The workflow for this may be "touch and go" for people who use the git install methods over the short term.
- Very large coin counts are now supported.
- Various RPC endpoints have been renamed to follow our switch to "just blocks" from sub blocks.
- We've made changes to the protocol handshake and the blockchain genesis process to support mainnet launch and running/farming more than one chain at a time. That also means we can't as easily determine when an old version of the peer tries to connect so we will put warnings in the logs for now.
- We no longer replace addresses in the config. **IMPORTANT** - This means if you change the target address in config.yml, you have to make sure you control the correct keys.
- We now only migrate Beta 19 and newer installations.
- We have removed cbor2 as a dependency.
- We updated various dependencies including cryptography, packaging, portalocker, and pyyaml - most of which are only development dependencies.

Fixed

- The function that estimated total farming space was counting space at twice the actual rate. Netspace will display half of the previous space estimate which is now a correct estimate of the actual space currently being farmed.
- We fixed many sync and stay in sync issue for both node and wallet including that both would send peaks to other peers multiple times and would validate the same transaction multiple times.
- The GUI was incorrectly reporting the time frame that the netspace estimate it displays utilizes. It is technically 312.5 minutes, on average, over the trailing 1000 sub blocks.
- Coloured coins were not working in the new consensus.
- Some Haswell processors do not have certain AVX extensions and therefor would not run.
- The cli wallet, `chik wallet`, was incorrectly displaying TXCK balances as if they were Coloured Coins.
- We addressed [CVE-2020-28477](https://nvd.nist.gov/vuln/detail/CVE-2020-28477) in the GUI.
- We made changes to CI to hopefully not repeat our skipped releases from the previous release cycle.

Added

- We now use our own faster primality test based on Baillie-PSW. The new primality test is based on the 2020 paper ["Strengthening the Baillie-PSW primality test" by Robert Baillie, Andrew Fiori, Samuel S. Wagstaff Jr](https://arxiv.org/abs/2006.14425). The new test works approximately 20% faster than GMP library's mpz_probab_prime_p() function when generating random 1024-bit primes. This lowers the load on Timelords and speeds up VDF verifications in full node.
- The GUI now checks for an an already running GUI and stops the second launch. Thank you for that PR to dkackman !
- Transactions are now validated in a separate process in full node.
- `chik plots check -l` will list all duplicate plot IDs found on the machine. Thanks very much for this PR eFishCent.

Changed

- Significant improvements have been made to how the full node handles the mempool. This generally cuts CPU usage of node by 2x or more. Part of this increase is that we have temporarily limited the size of transactions. If you want to test sending a transaction you should keep the value of your transaction below 20 TXCK as new consensus will cause you to use a lot of inputs. This will be returned to the expected level as soon as the integration of [klvm rust](https://github.com/Chik-Network/klvm_rs) is complete.
- We have changed the way TLS between nodes and between chik services work. Each node now has two certificate authorities. One is a public, shared CA that signs the TLS certificates that every node uses to connect to other nodes on 9678 or 59678. You now also have a self generated private CA that must sign e.g. farmer and harvester's certificates. To run a remote harvester you need a new harvester key that is then signed by your private CA. We know this is not easy for remote harvester in this release but will address it quickly.
- We have changed the way we compile the proof of space plotter and added one additional optimization. On many modern processors this will mean that using the plotter with the `-e` flag will be 2-3% faster than the Beta 17 plotter on the same CPU. We have found this to be very sensitive to different CPUs but are now confident that, at worst, the Beta 24 plotter with `-e` will be the same speed as Beta 17 if not slightly faster on the same hardware. Huge thanks to xorinox for meticulously tracking down and testing this.
- If a peer is not responsive during sync, node will disconnect it.
- Peers that have not sent data in the last hour are now disconnected.
- We have made the "Help Translate" button in the GUI open in your default web browser and added instructions for adding new translations and more phrases in existing translations at that [URL](https://github.com/Chik-Network/chik-blockchain/tree/main/electron-react/src/locales). Try the "Help Translate" option on the language selection pull down to the left of the dark/light mode selection at the top right of the GUI.
- Sync store now tracks all connected peers and removes them as they get removed.
- The Rate Limited Wallet has been ported to new consensus and updated Chiklisp methods.
- We are down to only one sub dependency that does not ship binary wheels for all four platforms. The only platform still impacted is ARM64 (generally Raspberry Pi) but that only means that you still need the minor build tools as outlined on the [wiki](https://github.com/Chik-Network/chik-blockchain/wiki/Raspberry-Pi).
- We upgraded to Electron 9.4.2 for the GUI.
- We have upgraded to py-setproctitle 1.2.2. We now have binary wheels for setproctitle on all four platforms and make it a requirement in setup.py. It is run-time optional if you wish to disable it.

Fixed

- On the Farm page of the GUI Latest Block Challenge is now populated. This shows you the actual challenge that came from the Timelord. Index is the signage point index in the current slot. There are 64 signage points every 10 minutes on average where 32 sub blocks can be won.
- Last Attempted Proof is now fixed. This will show you the last time one of your plots passed the [plot filter](https://github.com/Chik-Network/chik-blockchain/wiki/FAQ#what-is-the-plot-filter-and-why-didnt-my-plot-pass-it).
- Plot filename is now back in the Plots table of the GUI.
- There was a bug in adding a sub block to weight proofs and an issue in the weight proof index.
- Over time the node would think that there were no peers attached with peak sub block heights higher than 0.
- There was a potential bug in Python 3.9.0 that required us to update blspy, chikpos, chikvdf, and chikbip158.
- An off by one issue could cause syncing to ask for 1 sub block when it should ask for e.g. 32.
- Short sync and backtrack sync both had various issues.
- There was an edge case in bip158 handling.

Known issues

- There is a remaining sync disconnect issue where your synced node will stop hearing responses from the network even though it still gets a few peaks and then stalls. Restarting node should let you quickly short sync back to the blockchain tip.

Skipped

Skipped

Added

- The GUI now displays sub blocks as well as transaction blocks on the Full Node page.
- `chik plots check` enforces a minimum of `-n 5` to decrease false negatives. Thanks to eFishCent for these ongoing pull requests!
- Testnets and mainnets will now have an initial period of sub blocks where transactions are blocked.
- Transaction volume testing added to tests and various tests have been sped up.
- We have added connection limits for max_inbound_wallet, max_inbound_farmer, and max_inbound_timelord.

Changed

- On starting full node, the weight proof cache does not attempt to load all sub blocks. Startup times are noticeably improved though there remains a hesitation when validating the mempool. Our klvm Rust implementation, which will likely ship in the next release, will drop example processing times from 180 to 3 seconds.
- Changes to weight proofs and sub block storage and cacheing required a new database schema. This will require a re-sync or obtaining a synced blockchain_v23.db.
- klvm bytecode is now generated and confirmed that the checked-in klvm and ChikLisp code matches the CI compiled code.
- We have removed the '-r' flag from `chik` as it was being overridden in most cases by the `-r` for restart flag to `chik start`. Use `chik --root-path` instead.
- `chik -h` now recommends `chik netspace -d 192` which is approximately one hours worth of sub blocks. Use `-d 1000` to get the same estimate of netspace as the RPC and GUI.
- `chik show -c` now displays in MiB and the GUI has been changed to MiB to match.
- `chik configure` now accepts the shorter `-upnp` and `-log-level` arguments also.
- `chik plots check` now defaults to `-n 30` instead of `-n 1` - HT eFishCent.
- `chik plots create` now enforces a minimum of k=22. As a reminder, anything less than k=32 is just for testing and be careful extrapolating performance of a k less than 30 to a k=32 or larger.
- We have updated development dependencies for setuptools, yarl, idna, multidict, and chardet.
- Updated some copyright dates to 2021.

Fixed

- We upgraded our fork of aiosqlite to version 16.0 which has significant performance improvements. Our fixes to aiosqlite are waiting to be upstreamed.
- The Plots tab in the GUI will no longer show red/error when the node is still syncing.
- Inbound and outbound peer connection limits were not being honored.
- Weight proofs were not correctly extending.
- In some cases when closing a p2p connection to another node, there was an infinite "Closing" loop.
- `chik show -c` was showing upload MiB in the download column and vice versa. pyl and psydafke deserves credit for insisting it was broken and kd637xx for the PR assist.
- `chik show` handles sub block 0 better.

Added

- Node now attempts to pre-validate and cache transactions.
- The harvester will try to not load a plot file that is too small for its k size. This should help keep from partial plots being found when they are copied into a harvester directory. Harvester will check again on the next challenge and find a completed copy of a plot file then.
- `chik plots create -x` skips adding [final dir] to harvester for farming

Changed

- We now use bech32m and have added the bech32m tests from Pieter Wuille (sipa) outlined [here](https://gist.github.com/sipa/14c248c288c3880a3b191f978a34508e) with thanks.
- In the GUI, choosing to parallel plot with a delay now is a delay between the start of the parallel plots started in one session.
- Removed loading plot file names when starting `chik plots create`; decreases plotter time when there are a lot of plots on the machine. Huge thanks to eFishCent for this PR!

Fixed

- Various fixes to improve node's ability to sync. There are still plenty of additional performance improvements coming for node so expect it to get easier to run on less powerful devices.
- Wallet now handles large amounts of coins much better and generally syncs better.
- Thanks to nup002 for the PR to use scientific notation in the logs for address_manager.select_peer timings.
- `chik show -h` now correctly states that you use the first 8 characters of the node id to remove a node on the cli.
- Thank you to wallentx for adding better help for `chik configure --enable-upnp`.
- Pull requests from forks won't have failures on CI.