Rolls-blockchain

Added

- Rate limited wallets can now have unspent and un-spendable funds clawed back by the Admin wallet.
- You can now backup your wallet related metadata in an encrypted and signed file to a free service from PecanRolls Network at backup.pecanrolls.net. Simply having a backup of your private key will allow you to fully restore the state of your wallet including coloured coins, rate limited wallets, distributed identity wallets and many more. Your private key is used to automatically restore the last backup you saved to the PecanRolls backup cloud service. This service is open source and ultimately you will be able to configure your backups to go to backup.pecanrolls.net, your own installation, or a third party's version of it.
- Added a Code of Conduct in CODE_OF_CONDUCT.md.
- Added a bug report template in `.github/ISSUE_TEMPLATE/bug_report.md`.

Changed

- This is a new blockchain as we changed how the default puzzle hashes are generated and previous coins would not be easy to spend. Plots made with Beta 8 and newer continue to work, but all previous test rolls are left on the old chain and do not migrate over. Configuration data like plot directories automatically migrate in your `~/.rolls` directory.
- Proof of Space now requires significantly less temp space to generate a new plot. A k=32 that used to require 524GiB now requires only 313GiB - generally a 40% decrease across all k sizes.
- When plotting, instead of 1 monolithic temp file, there are now 8 files - one for each of the 7 tables and one for sorting plot data. These files are deleted as the `-2` or `-d` final file is written so the final file can fit within the footprint of the temporary files on the same filesystem.
- We've made various additional CPU optimizations to the Proof of Space plotter that reduces plotting time by an additional 13%. These changes will also reduce CPU utilization in harvesting.
- We have ruled out k=30 for mainnet minimum plot size. k=31 may still make mainnet. k=32 and larger will be viable on mainnet.
- We moved to react-styleguidist to develop reusable components in isolation and better document the UI. Thanks to embiem for this pull request.
- Coloured coins have been updated to simplify them, remove 'a', and stop using an 'auditor'.
- clvm has been significantly changed to support the new coloured coins implementation.
- Bumped cryptography to 3.1. Cryptography is now publishing ARM64 binary wheels to PyPi so Raspberry Pi installs should be even easier.
- `rolls init` now automatically discovers previous releases in each new release.

Fixed

- `rolls show -w` should now more reliably work. Wallet balances should be more often correct.
- View -> Developer -> Developer Tools now correctly opens the developer tools. Thank you to roxaaams for this pull request!
- Fixed 'Receive Address' typo in Wallet. Thanks meurtn on Keybase.
- Fixed a typo in `rolls show -w` with thanks to pyl on Keybase.
- In Windows the start menu item is now PecanRolls Network and the icon in Add/Remove is updated.

Added

- The PecanRolls UI now has a proper About menu entry that gives the various component versions and directs people to submit issues on GitHub. Thank you to freddiecoleman for this pull request!
- Ability to run only the farmer, wallet, or timelord services, for more advanced configurations (rolls run farmer-only, wallet-only, timelord-only)

Changed

- To complement the new About menu, we have revamped all Electron menus and made them OS native. There are now direct links to the Wiki, Keybase, and FAQ in the Help menu.
- There are minor improvements to how working space is calculated and displayed by the plotter. The plotter also has additional debugging information in its output.
- Successful plots only have an atomic rename.

Fixed

- kOffsetSize should have been 10 bits and not 9. This was causing plots, especially larger plots, to fail with "Error 0". This bug was introduced in Beta 8 with the new plot file format.
- A bug in aiosqlite was causing tests to hang - especially on the ci. This may also have been causing wallet database corruption.
- `rolls show -w` now correctly outputs all wallet types and balances from the local wallet.

Added

- Meet our new Rate Limited wallet. You can now fund a wallet from an Admin wallet that will set how many coins can be spent over a given range of blocks for a given User wallet. Once combined with on chain wallet recovery, this makes it much easier to secure your "spending money" wallet so that if it is compromised you have time to get most of the funds back before an attacker can steal them all. This wallet should be considered alpha in this release as additional fixes and functionality will be coming in subsequent releases.
- We've added unhardened HD keys to bls-signatures for the smart wallets that need them. We've added significant cross project testing to our BLS implementation.
- The python implementation of bls-signatures is now current to the new specification.
- `rolls show -b` now returns plot public key and pool public key for each block.
- Added cbor2 binary wheels for ARM64 to the PecanRolls simple site. Raspberry Pi should be just a little easier to install.

Changed

- Wallet addresses and other key related elements are now expressed in Chech32 which is the PecanRolls implementation of [Bech32](https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki). All of your old wallet addresses will be replaced with the new Chech32 addresses. The only thing you can't do is send test rolls between 1.8/1.9 and 1.10 software. Anyone who upgrades to 1.10 will keep their transactions and balances of test rolls from the earlier two releases however.
- We added a first few enhancements to plotting speed. For a k=30 on a ramdisk with `-b 64 GiB` it results in an 11% speedup in overall plotting speed and a 23% improvement in phase 1 speed. Many more significant increases in plotting speed are in the works.
- The proof of space document in chiapos has been updated to the new format and edited for clarity. Additionally GitHub actions now has the on demand ability to create the PDF version.
- Relic has upstreamed our changes required for the IETF BLS standard. We now build directly from the Relic repository for all but Windows and will be migrating Windows in the next release.
- Minor improvements to the Coloured Coin wallet were integrated in advance of an upcoming re-factor.
- Smart wallet backup was upgraded to encrypt and sign the contents of the backup.

Fixed

- Proof of space plotting now correctly calculates the total working space used in the `-t` directory.
- `rolls show -w` now displays a message when balances cannot be displayed instead of throwing an error. Thanks to freddiecoleman for this fix!
- Fix issue with shutting down full node (full node processes remained open, and caused a spinner when launching PecanRolls)
- Various code review alerts for comparing to a wider type in chiapos were fixed. Additionally, unused code was removed from chiapos
- Benchmarking has been re-enabled in bls-signatures.
- Various node security vulnerabilities were addressed.
- Updated keyring, various GitHub actions, colorlog, cbor2, and clvm_tools.

Added

- See wallet balances in command line: `rolls show -w`
- Retry opening invalid plots every 20 minutes (so you can copy a large plot into a plot directory.)
- We've added `rolls keys sign` and `rolls keys verify` to allow farmers to certify their ownership of keys.
- Windows BLS Signature library now uses libsodium for additional security.
- You can now backup and restore Smart Wallet metadata.
- Binary wheels for ARM64/aarch64 also build for python 3.7.
- See and remove plot directories from the UI and command line.
- You can now specify the memory buffer in UI.
- Optimized MPIR for Sandybridge and Ivybridge CPUs under Windows

Changed

- `rolls start wallet-server` changed to `rolls start wallet`, for consistency.
- All data size units are clarified to displayed in GiB instead of GB (powers of 1024 instead of 1000.)
- Better error messages for restoring wallet from mnemonic.

Fixed

- Fixed open_connection not being cancelled when node exits.
- Increase the robustness of node and wallet shutdown.
- Handle disconnection and reconnection of hard drives properly.
- Addressed pre-Haswell Windows signatures failing.
- MacOS, Linux x64, and Linux aarch64 were not correctly compiling libsodium in
the blspy/bls-signatures library.
- Removed outdated "200 plots" language from Plot tab.
- Fixed spelling error for "folder" on Plot tab.
- Various node dependency security vulnerabilities have been fixed.
- Request peers was not returning currently connected peers older than 1 day.
- Fixed timeout exception inheritance changes under python 3.8 (pull 13528)

Deprecated

- Removed legacy scripts such as rolls-stop-server, rolls-restart-harvester, etc.

Added

- We have released a new plot file format. We believe that plots made in this
format and with these IETF BLS keys will work without significant changes on
mainnet at launch.
- We now use [chacha8](https://cr.yp.to/chacha.html) and
[blake3](https://github.com/BLAKE3-team/BLAKE3) for proof of space instead of
the now deprecated AES methods. This should increase plotting speed and support
more processors.
- Plot refreshing happens during all new challenges and only new/modified files
are read.
- Updated [blspy](https://github.com/strandedathome/bls-signatures) to use the
new [IETF standard for BLS signatures](https://tools.ietf.org/html/draft-irtf-cfrg-bls-signature-02).
- Added a faster VDF process which generates n-wesolowski proofs quickly
after the VDF result is known. This requires a high number of CPUs. To use it,
set timelord.fast_algorithm = True in the config file.
- Added a new type of timelord helper - blue boxes, which generate compact
proofs of time for existing proven blocks. This helps reducing the database
size and speeds up syncing a node for new users joining the network. Full nodes
send 100 random un-compact blocks per hour to blue boxes, and if
timelord.sanitizer_mode = True, the blue box timelord will work on those
challenges. Unlike the main timelord, average machines can run blue boxes
and contribute to the chain. Expect improvements to the install method for
blue boxes in future releases.
- From the UI you can add a directory that harvester will always check for
existing and new plots. Harvester will only look in the specific directory you
specify so you'll have to add any subfolders you want to also contain plots.
- The UI now asks for confirmation before closing and shows shutdown progress.
- UI now tries to shut down servers gracefully before exiting, and also closes
the daemon before starting.
- The various sub repositories (chiapos, chiavdf, etc.) now build ARM64 binary
wheels for Linux with Python 3.8. This makes installing on Ubuntu 20.04 lts on
a Raspberry Pi 3 or 4 easy.
- Ci's check to see if they have secret access and attempt to fail cleanly so
that ci runs successfully complete from PRs or forked repositories.
- Farmer now sends challenges after a handshake with harvester.
- The bls-signatures binary wheels include libsodium on all but Windows which
we expect to add in future releases.
- The rolls executable is now available if installing from the Windows or MacOS
Graphical installer. Try `./rolls -h` from
`~\AppData\Local\PecanRolls-Blockchain\app-0.1.8\resources\app.asar.unpacked\daemon\`
in Windows or
`/Applications/PecanRolls.app/Contents/Resources/app.asar.unpacked/daemon` on MacOS.

Changed

- Minor changes have been made across the repositories to better support
compiling on OpenBSD. HT n1000.
- Changed ROLLS units to TROLLS units for testnet.
- A push to a branch will cancel all ci runs still running for that branch.
- Ci's now cache pip and npm caches between runs.
- Improve test speed with smaller discriminants, less blocks, less keys, and
smaller plots.
- RPC servers and clients were refactored.
- The keychain no longer supports old keys that don't have mnemonics.
- The keychain uses BIP39 for seed derivation, using the "" passphrase, and
also stores public keys.
- Plots.yaml has been replaced. Plot secret keys are stored in the plots,
and a list of directories that harvester can find plots in are in config.yaml.
You can move plots around to any directory in config.yaml as long as the farmer
has the correct farmer's secret key too.
- Auto scanning of plot directories for .plot files.
- The block header format was changed (puzzle hashes and pool signature).
- Coinbase and fees coin are now in merkle set, and bip158 filter.
- New harvester protocol with 2/2 harvester and farmer signatures, and modified
farmer and full node protocols.
- 255/256 filter which allows virtually unlimited plots per harvester or drive.
- Improved create_plots and check_plots scripts, which are now
"rolls plots create" and "rolls plots check".
- Add plot directories to config.yaml from the cli with "rolls plots add".
- Use real plot sizes in UI instead of a formula/
- HD keys now use EIP 2333 format instead of BIP32, for compatibility with
other chains.
- Keys are now derived with the EIP 2334 (m/12381/8444/a/b).
- Removed the ability to pass in sk_seed to plotting, to increase security.
- Linux builds of chiavdf and blspy now use a fresh build of gmp 6.2.1.

Fixed

- uPnP now works on Windows.
- Log rotation should now properly rotate every 20MB and keep 7 historical logs.
- Node had a significant memory leak under load due to an extraneous fork
in the network code.
- Skylake processors on Windows without AVX would fail to run.
- Harvester no longer runs into 512 maximum file handles open issue on Windows.
- The version generator for new installers incorrectly handled the "dev"
versions after a release tag.
- Due to a python bug, ssl connections could randomly fail. Worked around
[Python issue 29288](https://bugs.python.org/issue29288)
- Removed websocket max message limit, allowing for more plots
- Daemon was crashing when websocket gets improperly closed

Deprecated

- All keys generated before Beta 1.8 are of an old format and no longer useful.
- All plots generated before Beta 1.8 are no longer compatible with testnet and
should be deleted.

Known Issues

- For Windows users on pre Haswell CPUs there is a known issue that causes
"Given G1 element failed g1_is_valid check" when attempting to generate
keys. This is a regression from our previous fix when it was upstreamed into
relic. We will make a patch available for these systems shortly.

Added

- Added ability to add plot from filesystem (you will need pool_pk and sk from plots.yaml.)
- Added ability to import private keys in the UI.
- Added ability to see private keys and mnemonic seeds in the keys menu
- User can specify log level in the config file (defaults to info.)
- The Windows installer is now signed by a PecanRolls Network certificate. It may take some time to develop enough reputation to not warn multiple times during install.

Changed

- Plots are now refreshed in the UI after each plot instead of at the end of plotting.
- We have made performance improvements to plotting speed on all platforms.
- The command line plotter now supports specifying it's memory buffer size.
- Test plots for the simulation and testing harness now go into `~/.rolls/test-plots/`
- We have completely refactored all networking code towards making each PecanRolls service use the same default networking infrastructure and move to websockets as the default networking wire protocol.
- We added additional improvements and more RPCs to the start daemon and various services to continue to make rolls start/stop reliable cross platform.
- The install.sh script now discovers if it's running on Ubuntu less than 20.04 and correctly upgrades node.js to the current stable version.
- For GitHub ci builds of the Windows installer, editbin.exe is more reliably found.
- All installer ci builds now obtain version information automatically from setuptools_scm and convert it to an installer version number that is appropriate for the platform and type of release (dev versus release.)
- We now codesign the Apple .dmg installer with the PecanRolls Network developer ID on both GitHub Actions and Azure Pipelines. We will be notarizing and distributing the Azure Pipelines version as it's built on MacOS Mojave (10.14.6) for stronger cross version support.

Fixed

- Having spaces in the path to a plot or temporary directory caused plotting to fail.
- Changing keys will no longer interrupt plotting log.
- 1.6 introduced a bug where certain very many core machines would sync the blockchain very slowly.
- The plotter log in the UI should scroll more reliably.
- The plotter UI should display the correct log on all platforms
- Starting rolls now waits for the full node to be active before contacting the introducer.