Indy-node

| | | |

Major Fixes

| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Fixed and issue where logs were appearing in the old CLI. | | [INDY-1471](https://jira.hyperledger.org/browse/INDY-1471) |
| Fixed an issue where there were numerous blacklists under high loads. | | [INDY-1461](https://jira.hyperledger.org/browse/INDY-1461) |
| Fixed an issue where the pool stopped writing after 1114k txns (different view\_no). | | [INDY-1460](https://jira.hyperledger.org/browse/INDY-1460) |
| Fixed an issue where the "AttributeError: NoneType object has no attribute 'request' during load"; was appearing. | | [INDY-1464](https://jira.hyperledger.org/browse/INDY-1464) |
| Fixed an issue where the validator-info was reading an empty file. | | [INDY-1406](https://jira.hyperledger.org/browse/INDY-1406) |
| Fixed an issue where validator-info -v --json wasn't producing valid JSON. | | [INDY-1443](https://jira.hyperledger.org/browse/INDY-1443) |
| Fixed an issue where the first Pre-Prepare message had `incorrect state trie` root right after view\_change (on master replica). | | [INDY-1459](https://jira.hyperledger.org/browse/INDY-1459) |
| Fixed an issue where the pool could not order transactions because Node set incorrect watermarks after its restart. | | [INDY-1455](https://jira.hyperledger.org/browse/INDY-1455) |
| Fixed an issue where the pool stopped working due to several incomplete view changes. | | [INDY-1454](https://jira.hyperledger.org/browse/INDY-1454) |
| Fixed an issue where the node crashes on \_remove\_stashed\_checkpoints. | | [INDY-1427](https://jira.hyperledger.org/browse/INDY-1427) |
| Fixed an issue where memory was running out during non-completed viewChange process (under load). | | [INDY-1360](https://jira.hyperledger.org/browse/INDY-1360) |
| Fixed an issue where part of nodes continued ordering txns after `incorrect state trie` under load. | | [INDY-1422](https://jira.hyperledger.org/browse/INDY-1422) |
| Fixed an issue where the upgrade failed on pool from 1.3.62 to 1.4.66. | |[INDY-1447](https://jira.hyperledger.org/browse/INDY-1447) |
|Fixed an issue where a forced upgrade from 1.3.62 -> 1.5.67 without one node in schedule failed. | |[INDY-1519](https://jira.hyperledger.org/browse/INDY-1519) |
| | | | |



Changes - Additions - Known Issues

| Description | Workaround | Ticket |
| --- | --- | --- |
| Implemented periodic restart of client stack to allow new clients to connect. | | [INDY-1431](https://jira.hyperledger.org/browse/INDY-1431) |
| Got rid of peersWithoutRemotes. | | [INDY-1467](https://jira.hyperledger.org/browse/INDY-1467) |
| High Watermark on backup may be reset to 300. | | [INDY-1462](https://jira.hyperledger.org/browse/INDY-1462) |
| We now allow optional field in node-to-node and client-to-node. | | [INDY-1494](https://jira.hyperledger.org/browse/INDY-1494) |
| Catchup during view change may last forever under the load. | | [INDY-1463](https://jira.hyperledger.org/browse/INDY-1463) |
| Propagate Primary mode should not be set for already started view change. | | [INDY-1458](https://jira.hyperledger.org/browse/INDY-1458) |
| Catchup needs to be finished during high load. | | [INDY-1450](https://jira.hyperledger.org/browse/INDY-1450) |
| Included reviewed logging strings in Indy. | | [INDY-1416](https://jira.hyperledger.org/browse/INDY-1416) |
| Added benchmark performance impact of recorder tool. | | [INDY-1483](https://jira.hyperledger.org/browse/INDY-1483) |
| Decreased the amount of logging with INFO level. | | [INDY-1311](https://jira.hyperledger.org/browse/INDY-1311) |
| Made it so that throughput measurements in monitor should are windowed. | | [INDY-1435](https://jira.hyperledger.org/browse/INDY-1435) |
| Limited the number of requested PROPAGATES in MessageRequests. | | [INDY-1386](https://jira.hyperledger.org/browse/INDY-1386) |
| Made it so that any client requests during view change are not processed. | | [INDY-1453](https://jira.hyperledger.org/browse/INDY-1453) |
| Made it so that a node must send LEDGER\_STATUS with correct last ordered 3PC after catch-up. | | [INDY-1452](https://jira.hyperledger.org/browse/INDY-1452) |
| Fixed calculation of prepared certificates during View Change. | | [INDY-1385](https://jira.hyperledger.org/browse/INDY-1385) |
| Made it so that catchup should not be interrupted by external events. | | [INDY-1404](https://jira.hyperledger.org/browse/INDY-1404) |
| **Known Issue:** Upgrade failed on pool from 1.3.62 to 1.4.66. Note that INDY-1447 was fixed in indy-node 1.5.68, but it still presents in indy-node 1.3.62 and 1.4.66 code. So, some of the nodes may not to be upgraded during simultaneous pool-upgrade. If this problem will appear, stewards should perform manual upgrade of indy-node in accordance with this instruction: (!) To reduce the risk of reproducing INDY-1447, it is recommended to use old CLI for pool upgrade. | | [INDY-1447](https://jira.hyperledger.org/browse/INDY-1447) |
| | | | |

Upgrade Scripts:

**Pool upgrade from indy-node 1.3.62 should be performed simultaneously for all nodes due to txn format changes.**

Additional Information:

**All indy-cli pools should be recreated with actual genesis files.
For more details about txn format changes see INDY-1421.**

| | | |

Major Fixes

| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Fixed and issues where one of the nodes stopped writing after 44287 txns with errors in status. | | [INDY-1410](https://jira.hyperledger.org/browse/INDY-1410) |
| Fixed an issue where the pool stopped accepting transactions on 5731 txns (1 sec delays, no logging). | | [INDY-1365](https://jira.hyperledger.org/browse/INDY-1365) |
| Fixed an issue where the pool stopped writing after ~300,000 txns from 5 clients | | [INDY-1315](https://jira.hyperledger.org/browse/INDY-1315) |
| Fixed an issue where STN was not accepting transactions with only one node down. | | [INDY-1351](https://jira.hyperledger.org/browse/INDY-1351) |
| Fixed an issue where the pool stops taking txns at ~178k txns written in ledger. | | [INDY-1260](https://jira.hyperledger.org/browse/INDY-1260) |
| Fixed an issue where `ReqIdrToTxn` does not store information about the ledger. | | [INDY-1327](https://jira.hyperledger.org/browse/INDY-1327) |
| Made simple Timeout fixes of the current View Change protocol. | | [INDY-1341](https://jira.hyperledger.org/browse/INDY-1341) |
| Fixed an issue where the migration fails in case of upgrade to version with new transactions format. | | [INDY-1379](https://jira.hyperledger.org/browse/INDY-1379) |
| Fixed an issue where `--network parameter of read_ledger ` doesn't work. | | [INDY-1318](https://jira.hyperledger.org/browse/INDY-1318) |
| Fixed an issue where the /var/log/indy/validator-info.log was inappropriately owned by root. | | [INDY-1310](https://jira.hyperledger.org/browse/INDY-1310) |
| Created a fix around the issues found in the current logic of catch-up. | | [INDY-1298](https://jira.hyperledger.org/browse/INDY-1298) |
| Fixed GetValidatorInfo so it has correct validation for signature and permissions. | | [INDY-1363](https://jira.hyperledger.org/browse/INDY-1363) |
| Fixed an issue where there was an unhandled exception during node working. | | [INDY-1316](https://jira.hyperledger.org/browse/INDY-1316) |
| Fixed an issue where `validator-info` and `read_ledger` were giving inconsistent responses in node on provisional. | | [INDY-1219](https://jira.hyperledger.org/browse/INDY-1219) |
| Fixed an issue where the pool stops taking txns at 3000 writing connections. | | [INDY-1259](https://jira.hyperledger.org/browse/INDY-1259) |
| | | |
| | | | |

Changes - Additions - Known Issues

| Description | Workaround | Ticket |
| --- | --- | --- |
| Reviewed and replaced `assert` with exceptions in indy-plenum where needed. | | [INDY-810](https://jira.hyperledger.org/browse/INDY-810) |
| Tuned RocksDB options for the best performance. | | [INDY-1245](https://jira.hyperledger.org/browse/INDY-1245) |
| Created a migration guide from Indy-node 1.3 to 1.4. | | [INDY-1392](https://jira.hyperledger.org/browse/INDY-1392) |
| Сhanged a key in the requests map and field reqIdr in Pre Prepare and Ordered. | | [INDY-1370](https://jira.hyperledger.org/browse/INDY-1370) |
| Investigated issues found during load testing of 25-nodes pool with increased timeouts for catchups and viewchange. | | [INDY-1400](https://jira.hyperledger.org/browse/INDY-1400) |
| We now support binding on separate NICs for Client-to-Node and Node-to-Node communication. | | [INDY-1332](https://jira.hyperledger.org/browse/INDY-1332) |
| Added short checkpoints stabilization without matching digests. | | [INDY-1329](https://jira.hyperledger.org/browse/INDY-1329) |
| Added indy-crypto package to the hold list. | | [INDY-1323](https://jira.hyperledger.org/browse/INDY-1323) |
| Removed ledger status based catch-up trigger together with the wrong catch-up workflow. | | [INDY-1297](https://jira.hyperledger.org/browse/INDY-1297) |
| Read-ledger without storage copy in case of RocksDB (RocksDB read-only mode support). | | [INDY-1243](https://jira.hyperledger.org/browse/INDY-1243) |
| Applied state machine to Catchup code. | | [INDY-971](https://jira.hyperledger.org/browse/INDY-971) |
| Refactored the common Request structure. | | [INDY-1124](https://jira.hyperledger.org/browse/INDY-1124) |
| Refactored the common transactions structure. | | [INDY-1123](https://jira.hyperledger.org/browse/INDY-1123) |
| We now support the new libindy with changed txn format. | | [INDY-1319](https://jira.hyperledger.org/browse/INDY-1319) |
| Explored config parameters to find the best performance/stability settings. | | [INDY-1334](https://jira.hyperledger.org/browse/INDY-1334) |
| Extended the Validator Info tool to provide more information about the current state of the pool. | | [INDY-1175](https://jira.hyperledger.org/browse/INDY-1175) |
| A Steward needs to be able to get validator-info from all nodes. | | [INDY-1184](https://jira.hyperledger.org/browse/INDY-1184) |
| Modified existing load scripts for a better load testing. | | [INDY-1279](https://jira.hyperledger.org/browse/INDY-1279) |
| Performed a migration from LevelDB to RocksDB | | [INDY-1244](https://jira.hyperledger.org/browse/INDY-1244) |
| A Trustee needs to be able to restart the pool in critical situations. | | [INDY-1173](https://jira.hyperledger.org/browse/INDY-1173) |
| Move the log compression into separate process. | | [INDY-1275](https://jira.hyperledger.org/browse/INDY-1275) |
| **Known Issue:** There's an incorrect read\_ledger info with seq\_no parameter. | | [INDY-1415](https://jira.hyperledger.org/browse/INDY-1415) |
| **Known Issue:** Pool upgrade should be performed simultaneously for all nodes due to txn format changes. All indy-cli pools should be recreated with actual genesis files. | | |
| **List of breaking changes for migration from indy-node 1.3 to 1.4:** | | [1.3-1.4 Migration Guide](https://github.com/hyperledger/indy-node/blob/master/docs/1.3_to_1.4_migration_guide.md) |

Upgrade Scripts:

**Pool upgrade should be performed simultaneously for all nodes due to txn format changes.**

**All indy-cli pools should be recreated with actual genesis files.**

CLI Upgrading:

**Old CLI (`indy`):**

upgrade from 1.3 to 1.4 version
delete `~.ind-cli/networks/<network_name>/data` folder
replace both old genesis files by new ones (from 1.4 node)

**New CLI (`indy-cli`):**

upgrade from 1.4 to 1.5 version
recreate indy-cli pool using 1.4 pool genesis file (from 1.4 node)

Additional Information:

**List of breaking changes for migration from indy-node 1.3 to 1.4:**

https://github.com/hyperledger/indy-node/blob/master/docs/1.3\_to\_1.4\_migration\_guide.md