Indy-node

Additional Information:
**There are possible OOM issues during 3+ hours of target load or large catch-ups at 8 GB RAM nodes pool so 32 GB is recommended.**
**Pool upgrade to sovrin 1.1.32 and above should be performed simultaneously for all nodes due to txn format changes.**

Major Fixes
| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Validator-info to client times out if there are many upgrade attempts by node | | [INDY-1922](https://jira.hyperledger.org/browse/INDY-1922) |
| Node on Sovrin TestNet did not upgrade automatically | | [INDY-1919](https://jira.hyperledger.org/browse/INDY-1919) |
| Node that does not upgrade spams the config ledger | | [INDY-1918](https://jira.hyperledger.org/browse/INDY-1918) |
| Incorrect pool upgrade txn validation | | [INDY-1953](https://jira.hyperledger.org/browse/INDY-1953) |
| Upgrade appears to have broken "validator-info --nagios" | | [INDY-1920](https://jira.hyperledger.org/browse/INDY-1920) |
| Node can't order after view change and catch up | | [INDY-1955](https://jira.hyperledger.org/browse/INDY-1955) |
| Unclear error messages when Trustee send a NYM with the same verkey | | [INDY-1963](https://jira.hyperledger.org/browse/INDY-1963) |
| A role that has been removed can't be added back | | [INDY-1971](https://jira.hyperledger.org/browse/INDY-1971) |

Changes and Additions
| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Limit the number of attributes in schema | | [INDY-1914](https://jira.hyperledger.org/browse/INDY-1914) |
| Enable Clear Request Queue strategy | | [INDY-1836](https://jira.hyperledger.org/browse/INDY-1836) |
| A Node needs to be able to order requests received during catch-up | | [INDY-1876](https://jira.hyperledger.org/browse/INDY-1876) |
| Network maintenance role | | [INDY-1916](https://jira.hyperledger.org/browse/INDY-1916) |
| There should always be fresh enough signature of a state | | [INDY-933](https://jira.hyperledger.org/browse/INDY-933) |
| Node stops working without any services failure | | [INDY-1949](https://jira.hyperledger.org/browse/INDY-1949) |
| As a user of Valdiator Info script, I need to know whether the pool has write consensus and when the state was updated the last time | | [INDY-1928](https://jira.hyperledger.org/browse/INDY-1928) |
| Endorser permission not needed for ledger writes | | [INDY-1528](https://jira.hyperledger.org/browse/INDY-1528) |

Additional Information:
**There are possible OOM issues during 3+ hours of target load or large catch-ups at 8 GB RAM nodes pool so 32 GB is recommended.**

Changes - Additions
| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Add old instance change messages discarding | | [INDY-1909](https://jira.hyperledger.org/browse/INDY-1909) |
| Increase ToleratePrimaryDisconnection and bind re-try time | | [INDY-1836](https://jira.hyperledger.org/browse/INDY-1836) |
| Add check for None of replica's primary name during logging | | [INDY-1926](https://jira.hyperledger.org/browse/INDY-1926) |

Additional Information:
**There are possible OOM issues during 3+ hours of target load or large catch-ups at 8 GB RAM nodes pool so 32 GB is recommended.**

Major Fixes
| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Issuing a pool restart does not work if there is no consensus | | [INDY-1896](https://jira.hyperledger.org/browse/INDY-1896) |
| Intermittent test failure: test_primary_selection_increase_f | | [INDY-1872](https://jira.hyperledger.org/browse/INDY-1872) |
| Fix throughput class creation bug | | [INDY-1888](https://jira.hyperledger.org/browse/INDY-1888) |

Additional Information:

**Warning: Embedded command-line tool _indy_ is no longer available.** For further pool interaction use _indy-cli_ package https://github.com/hyperledger/indy-sdk/tree/master/cli

[**indy-test-automation repo**](https://github.com/hyperledger/indy-test-automation) has been created for end-to-end tests and additional test tools [**(INDY-1766)**](https://jira.hyperledger.org/browse/INDY-1766)

**The** [**INDY-1818**](https://jira.hyperledger.org/browse/INDY-1818) **(Init Indy Node should output Base58-encrypted verkey already) affects nodes adding.**

**validator-info output has been changed.** If you use validator-info as data source make sure that you have done necessary changes for compatibility

**There are possible OOM issues during 3+ hours of target load or large catch-ups at 8 GB RAM nodes pool so 32 GB is recommended.**

Major Fixes

| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Intermittent test failure: test_primary_selection_increase_f | | [INDY-1872](https://jira.hyperledger.org/browse/INDY-1872) |
| Pool stopped writing after production load with fees | | [INDY-1867](https://jira.hyperledger.org/browse/INDY-1867) |
| Node does not validate CLAIM_DEF's filed ref | | [INDY-1862](https://jira.hyperledger.org/browse/INDY-1862) |
| Sovrin package can't be upgraded | | [INDY-1850](https://jira.hyperledger.org/browse/INDY-1850) |
| Expected object or value error during batch handling | | [INDY-1849](https://jira.hyperledger.org/browse/INDY-1849) |
| Node that is not on network is shown as 'unreachable' | | [INDY-1842](https://jira.hyperledger.org/browse/INDY-1842) |
| Not enough information about upgrade in journalctl | | [INDY-1834](https://jira.hyperledger.org/browse/INDY-1834) |
| Need to fix 'aws_manage' playbook that fails when inventory directory is not specified | | [INDY-1827](https://jira.hyperledger.org/browse/INDY-1827) |
| Validator on Sovrin MainNet fails to upgrade, then fails to revert | | [INDY-1824](https://jira.hyperledger.org/browse/INDY-1824) |
| Pool stops writing during load testing against domain and pool ledgers together | | [INDY-1823](https://jira.hyperledger.org/browse/INDY-1823) |
| Node service stops during node key validation | | [INDY-1820](https://jira.hyperledger.org/browse/INDY-1820) |
| Init Indy Node should output Base58-encrypted verkey already | | [INDY-1818](https://jira.hyperledger.org/browse/INDY-1818) |
| Investigate slowness on TestNet due to demotion | | [INDY-1816](https://jira.hyperledger.org/browse/INDY-1816) |
| New instance was removed after creating | | [INDY-1815](https://jira.hyperledger.org/browse/INDY-1815) |
| Upgrade from 1.6.645+ version result errors about packages versions in journalctl | | [INDY-1781](https://jira.hyperledger.org/browse/INDY-1781) |
| RequestQueue in Replica doesn't clear after View Change | | [INDY-1765](https://jira.hyperledger.org/browse/INDY-1765) |

Changes - Additions - Known Issues

| Description | Workaround | Ticket |
| --- | --- | --- |
| Create a Diagram for Components | | [INDY-1870](https://jira.hyperledger.org/browse/INDY-1870) |
| Create Catch-up Sequence Diagram | | [INDY-1869](https://jira.hyperledger.org/browse/INDY-1869) |
| Write and Read request flow | | [INDY-1868](https://jira.hyperledger.org/browse/INDY-1868) |
| Plenum Consensus Protocol Diagram | | [INDY-1851](https://jira.hyperledger.org/browse/INDY-1851) |
| Change pool state root hash for BLS-signature in Commit messages | | [INDY-1846](https://jira.hyperledger.org/browse/INDY-1846) |
| 3rd party open source manifest | | [INDY-1839](https://jira.hyperledger.org/browse/INDY-1839) |
| Enable PreViewChange Strategy | | [INDY-1835](https://jira.hyperledger.org/browse/INDY-1835) |
| Need to add Names to AWS ec2 instances and security groups | | [INDY-1828](https://jira.hyperledger.org/browse/INDY-1828) |
| Need to securely automate SSH authenticity checking | | [INDY-1826](https://jira.hyperledger.org/browse/INDY-1826)- |
| Limit RocksDB memory consumption | | [INDY-1822](https://jira.hyperledger.org/browse/INDY-1822) |
| Run very long load test on a small local pool | | [INDY-1821](https://jira.hyperledger.org/browse/INDY-1821) |
| AWS tags for pool automation AWS resources | | [INDY-1813](https://jira.hyperledger.org/browse/INDY-1813) |
| Adjust last_ordered_3pc and perform GC when detecting lag in checkpoints on backup | | [INDY-1795](https://jira.hyperledger.org/browse/INDY-1795) |
| Improve usability of current pool automation PoC | | [INDY-1792](https://jira.hyperledger.org/browse/INDY-1792) |
| As a dev/QA I need to be able to refer different groups in the same namespace using one inventory | | [INDY-1788](https://jira.hyperledger.org/browse/INDY-1788) |
| Remove security groups at tear-down phase for both tests and playbooks | | [INDY-1784](https://jira.hyperledger.org/browse/INDY-1784) |
| Clear Requests queue periodically | | [INDY-1780](https://jira.hyperledger.org/browse/INDY-1780) |
| Test ZMQ Memory Consumption with restarting of listener on every X connections | | [INDY-1776](https://jira.hyperledger.org/browse/INDY-1776) |
| Get information about how many client connections is usually in progress | | [INDY-1775](https://jira.hyperledger.org/browse/INDY-1775) |
| Do a long test with a load pool can handle | | [INDY-1774](https://jira.hyperledger.org/browse/INDY-1774) |
| Find out why validation of PrePrepares with Fees takes so long | | [INDY-1773](https://jira.hyperledger.org/browse/INDY-1773) |
| Check why backup instances stop ordering so often | | [INDY-1772](https://jira.hyperledger.org/browse/INDY-1772) |
| As a dev I need to be able to perform tests on docker | | [INDY-1771](https://jira.hyperledger.org/browse/INDY-1771) |
| Test ZMQ Memory Consumption with restricted number of client connections | | [INDY-1770](https://jira.hyperledger.org/browse/INDY-1770) |
| Run load tests with file storages | | [INDY-1769](https://jira.hyperledger.org/browse/INDY-1769) |
| Change dependency building for upgrade procedure | | [INDY-1762](https://jira.hyperledger.org/browse/INDY-1762) |
| Use persisted last_pp_seq_no for recovery of backup primaries | | [INDY-1759](https://jira.hyperledger.org/browse/INDY-1759) |
| Extend Load Script with GET_TXN | | [INDY-1756](https://jira.hyperledger.org/browse/INDY-1756) |
| Avoid redundant static validation during signature verification | | [INDY-1753](https://jira.hyperledger.org/browse/INDY-1753) |
| Find out why max node prod time increases during long load test | | [INDY-1747](https://jira.hyperledger.org/browse/INDY-1747) |

Upgrade Scripts:

No further action is required

| | | |

Major Fixes

| Description | Additional Information | Ticket Number |
| --- | --- | --- |
| Re-asking for ledger statuses and maximal consistency proofs is not canceled. | | [INDY-1740](https://jira.hyperledger.org/browse/INDY-1740) |
| Bug in calling notifier methods in Restarter. | | [INDY-1741](https://jira.hyperledger.org/browse/INDY-1741) |
| 35 view changes were happened during 10 minutes after nodes failure because of invalid request. | | [INDY-1696](https://jira.hyperledger.org/browse/INDY-1696) |
| Requests queue is not cleared in case of reject-nym transactions. | | [INDY-1700](https://jira.hyperledger.org/browse/INDY-1700) |
| Throughput critically decreases without causing view_change. | | [INDY-1672](https://jira.hyperledger.org/browse/INDY-1740) |
| Node can&39;t catch up large ledger. | | [INDY-1595](https://jira.hyperledger.org/browse/INDY-1595) |
| Unable to demote node in STN. | | [INDY-1621](https://jira.hyperledger.org/browse/INDY-1621) |
| View changes happen when all responses should be rejected during load testing scenario. | | [INDY-1653](https://jira.hyperledger.org/browse/INDY-1653) |
| Node doesn&39;t write txns after disconnection from the rest nodes. | | [INDY-1580](https://jira.hyperledger.org/browse/INDY-1580) |
| Throughput is degrading if backup primary is stopped. | | [INDY-1618](https://jira.hyperledger.org/browse/INDY-1618) |
| | | | |

Changes - Additions - Known Issues

| Description | Workaround | Ticket |
| --- | --- | --- |
| Switch off a replica that stopped because disconnected from a backup primary. | | [INDY-1681](https://jira.hyperledger.org/browse/INDY-1681) |
| Extend load scripts emulating non-smooth load according to the changes in the core script. | | [INDY-1667](https://jira.hyperledger.org/browse/INDY-1667) |
| Proof of stability under load. | | [INDY-1607](https://jira.hyperledger.org/browse/INDY-1607) |
| Investigate Out of memory issues with the current load testing. | | [INDY-1688](https://jira.hyperledger.org/browse/INDY-1688) |
| Do not re-verify signature for Propagates with already verified requests. | | [INDY-1649](https://jira.hyperledger.org/browse/INDY-1649) |
| POA: Require multiple signatures for important transactions. | | [INDY-1704](https://jira.hyperledger.org/browse/INDY-1704) |
| Support all FEEs txns in the load script. | | [INDY-1665](https://jira.hyperledger.org/browse/INDY-1665) |
| Test domain transactions with FEEs. | | [INDY-1661](https://jira.hyperledger.org/browse/INDY-1661) |
| 3PC Batch should preserve the order of requests when applying PrePrepare on non-primary. | | [INDY-1642](https://jira.hyperledger.org/browse/INDY-1642) |
| Ability to switch off (remove) replicas with no changes of F value. | | [INDY-1680](https://jira.hyperledger.org/browse/INDY-1680) |
| A node should be able to participate in BLS multi-signature only if it has a valid proof of possession. | | [INDY-1589](https://jira.hyperledger.org/browse/INDY-1589) |
| Make validator info as a historical data. | | [INDY-1637](https://jira.hyperledger.org/browse/INDY-1637) |
| | | | |
| **Known Issue:** Upgrade failed on pool from 1.3.62 to 1.4.66. Note that INDY-1447 was fixed in indy-node 1.5.68, but it still presents in indy-node 1.3.62 and 1.4.66 code. | **So, some of the nodes may not to be upgraded during simultaneous pool-upgrade.** If this problem will appear, stewards should perform manual upgrade of indy-node in accordance with this [instruction:](https://docs.google.com/document/d/1vUvbioL5OsmZMSkwRcu0p0jdttJO5VS8K3GhDLdNaoI)**(!)** To reduce the risk of reproducing INDY-1447, it is **recommended to use old CLI for pool upgrade.** | [INDY-1447](https://jira.hyperledger.org/browse/INDY-1447) |
| | | | |

Upgrade Scripts:

**Pool upgrade from indy-node 1.3.62 to indy-node 1.6.78 should be performed simultaneously for all nodes due to txn format changes.**

Additional Information:

**All indy-cli pools should be recreated with actual genesis files.**

**For more details about txn format changes see** [**INDY-1421**](https://jira.hyperledger.org/browse/INDY-1421) **.**

**There are possible OOM issues during 3+ hours of target load or large catch-ups at 8 GB RAM nodes pool so 32 GB is recommended.**