Salt-nornir

BUGS

1. Netbox Pillar. Fixed handling of GraphQL queries that have significant number of filters, with previous behaviour query was encoded in parameters and was causing request to fail due to maximum parameters size was exceeded, this fix moves query content into reuqest body.
2. Netbox Pillar. Fixed handling of unsupported connections for get_connection function. This function only supports (for now) consoleports, consoleserverport and interface type of terminations, made sure that all other termination types are filtered out.
3. Netbox Pillar. Fixed `fetch_username` and `fetch_password` handling for actual secret values that are not a path to the secret in `nb://xyz` format, made sure to ignore them.

CHANGES

1. Netbox Pillar. Enhanced handling of hosts encoded in a config context of minion_id device by retrieving all devices details in one query as opposed to previously devices have been retrieved one by one, added tests to test this behaviour.
2. Netbox Pillar. On big number of hosts to retrieve from Netbox, minion was timing out, introduced multi-trheading support to speed up the process of retrieving data from Netbox. Adding new parameters `data_retrieval_num_workers` and `data_retrieval_timeout`.
3. Netbox Utils. Refactored nb_graphql function to support aliases
4. Netbox Pillar. Refactored ``get_connection``s functions to use GraphQL aliases to request termination points data in single query, as a result instead of 4 queries 2 queries required to retrieve data from Netbox
5. Netbox Pillar. Refactored ``get_interfaces`` function to retireve interfaces, ip adresses and inventory items in a single query using aliases, instead of 2 need to send 1 query now.
6. Netbox Pillar. ENhanced the process of rtetrieving hosts by host filters by packing all filters in a single query using GraphQL alises, now nomatter how many filters defined, all devices retrieved in a single query, while previously it was done one filter by one filter.
7. Dependencies updated and tested:
- netmiko 4.1.1 -> 4.1.2
- paramiko 2.11.0 -> 2.12.0
- pynetbox 6.6.2 -> 7.0.0
- tabulate 0.8.10 -> 0.9.0
- lxml 4.9.1 -> 4.9.2
- rich 12.5.1 -> 12.6.0

FEATURES

1. Execution Module nr.netbox ``query`` function added support for ``queries`` and ``query_string`` arguments

BUGS

1. Execution module nr.test - fix event progress job identity was lacking JID and Username
2. Proxy Module - list_hosts improved error handling while nornir workers are refreshing to return friendly message instead of traceback
3. Netbox pillar module - improved handling of connections that does not have remote device termination

FEATURES

1. Added support to Salt-Nornir Netbox Pillar to disable SSL verification if Netbox hosted over HTTPS using `ssl_verify` configuration parameter
2. Added interfaces speed and duplex to interfaces data retrieved from netbox

CHANGES

1. nr.test now renders tests suites using SaltStack as opposed to using Jinja2 directly within TestsProcessor, that allows to have access to all SaltStack execution modules functions and have access to wider set of inventory data while generating tests suites
2. Because of change 1 nr.test no longer supports `wait_timeout` features
3. Because of change 1 test `cli` argument now only support `Fx` filters attributes everything else is ignored, `Fx` filters used in conjunction with `subset` argument to limit the set of tests to run on certain hosts. Instead, additional arguments provided on nr.test call are shared across all nr.cli calls for the tests:


salt nrp1 nr.test suite="salt://tests/suite1.txt" enable=True


`enable=True` shared across all suite tests.

4. Removed support for netbox_tasks, as respective functionality removed from Nornir-Salt as well, the reason is - this could be replaced by nr.netbox execution module and netbox pillar module

FEATURES

1. Execution module nr.test now has `dry_run` argument to return per-host rendered test suites content without running tests
2. Execution module nr.test test suites rendered using SaltStack renreing system producing per-host test suites to run
3. nr.netbox - added new function `query` to run any arbitrary graphql query to Netbox, that would allow to use any Netbox data in Jinja2 templates for renring of configuration or test suites
3. Nornir-Salt TestsProcessor updated to do tests filtering and now supports `subset` argument
4. Nornir-Salt TestsProcessor extended to support dictionary test suites keyed by host names
5. Execution module nr.netbox and netbox pillar modules updated to use same query function from netbox_utils for interfaces and connection retrieval
6. Execution module nr.netbox added `get_interfaces` query
7. Execution module nr.netbox added `get_connections` query
8. Salt-Nornir Netbox pillar added `host_add_interfaces_inventory_items` argument to retireve interface inventory items
9. Salt-Nonir Netbox pillar added secret_name_map parameter to provide capability to record secret names in Nornir inventory.
10. Salt-Nonir Netbox pillar netbox_secretsore added url_override parameter to customize secrets plugin url
11. Execution module nr.netbox added `update_config_context` function to parse devices config and add it to configration context
12. Execution module nr.netbox added `update_vrf` function to parse devices config and create/update VRF and RT in Netbox

BUGS

1. Fixed nr.workflow state handling of steps that did not match any hosts to make sure we not running that step but going to next one instead.

CHANGES

1. Added `termionation_type` and `remote_termination_type` to Salt-Nornir Netbox pillar connections dictionary
2. Changes to introduce support for parsing devices config and pushing it to Netbox, such as added support for new inventory function `list_hosts_platforms` and updated lab setup to include TTP_Templates repository

BUGS

1. Fix Salt-Nornir Netbox pillar to properly form host data's "data" key if it does not exists prior, add interfaces and connection were affected in a way that "data" did not exist within host dictionary causing KeyError exception

CHANGES

1. Proxy module refresh_nornir behavior updated to not do saltutil.sync_all and saltutil.refresh_pillar as call pillar.items updates pillar regardless, that should improve nr.nornir refresh calls speed, as pillar no longrer updated three times. Had to ad this change as new netbox pillar netbx-secretstore was failing due to salt was trying to refresh secrets simultaneously due to previous behaviour.
2. Dependencies updated and tested:
- pydantic 1.9.1 -> 1.10.2
- jmespath 1.0.0 -> 1.0.1
- nornir-scrapli 2022.1.30 -> 2022.07.30
- pygnmi 0.8.4 -> 0.8.9
- requests 2.27.1 -> 2.28.1
- scrapli 2022.1.30 -> 2022.07.30
- scrapli-community 2022.1.30 -> 2022.07.30
- scrapli-netconf 2022.1.30 -> 2022.07.30
- tabulate 0.8.9 -> 0.8.10
- lxml 4.8.0 -> 4.9.1
- jinja2 3.0.3 -> 3.1.2

FEATURES

1. Added new pillar module salt_nornir_netbox to source proxy minion pillar data from Netbox and to source hosts data from Netbox, this module has integration with netbox-secretstore plugin to source secrets from Netbox.
2. Execution module ``nr.test`` added support for Jinja2 templates test suites