Salt-nornir

BUGS:

1. NETBOX secrets - If uses `secret_name_map: {password: username}` and `fetch_password: True`, secret name does not end up in username value, updated code to check `secret_name_map` for `username` and `password` keys to store secret names accordingly for `fetch_username` and `fetch_password` cases.
2. NETBOX secrets - if no plugins defined under secrets section, KeyError raised - updated code to improve error handling by raising RuntimeError with useful message instead of key error
3. nr.test suite validation was not working

FEATURES:

1. NETBOX secrets - added support to match secret role by secret role slug in addition to secret role name
2. Netbox pillar - added `host_primary_ip` parameter to control which primary IP (ip4 or ip6) to use as host's hostname
3. Netbox pillar - when `host_add_netbox_data` configured, sites now has list of tags retrieved
4. nr.netbox execution module added `rest` function to retrieve data from Netbox over REST API
5. Netbox pillar - added support for `netbox_secrets` plugin
6. Runner - added experimental nr.shell runner function to start interactive shell (using PICLE) based on salt nornir pydantic models
7. Execution module nr.test suite argument can now be a string which is either a YAML blob or a Jinja2 template to render to produce YAML blob of tests
8. Execution module nr.nornir connect added support for ``via`` attribute
9. Execution module nr.nornir connect added support to redipatch connection via terminal server using Netmiko
10. Execution module nr.nornir got new read_host_data function to read portions of host inventory data
11. Execution modeule nr.test added ``tests`` argument to support retrieving tests from host's inventory data. also added ``strict`` argument to control behavior if tests data not present in host's inventory
12. Added integration with Robot framework for nr.test function
13. Added integration with Robot framework for nr.cli function
14. Added new execution module function `nr.network` to implement network related utilites, so far added `resolve_dns` utility.

CHANGES

1. Netbox pillar - when `host_add_netbox_data` configured, device tags transformed into a list of tag names instead of list of tags dictionaries
2. nr.netbox `get_connections` and `get_interfaces` fully compatible with Netbox 3.4 only, this is due to changes introduced to GraphQL API on Netbox side
3. Runner nr.diagram set to increase netmiko read_timeout timer to 120 seconds by default
4. Netbox pillar no longer raises KeyError if no platform found for host in Netbox

DOCS

1. Converted Netbox Pillar tables to lists of items, as tables did not fit read the docs screen entirely.

BUGS

1. Netbox Pillar. Fixed handling of GraphQL queries that have significant number of filters, with previous behaviour query was encoded in parameters and was causing request to fail due to maximum parameters size was exceeded, this fix moves query content into reuqest body.
2. Netbox Pillar. Fixed handling of unsupported connections for get_connection function. This function only supports (for now) consoleports, consoleserverport and interface type of terminations, made sure that all other termination types are filtered out.
3. Netbox Pillar. Fixed `fetch_username` and `fetch_password` handling for actual secret values that are not a path to the secret in `nb://xyz` format, made sure to ignore them.

CHANGES

1. Netbox Pillar. Enhanced handling of hosts encoded in a config context of minion_id device by retrieving all devices details in one query as opposed to previously devices have been retrieved one by one, added tests to test this behaviour.
2. Netbox Pillar. On big number of hosts to retrieve from Netbox, minion was timing out, introduced multi-trheading support to speed up the process of retrieving data from Netbox. Adding new parameters `data_retrieval_num_workers` and `data_retrieval_timeout`.
3. Netbox Utils. Refactored nb_graphql function to support aliases
4. Netbox Pillar. Refactored ``get_connection``s functions to use GraphQL aliases to request termination points data in single query, as a result instead of 4 queries 2 queries required to retrieve data from Netbox
5. Netbox Pillar. Refactored ``get_interfaces`` function to retireve interfaces, ip adresses and inventory items in a single query using aliases, instead of 2 need to send 1 query now.
6. Netbox Pillar. ENhanced the process of rtetrieving hosts by host filters by packing all filters in a single query using GraphQL alises, now nomatter how many filters defined, all devices retrieved in a single query, while previously it was done one filter by one filter.
7. Dependencies updated and tested:
- netmiko 4.1.1 -> 4.1.2
- paramiko 2.11.0 -> 2.12.0
- pynetbox 6.6.2 -> 7.0.0
- tabulate 0.8.10 -> 0.9.0
- lxml 4.9.1 -> 4.9.2
- rich 12.5.1 -> 12.6.0

FEATURES

1. Execution Module nr.netbox ``query`` function added support for ``queries`` and ``query_string`` arguments

BUGS

1. Execution module nr.test - fix event progress job identity was lacking JID and Username
2. Proxy Module - list_hosts improved error handling while nornir workers are refreshing to return friendly message instead of traceback
3. Netbox pillar module - improved handling of connections that does not have remote device termination

FEATURES

1. Added support to Salt-Nornir Netbox Pillar to disable SSL verification if Netbox hosted over HTTPS using `ssl_verify` configuration parameter
2. Added interfaces speed and duplex to interfaces data retrieved from netbox

CHANGES

1. nr.test now renders tests suites using SaltStack as opposed to using Jinja2 directly within TestsProcessor, that allows to have access to all SaltStack execution modules functions and have access to wider set of inventory data while generating tests suites
2. Because of change 1 nr.test no longer supports `wait_timeout` features
3. Because of change 1 test `cli` argument now only support `Fx` filters attributes everything else is ignored, `Fx` filters used in conjunction with `subset` argument to limit the set of tests to run on certain hosts. Instead, additional arguments provided on nr.test call are shared across all nr.cli calls for the tests:


salt nrp1 nr.test suite="salt://tests/suite1.txt" enable=True


`enable=True` shared across all suite tests.

4. Removed support for netbox_tasks, as respective functionality removed from Nornir-Salt as well, the reason is - this could be replaced by nr.netbox execution module and netbox pillar module

FEATURES

1. Execution module nr.test now has `dry_run` argument to return per-host rendered test suites content without running tests
2. Execution module nr.test test suites rendered using SaltStack renreing system producing per-host test suites to run
3. nr.netbox - added new function `query` to run any arbitrary graphql query to Netbox, that would allow to use any Netbox data in Jinja2 templates for renring of configuration or test suites
3. Nornir-Salt TestsProcessor updated to do tests filtering and now supports `subset` argument
4. Nornir-Salt TestsProcessor extended to support dictionary test suites keyed by host names
5. Execution module nr.netbox and netbox pillar modules updated to use same query function from netbox_utils for interfaces and connection retrieval
6. Execution module nr.netbox added `get_interfaces` query
7. Execution module nr.netbox added `get_connections` query
8. Salt-Nornir Netbox pillar added `host_add_interfaces_inventory_items` argument to retireve interface inventory items
9. Salt-Nonir Netbox pillar added secret_name_map parameter to provide capability to record secret names in Nornir inventory.
10. Salt-Nonir Netbox pillar netbox_secretsore added url_override parameter to customize secrets plugin url
11. Execution module nr.netbox added `update_config_context` function to parse devices config and add it to configration context
12. Execution module nr.netbox added `update_vrf` function to parse devices config and create/update VRF and RT in Netbox

BUGS

1. Fixed nr.workflow state handling of steps that did not match any hosts to make sure we not running that step but going to next one instead.

CHANGES

1. Added `termionation_type` and `remote_termination_type` to Salt-Nornir Netbox pillar connections dictionary
2. Changes to introduce support for parsing devices config and pushing it to Netbox, such as added support for new inventory function `list_hosts_platforms` and updated lab setup to include TTP_Templates repository

BUGS

1. Fix Salt-Nornir Netbox pillar to properly form host data's "data" key if it does not exists prior, add interfaces and connection were affected in a way that "data" did not exist within host dictionary causing KeyError exception