Ttp

CHANGES

1. `ttp.lazy_import_functions` changes log level to warning if fails to save `ttp_dict.pickle` to the file system 54
2. Fixed support for group set function to retrieve variable values from `_ttp_["global_vars"]`

FEATURE

1. Added support for table formatters (excel, tabulate) to support "strict" keyword for traverse function to not fail if no path present in result

CHANGES

1. ttp.lazy_import_functions added explicit check for ttp dictionary cache existence instead of relying on try/except
2. ttp.lazy_import_functions added encoding keyword to open() function while loading TTP files

BUGS

1. Added empty tag handling for Python loader to not throw an error.

FEATURES
1. Added support for file returner to format filename using _ttp_["vars"] and _ttp_["global_vars"] dictionaries, as a result filenames can now be formatted on a per results basis 41
2. Added support to gethostname to extract hostname from Nokia SROS prompt 43
3. Made regexes to account for \r\n ending in addition to \n to detect a new line 44
4. Added support for excel formatter to update existing spreadsheets using "update" parameter
5. Made comments to support indentation without generating log message
6. Added outputters "condition" attribute to be able to run outputters conditionally based on variable in vars passed to TTP object

BUGS

1. Improved behavior of results selection and fixed bug with same group name/path results selection by adding grp_id in self.record to track previous group identification 37
2. Added support for \r\n line ending in addition to \n 44
3. Fixed joinmatches to properly handle sanitized match variable names
4. Fixed BUG with match result selection for multiple start re matches

CHANGES

1. TTP Lazy importing system modified to use pickle module to cache _ttp_ dictionary and to scan subfolders' files one by one,
that is to avoid "OSError: [Errno 24] Too many open files" problem when TTP scans through folders to find all function;
additionally, added caching functions to speed up initialization process roughly by 60ms.
2. Output traverse function added "strict=True" boolean parameter, if True - traverse raises KeyError if no path item found
in results, if False, will return empty dictionary if no item found in results
3. Improved Syslog returner - wrapped sending logs to try-except blocks with logging of errors on failure
4. Improved results path selection to use group.group_id instead of path and index attributes
5. Generalized _anonymous_ group name, path and group_id formation to account for nested/child _anonymous_ groups
6. Modified yangson validate behavior to return error if no yangson installed, added option to just return results without metadata if validation was successful

BUGS fixes
1. Fixed _headers_ to not match a lot of garbage, even when line is much shorter than headers string
2. Fixed bug with 'add' adding results for previous group in next group due to false matches 37

FEATURES
1. Added support to match Huawei style MAC addresses in MAC pattern
2. Group default attribute can reference template variable, template variable can be a dictionary or a string
3. Added support for parsing results YANG modules validation using yangson library
4. TTP Templates can be referenced using "ttp://<path>" syntaxis

NEW FEATURES
1. Added TTP object add_function method to add reference to custom functions for groups, variables, outputs, macro etc.
2. Added support for output deepdiff to reference template variable content as a data to compare with.
3. Inputs can accept structured data, dictionary or list or etc. This data need to be pre-processed within templates using input macro functions
4. Added support to sanitize match variables names so it can contain any symbols or start with digit
5. Added inter-log interval to syslog returner, set default to 1ms

CHANGES
1. form_result template function logic simplified, to produce consistent results structure across normal and _anonymous_ groups
2. made name="/" not to produce empty key
3. if output formatter is jinja2, loader for Outputter automatically set to text
4. passing self.attributes to returners and formatters in outputs to unpack them with **kwargs, useful for custom functions
5. Removed support for adding file objects to inputs, instead string with OS path to file should be used
6. Added "vars" keyword to _ttp_dictionary to store in run variables in addition to results_object.vars reference, updated built-in functions to use _ttp_["vars"]

BUGS fixes
1. Make file returner to convert non string data to text and save it
2. Fixed bug with headers indicator when last column value is shorter then headers or last column is empty
3. Fixed csv loader include statement, now lookups and variables can be loaded from csv files
4. Fixed/Added support for joinmatches to have "ignore" indicator in line

NEW FEATURES
1. resub and resuball function can reference regex patterns from template vars
2. _exact_space_ match variable indicator
3. Added group record function
4. Added group set function
5. Added support to groups for absolute path
6. Added support for filter list to ttp.add_template method, to allow filtering of child templates that need to be loaded from text file that contains several templates or from template that has child <template> tags defined, filter contains names of child templates to be loaded
7. Added filter support for parent templates names, now they can be filtered as well - if no name of template specified in filter, that template content will be ignored and no results produced
8. Added gpvlookup to lookup match results against glob patterns
9. Added support for geoip2 db lookups - geoip_lookup match variable function
10. Added support for lookup tag databse loaders
11. Added match variable geoip_lookup function
12. Added support for group contains_val and exclude_val to reference <vars> variable;
13. Added support for match variable equal, non_equal and cidr_match to reference <vars> variable;
14. Added cli tool -v/--vars keyword to supply json string-dictionary of variables to add to ttp parser object
15. Added colouring to terminal returner using colorama
16. Added group validate function that used Cerberus to check group result
17. Modified outputter validate function to have features similar to group validate function
18. Match variable function equal/notequal/exclude/append/prepend/contains/replaceall can reference Template variables
19. Added TTP variable get_timestamp_iso function
20. Added support for flat_list result structure to flatten overall results
21. Added group 'expand' function to expand dot separated variables names to nested dictionary
22. Added 'clear_result' method to ttp parser object
23. Added outputter support for N2G formatter to produce xml diagrams
24. Added group lookup function
25. added support for "doc" tag - TTP will not complain about it and this tag can be used to put template description in it
26. Added support for group functions to reference list of functions in template variable using chain or functions attribute
27. Added default name to templates - "_root_template_", added template_name attribute to add_input, set_input, clear_input methods; get_inout_load method now returns dictionary of templates and input names.
28. Added syslog returner to outputs using logging syslog handler to return results to syslog server over udp
29. Added support for netmiko as an input source function
30. Added support for "_headers_" indicator to dynamically construct regexes line to match tabulated data using exact number of spaces
31. Added support for Nornir as an input source
32. Added reference to variables for match variable conditional functions to introduce dynamic filtering capabilities


CHANGES
1. Modified variable and pattern line escaping behavior, to fix bug with several match variables with digits in name e.g. d1_1, d1_2
2. Modified behavior of DEFAULTS handling in results object to add them at the very end and make them the least preferred in comparison to results values.
3. Modified per_template results method handling to flatten results structure by converting overall result to dictionary for named groups and using single list for anonymous group
4. Results object processgrp now adds default to results instead of save_results method
5. Updated template object form results to handle _anonymous_ and named groups so that results for both groups included in output
6. Deleted get_input_commands_list and get_input_commands_dict parser objects, as they no longer deemed to be useful
7. Data on add associated with templates straight away instead of being saved in ttp.__data list. As a result, templates must be added before data will be added, if data added before template, that data will not be processed, as there are not templates/inputs to associate it with.
8. Outputters now load tag data and merge it with attributes if its a dictionary.

TESTS
1. test1-48 for variable with digits in name and for _exact_ indicator
2. Test 1-50-x for group record and set functions
3. Test 1-51 to test group absolute path handling
4. Test 1-52 to test let vs set behavior when defaults must be overridden by match values
5. Test 1-53 for _exact_space_ indicator
6. Test 29-1 and 29-2 gpvlookup tests
7. Test 1-50 for group macro attribute with comma separated macro names
8. Added tests 1-56, 1-57 to test match cidr_match and group contains_val functions referencing to <vars> variable
9. Test 1-58 to test group validate function
10. Results forming for template with anonymous and named group to test change-5
11. Started moving tests to pytest - 31 tests added so far

DOCS
1. Added HOWTO for parsing text data structured in tables
2. Added docs for group record and set functions
3. Added docs for group absolute path
4. Added docs to 'ignore' indicator saying that regex can come from re patterns or from template variables
5. Added docs to cli tool about new keywords options -ot and -s
6. Added docs about _exact_space_ indicator
7. Added docs for gpvlookup match variable function
8. Added docs for geoip_lookup function
9. Updated docs for group contains_val and exclude_val, for match variable equal, non_equal and cidr_match to mention that they can reference <vars> variable
10. Added docs for group 'expand' function
11. Updated docstring for 'flat_list' structure
12. Flattened docs structure, created separate section for results structure
13. Wrote docs for _headers_ indicator
14. Updated summary notes to installation page for other packages to be installed for certain features to work
15. Documented N2G output formatter
15. Changed output formatters to include all supported arguments within each of them, rather then having separate section for formatter's attributes.
16. Added new cli tool -v/--vars argument in the docs
17. Added docs about coloring terminal returner output using colorama
18. Modified inputs docs - merged attributes and parameters together
19. Added docs for syslog returner
20. Added docs for netmiko source
21. Added docs about Norninr source
22. Added docs about group validate function

BUGS
1. fixed bug with get_attributes function when empty match variable item give e.g. {{ var | | func }}
2. Fixed bug for variables having digit in name, that digit was substituted with \d+ pattern, resulting in match varibales have the same name, which is prohibited for regex groups
3. Fixed bug for py2.7 struct load for exec() function when text contains "True" of "False" by skipping the error, as exec chokes on it while eval used to load it. Not an issue in py3 though.
4. Fixed problem with set overriding let values due to default values override results dictionary when group method="table" and we have dynamic path in use to form dictionary, override happen due to the fact that each match was written to results straight away together with defaults values and if we have let function and unconditional set (set to defaults) values for same variable, let value was overridden by defaults from set.
5. Fixed handling of absolute path together with _anonymous_ parent group
6. Fixed template form_results function to handle the case when _anonymous_ group produces dictionary results in per_template mode
7. Load struct function was fixed to correctly handle the case when template vars tag has include attribute but has not text defined in XML element
8. Fixed traceback display issue with get_attributes function, log string had wrong formatting
9. Fixed truncate function and added test to test it
10.Fixed issue with group macro having several macro names separated by comma, added test to test it
11. TTP duplicate logs when used as a module - changed log config function
12. Fixed bug with _anonymous_ template introduced by adding r'' raw strings
13. Fixed csv formatter results handling when they contains integers
14. Fixed this bug:
<group name="interfaces.{{ interface }}" input="run config">
interface {{ interface }}
description {{ description | ORPHRASE }}
</group>
was returning no results for interfaces without descriptions
15. Fixed CSV formatter to handle "," properly in matches, by vrapping items in '"' or "'" characters