Wrangles

- Bugfix: fix issues caused for sql read/write with possible incompatible sqlalchemy/pandas versions.
- Enable support and tests for python 3.12

- _extract.ai_
- Pass through any additional unspecified parameters to the backend API.
- Assume outputs of type array without a child type defined should be strings.
- Added the ability to include header level messages.
- Improved error handling. Clearer errors for invalid schema or API keys.
- Allow the URL to be overridden.
- _create.embeddings_
- Pass through any additional unspecified parameters to the backend API.
- Allow the URL to be overridden.
- Bugfix: Exponential backoff for retries was not working.
- Added model parameter to schema.
- _merge.concatenate_
- Fix errors caused by non-string values.
- Add skip_empty parameter to skip empty values.
- Allow wrangles to be used to _rename_ columns.
- Added by parameter for _create.index_. This will create a sequential index grouped by the defined columns.
- Added a default parameter for _convert.from_json_ and _convert.from_yaml_ in the case of an empty of erroneous input.
- Added a handler for more data types for _convert.to_json_ including datetimes, numpy arrays and numpy floats/ints.
- Added _select.sample_ - the rows can be specified as whole numbers or a decimal. 5 = 5 rows, 0.5 = 50% of rows.
- Make start or length optional for _select.substring_.
- Added the ability to reference custom functions to set the value of a variable.
- Added a _sort_ wrangle.
- Bugfix: Fixed the schema definition for the wrangle _recipe_.
- Added the name of the recipe to the log entry.
- Add the ability to log info, warning or error messages with the _log_ wrangle.
- Fixed pytest on v7.4.4 due to breaking changes in 8.0.0
- Allow the value for _create.column_ to be a list or other complex object.
- Added use_spellcheck parameter to _extract.custom_.

- Fixed a bug where select.list_elements could fail rather than returning an empty result.
- Fix columns with spaces for math - should be referenced with underscores.
- Various remove_words bugs: fix dropped characters, non-string values and support a mixture of string/lists.
- Bugfix: ensure columns wildcard searches are treated as strings
- Support pickle files with the file connector
- Enable where for read
- Enable optional columns by adding a ?
- Bugfix: adjusted cosine similarity could fail due to floats being fractionally above 1
- Bugfix: capped adjusted cosine similarity score to 0-1 range.

- Fixed a bug that caused issues when writing to subdirectories using the SFTP connector.

- Added excel.sheet connector. This is for use by WranglesXL only.

- _select.group_by_:
- Added list option to gather rows as a python list.
- Bugfix: Enable where to work.
- Bugfix: Fix an issue when by and an aggregation contained the same column.
- Added _format.significant_figures_ to round numbers to a specified number of significant figures.
- Added _select.columns_ to be able to only include specified columns.
- Added _explode_ to explode rows that contain lists over many rows.
- Added _select.head_ to get the first n rows.
- Added _select.tail_ to get the last n rows.
- Added unit conversion schema and tests for _extract.attributes_.
- Added _similarity_ wrangle to calculate the similarity between two vectors. This can use cosine or euclidean calculations.
- Added case sensitive option for _standardize_.
- Added case sensitive option for _extract.custom_.
- _create.embeddings_:
- Prevent empty strings causing failure.
- Added ability to return embeddings as a numpy array.
- Allow multiple inputs and outputs.
- Optimize by using base64 encoding for requests.
- Added retries parameter to specify the number of times to retry if the API request fails. This uses exponential backoff to help with rate limiting.
- _extract.ai_ - Added retries parameter to specify the number of times to retry if the API request fails. This uses exponential backoff to help with rate limiting.
- Added a more succinct option for creating columns by defining as a dictionary of column headings and values. This supports the same dynamic column generation options as the test connector.
- Bugfix: Rename had strange behaviour when the target column already existed.
- Give a clearer error message if the recipe file is invalid.