Hlink

Added

* Documented the `concat_two_cols` column mappings transform. You can see the
documentation [here][concat-two-cols-docs]. [PR 126][pr126]
* Documented column mapping overrides, which can let you read two columns with
different names in the input files into a single hlink column. The documentation for
this feature is [here][column-mapping-overrides-docs]. [PR 129][pr129]

Fixed

* Fixed a bug where config validation checks did not respect column mapping overrides.
[PR 131][pr131]

Added

* Added config validation checks for duplicate comparison features, feature selections,
and column mappings. [PR 113][pr113]
* Added support for Python 3.12. [PR 119][pr119]
* Put the config file name in the script prompt. [PR 123][pr123]

Fixed

* Reverted to keeping invalid categories in training data instead of erroring out.
This case actually does occasionally happen, and so we would rather not error out
on it. This reverts a change made in [PR 109][pr109], released in v3.5.2. [PR 121][pr121]

Changed

* Made some minor updates to the format of training step 3's output. There are now
3 columns: `feature_name`, `category`, and `coefficient_or_importance`. Feature
names are not suffixed with the category value anymore. [PR 112][pr112]
* BUG reverted in v3.5.3: Started erroring out on invalid categories in training
data instead of creating a new category for them. [PR 109][pr109]

Fixed

* Fixed a bug with categorical features in training step 3. Each categorical feature
was getting a single coefficient when each *category* should get its own coefficient
instead. [PR 104][pr104], [PR 107][pr107]

Added

* Made a new training step 3 to replace model exploration step 3, which was buggy.
Training step 3 saves model feature importances or coefficients when `training.feature_importances`
is set to true. [PR 101][pr101]

Removed

* Removed the buggy implementation of model exploration step 3. Training step 3 replaces
this. [PR 101][pr101]

Added

* Added support for Python 3.11. [PR 94][pr94]
* Created a new `multi_jaro_winkler_search` comparison feature. This is a complex
comparison feature which supports conditional Jaro-Winkler comparisons between
lists of columns with similar names. You can read more in the documentation [here][multi-jaro-winkler-search-docs].
[PR 99][pr99]

Changed

* Upgraded from PySpark 3.3 to 3.5. [PR 94][pr94]

Deprecated

* Deprecated the `hlink.linking.transformers.interaction_transformer` module.
Please use PySpark 3's [`pyspark.ml.feature.Interaction`][pyspark-interaction-docs]
class instead. Hlink's `interaction_transformer` module is scheduled for removal
in version 4. [PR 97][pr97]

Fixed

* Fixed a bug where the hlink script's autocomplete feature sometimes did not work
correctly. [PR 96][pr96]

Added

* Created a new `convert_ints_to_longs` configuration setting for working with CSV
files. Documentation for this setting is available [here][ints-to-longs-docs]. [PR 87][pr87]
* Improved the link tasks documentation by adding more detail. This page is available
[here][link-tasks-docs]. [PR 86][pr86]

Removed

* Dropped the `comment` column from the script's `desc` command. This column was
always full of nulls and cluttered up the screen. [PR 88][pr88]