Smartredis

Released on April 12, 2023

Description

This release provides a variety of features to improve usability and
debugging of the SmartRedis library, notably including Unix domain
socket support, logging, the ability to print a textual representation
of a string or dataset, dataset inspection, documentation updates, fixes
to the multi-GPU support, and much more:

- Prepare 0.4.0 release
- Disable codecov CI tests
- Improved error message in to_string methods in C interface
- Streamlined PyBind interface layer
- Updated Python API documentation
- Streamlined C interface layer
- Improved performance of get, put, and copy dataset methods
- Fix a bug which prevented multi-GPU model set in some cases
- Streamline pipelined execution of tasks for backend database
- Enhance code coverage to include all 4 languages supported by
SmartRedis
- Fix a bug which resulted in wrong key prefixing when retrieving
aggregation lists in ensembles
- Correct assorted API documentation errors and omissions
- Improve documentation of exception handling in Redis server classes
- Improve error handling for setting of scripts and models
- Add support to inspect the dimensions of a tensor via
get_tensor_dims()
- Split dataset prefixing control from use_tensor_ensemble_prefix() to
use_dataset_ensemble_prefix()
- Update to the latest version of redis-plus-plus
- Update to the latest version of PyBind
- Change documentation theme to sphinx_book_theme and fix doc strings
- Add print capability for Client and DataSet
- Add support for inspection of tensors and metadata inside datasets
- Add support for user-directed logging for Python clients, using
Client, Dataset, or LogContext logging methods
- Add support for user-directed logging for C and Fortran clients
without a Client or Dataset context
- Additional error reporting for connections to and commands run
against Redis databases
- Improved error reporting capabilities for Fortran clients
- Python error messages from SmartRedis contain more information
- Added logging functionality to the SmartRedis library
- A bug related to thread pool initialization was fixed.
- This version adds new functionality in the form of support for Unix
Domain Sockets.
- Fortran client can now be optionally built with the rest of the
library
- Initial support for dataset conversions, specifically Xarray.

Detailed Notes

- Update docs and version numbers in preparation for version 0.4.0.
Clean up duplicate marking of numpy dependency
([PR321](https://github.com/CrayLabs/SmartRedis/pull/321))
- Remove codecov thresholds to avoid commits being marked as
\'failed\' due to coverage variance
([PR317](https://github.com/CrayLabs/SmartRedis/pull/317))
- Corrected the error message in to_string methods in C interface to
not overwrite the returned error message and to name the function
([PR320](https://github.com/CrayLabs/SmartRedis/pull/320))
- Streamlined PyBind interface layer to reduce repetitive boilerplate
code ([PR315](https://github.com/CrayLabs/SmartRedis/pull/315))
- Updated Python API summary table to include new methods
([PR313](https://github.com/CrayLabs/SmartRedis/pull/313))
- Streamlined C interface layer to reduce repetitive boilerplate code
([PR312](https://github.com/CrayLabs/SmartRedis/pull/312))
- Leveraged Redis pipelining to improve performance of get, put, and
copy dataset methods
([PR311](https://github.com/CrayLabs/SmartRedis/pull/311))
- <Redis::set_model_multigpu>() will now upload the correct model to
all GPUs ([PR310](https://github.com/CrayLabs/SmartRedis/pull/310))
- RedisCluster::\_run_pipeline() will no longer unconditionally apply
a retry wait before returning
([PR309](https://github.com/CrayLabs/SmartRedis/pull/309))
- Expand code coverage to all four languages and make the CI/CD more
efficent ([PR308](https://github.com/CrayLabs/SmartRedis/pull/308))
- An internal flag was set incorrectly, it resulted in wrong key
prefixing when accessing (retrieving or querying) lists created in
ensembles ([PR306](https://github.com/CrayLabs/SmartRedis/pull/306))
- Corrected a variety of Doxygen errors and omissions in the API
documentation
([PR305](https://github.com/CrayLabs/SmartRedis/pull/305))
- Added throw documentation for exception handling in redis.h,
redisserver.h, rediscluster.h
([PR301](https://github.com/CrayLabs/SmartRedis/pull/301))
- Added error handling for a rare edge condition when setting scripts
and models
([PR300](https://github.com/CrayLabs/SmartRedis/pull/300))
- Added support to inspect the dimensions of a tensor via new
get_tensor_dims() method
([PR299](https://github.com/CrayLabs/SmartRedis/pull/299))
- The use_tensor_ensemble_prefix() API method no longer controls
whether datasets are prefixed. A new API method,
use_dataset_ensemble_prefix() now manages this.
([PR298](https://github.com/CrayLabs/SmartRedis/pull/298))
- Updated from redis-plus-plus v1.3.2 to v1.3.5
([PR296](https://github.com/CrayLabs/SmartRedis/pull/296))
- Updated from PyBind v2.6.2 to v2.10.3
([PR295](https://github.com/CrayLabs/SmartRedis/pull/295))
- Change documentation theme to sphinx_book_theme to match SmartSim
documentation theme and fix Python API doc string errors
([PR294](https://github.com/CrayLabs/SmartRedis/pull/294))
- Added print capability for Client and DataSet to give details
diagnostic information for debugging
([PR293](https://github.com/CrayLabs/SmartRedis/pull/293))
- Added support for retrieval of names and types of tensors and
metadata inside datasets
([PR291](https://github.com/CrayLabs/SmartRedis/pull/291))
- Added support for user-directed logging for Python clients via
{Client, Dataset, LogContext}.{log_data, log_warning, log_error}
methods ([PR289](https://github.com/CrayLabs/SmartRedis/pull/289))
- Added support for user-directed logging without a Client or Dataset
context to C and Fortran clients via \_string() methods
([PR288](https://github.com/CrayLabs/SmartRedis/pull/288))
- Added logging to capture transient errors that arise in the \_run()
and \_connect() methods of the Redis and RedisCluster classes
([PR287](https://github.com/CrayLabs/SmartRedis/pull/287))
- Tweak direct testing of Redis and RedisCluster classes
([PR286](https://github.com/CrayLabs/SmartRedis/pull/286))
- Resolve a disparity in the construction of Python client and
database classes
([PR285](https://github.com/CrayLabs/SmartRedis/pull/285))
- Fortran clients can now access error text and source location
([PR284](https://github.com/CrayLabs/SmartRedis/pull/284))
- Add exception location information from CPP code to Python
exceptions
([PR283](https://github.com/CrayLabs/SmartRedis/pull/283))
- Added client activity and manual logging for developer use
([PR281](https://github.com/CrayLabs/SmartRedis/pull/281))
- Fix thread pool error
([PR280](https://github.com/CrayLabs/SmartRedis/pull/280))
- Update library linking instructions and update Fortran tester build
process ([PR277](https://github.com/CrayLabs/SmartRedis/pull/277))
- Added [add_metadata_for_xarray]{.title-ref} and
[transform_to_xarray]{.title-ref} methods in
[DatasetConverter]{.title-ref} class for initial support with Xarray
([PR262](https://github.com/CrayLabs/SmartRedis/pull/262))
- Change Dockerfile to use Ubuntu 20.04 LTS image
([PR276](https://github.com/CrayLabs/SmartRedis/pull/276))
- Implemented support for Unix Domain Sockets, including
refactorization of server address code, test cases, and check-in
tests. ([PR252](https://github.com/CrayLabs/SmartRedis/pull/252))
- A new make target [make lib-with-fortran]{.title-ref} now compiles
the Fortran client and dataset into its own library which
applications can link against
([PR245](https://github.com/CrayLabs/SmartRedis/pull/245))

lists for pipelined retrieval of data, convenient support for multiple
GPUs, and the ability to delete scripts and models from the backend
database. It also introduces multithreaded execution for certain tasks
that span multiple shards of a clustered database, and it incorporates a
variety of internal improvements that will enhance the library going
forward.

Detailed Notes

- Implemented DataSet aggregation lists in all client languages, for
pipelined retrieval of data across clustered and non-clustered
backend databases.
([PR258](https://github.com/CrayLabs/SmartRedis/pull/258))
([PR257](https://github.com/CrayLabs/SmartRedis/pull/257))
([PR256](https://github.com/CrayLabs/SmartRedis/pull/256))
([PR248](https://github.com/CrayLabs/SmartRedis/pull/248)) New
commands are:
- append_to_list()
- delete_list()
- copy_list()
- rename_list()
- get_list_length()
- poll_list_length()
- poll_list_length_gte()
- poll_list_length_lte()
- get_datasets_from_list()
- get_dataset_list_range()
- use_list_ensemble_prefix()
- Implemented multithreaded execution for parallel dataset list
retrieval on clustered databases. The number of threads devoted for
this purpose is controlled by the new environment variable
SR_THERAD_COUNT. The value defaults to 4, but may be any positive
integer or special value zero, which will cause the SmartRedis
runtime to allocate one thread for each available hardware context.
([PR251](https://github.com/CrayLabs/SmartRedis/pull/251))
([PR246](https://github.com/CrayLabs/SmartRedis/pull/246))
- Augmented support for GPUs by implementing multi-GPU convenience
functions for all client languages.
([PR254](https://github.com/CrayLabs/SmartRedis/pull/254))
([PR250](https://github.com/CrayLabs/SmartRedis/pull/250))
([PR244](https://github.com/CrayLabs/SmartRedis/pull/244)) New
commands are:
- set_model_from_file_multigpu()
- set_model_multigpu()
- set_script_from_file_multigpu()
- set_script_multigpu()
- run_model_multigpu()
- run_script_multigpu()
- delete_model_multigpu()
- delete_script_multigpu()
- Added API calls for all clients to delete models and scripts from
the backend database.
([PR240](https://github.com/CrayLabs/SmartRedis/pull/240)) New
commands are:
- delete_script()
- delete_model()
- Updated the use of backend RedisAI API calls to discontinue use of
deprecated methods for model selection (AI.MODELSET) and execution
(AI.MODELRUN) in favor of current methods AI.MODELSTORE and
AI.MODELEXECUTE, respectively.
([PR234](https://github.com/CrayLabs/SmartRedis/pull/234))
- SmartRedis will no longer call the C runtime method srand() to
ensure that it does not interfere with random number generation in
client code. It now uses a separate instance of the C++ random
number generator.
([PR233](https://github.com/CrayLabs/SmartRedis/pull/233))
- Updated the way that the Fortran enum_kind type defined in the
fortran_c\_interop module is defined in order to better comply with
Fortran standard and not interfere with GCC 6.3.0.
([PR231](https://github.com/CrayLabs/SmartRedis/pull/231))
- Corrected the spelling of the word \"command\" in a few error
message strings.
([PR221](https://github.com/CrayLabs/SmartRedis/pull/221))
- SmartRedis now requires a CMake version 3.13 or later in order to
utilize the add_link_options CMake command.
([PR217](https://github.com/CrayLabs/SmartRedis/pull/217))
- Updated and improved the documentation of the SmartRedis library. In
particular, a new SmartRedis Integration Guide provides an
introduction to using the SmartRedis library and integrating it with
existing software.
([PR261](https://github.com/CrayLabs/SmartRedis/pull/261))
([PR260](https://github.com/CrayLabs/SmartRedis/pull/260))
([PR259](https://github.com/CrayLabs/SmartRedis/pull/259))
([SSPR214](https://github.com/CrayLabs/SmartSim/pull/214))
- Added clustered Redis testing to automated GitHub check-in testing.
([PR239](https://github.com/CrayLabs/SmartRedis/pull/239))
- Updated the SmartRedis internal API for building commands for the
backend database.
([PR223](https://github.com/CrayLabs/SmartRedis/pull/223)) This
change should not be visible to clients.
- The SmartRedis example code is now validated through the automated
GitHub checkin process. This will help ensure that the examples do
not fall out of date.
([PR220](https://github.com/CrayLabs/SmartRedis/pull/220))
- Added missing copyright statements to CMakeLists.txt and the
SmartRedis examples.
([PR219](https://github.com/CrayLabs/SmartRedis/pull/219))
- Updated the C++ test coverage to ensure that all test files are
properly executed when running \"make test\".
([PR218](https://github.com/CrayLabs/SmartRedis/pull/218))
- Fixed an internal naming conflict between a local variable and a
class member variable in the DataSet class.
([PR215](https://github.com/CrayLabs/SmartRedis/pull/215)) This
should not be visible to clients.
- Updated the internal documentation of methods in SmartRedis C++
classes with the override keyword to improve compliance with the
latest C++ standards.
([PR214](https://github.com/CrayLabs/SmartRedis/pull/214)) This
change should not be visible to clients.
- Renamed variables internally to more cleanly differentiate between
names that are given to clients for tensors, models, scripts,
datasets, etc., and the keys that are used when storing them in the
backend database.
([PR213](https://github.com/CrayLabs/SmartRedis/pull/213)) This
change should not be visible to clients.

Released on February 11, 2022

Description

- Improve error handling across all SmartRedis clients
([PR159](https://github.com/CrayLabs/SmartRedis/pull/159))
([PR191](https://github.com/CrayLabs/SmartRedis/pull/191))
([PR199](https://github.com/CrayLabs/SmartRedis/pull/199))
([PR205](https://github.com/CrayLabs/SmartRedis/pull/205))
([PR206](https://github.com/CrayLabs/SmartRedis/pull/206))
Includes changes to C and Fortran function prototypes that are
not backwards compatible. Includes changes to error class names
and enum type names that are not backwards compatible
- Add `poll_dataset` functionality to all SmartRedis clients
([PR184](https://github.com/CrayLabs/SmartRedis/pull/184))
Due to other breaking changes made in this release, applications
using methods other than `poll_dataset` to check for the
existence of a dataset should now use `poll_dataset`
- Add environment variables to control client connection and command
timeout behavior
([PR194](https://github.com/CrayLabs/SmartRedis/pull/194))
- Add AI.INFO command to retrieve statistics on scripts and models via
Python and C++ clients
([PR197](https://github.com/CrayLabs/SmartRedis/pull/197))
- Create a Dockerfile for SmartRedis
([PR180](https://github.com/CrayLabs/SmartRedis/pull/180))
- Update `redis-plus-plus` version to 1.3.2
([PR162](https://github.com/CrayLabs/SmartRedis/pull/162))
- Internal client performance and API improvements
([PR138](https://github.com/CrayLabs/SmartRedis/pull/138))
([PR141](https://github.com/CrayLabs/SmartRedis/pull/141))
([PR163](https://github.com/CrayLabs/SmartRedis/pull/163))
([PR203](https://github.com/CrayLabs/SmartRedis/pull/203))
- Expose Redis `FLUSHDB`, `CONFIG GET`, `CONFIG SET`, and `SAVE`
commands to the Python client
([PR139](https://github.com/CrayLabs/SmartRedis/pull/139))
([PR160](https://github.com/CrayLabs/SmartRedis/pull/160))
- Extend inverse CRC16 prefixing to all hash slots
([PR161](https://github.com/CrayLabs/SmartRedis/pull/161))
- Improve backend dataset representation to enable performance
optimization
([PR195](https://github.com/CrayLabs/SmartRedis/pull/195))
- Simplify SmartRedis build proccess
([PR189](https://github.com/CrayLabs/SmartRedis/pull/189))
- Fix zero-length array transfer in Fortran `convert_char_array_to_c`
([PR170](https://github.com/CrayLabs/SmartRedis/pull/170))
- Add continuous integration for all SmartRedis tests
([PR165](https://github.com/CrayLabs/SmartRedis/pull/165))
([PR173](https://github.com/CrayLabs/SmartRedis/pull/173))
([PR177](https://github.com/CrayLabs/SmartRedis/pull/177))
- Update SmartRedis docstrings
([PR200](https://github.com/CrayLabs/SmartRedis/pull/200))
([PR207](https://github.com/CrayLabs/SmartRedis/pull/207))
- Update SmartRedis documentation and examples
([PR202](https://github.com/CrayLabs/SmartRedis/pull/202))
([PR208](https://github.com/CrayLabs/SmartRedis/pull/208))
([PR210](https://github.com/CrayLabs/SmartRedis/pull/210))

Released on August, 5, 2021

Description

- Improved tensor memory management in the Python client
([PR70](https://github.com/CrayLabs/SmartRedis/pull/70))
- Improved metadata serialization and removed protobuf dependency
([PR61](https://github.com/CrayLabs/SmartRedis/pull/61))
- Added unit testing infrastructure for the C++ client
([PR96](https://github.com/CrayLabs/SmartRedis/pull/96))
- Improve command execution fault handling
([PR65](https://github.com/CrayLabs/SmartRedis/pull/65))
([PR97](https://github.com/CrayLabs/SmartRedis/pull/97))
([PR105](https://github.com/CrayLabs/SmartRedis/pull/105))
- Bug fixes ([PR52](https://github.com/CrayLabs/SmartRedis/pull/52))
([PR72](https://github.com/CrayLabs/SmartRedis/pull/72))
([PR76](https://github.com/CrayLabs/SmartRedis/pull/76))
([PR84](https://github.com/CrayLabs/SmartRedis/pull/84))
- Added copy, rename, and delete tensor and DataSet commands in the
Python client
([PR66](https://github.com/CrayLabs/SmartRedis/pull/66))
- Upgrade to RedisAI 1.2.3
([PR101](https://github.com/CrayLabs/SmartRedis/pull/101))
- Fortran and C interface improvements
([PR93](https://github.com/CrayLabs/SmartRedis/pull/93))
([PR94](https://github.com/CrayLabs/SmartRedis/pull/94))
([PR95](https://github.com/CrayLabs/SmartRedis/pull/95))
([PR99](https://github.com/CrayLabs/SmartRedis/pull/99))
- Add Redis INFO command execution to the Python client
([PR83](https://github.com/CrayLabs/SmartRedis/pull/83))
- Add Redis CLUSTER INFO command execution to the Python client
([PR105](https://github.com/CrayLabs/SmartRedis/pull/105))

Released on May 5, 2021

Description

- Compiled client library build and install update to remove
environment variables
([PR47](https://github.com/CrayLabs/SmartRedis/pull/47))
- Pip install for Python client
([PR45](https://github.com/CrayLabs/SmartRedis/pull/45))

Released on April 1, 2021

Description

- Initial 0.1.0 release of SmartRedis