Arcticdb

🚀 Features

✅ `defragment_symbol_data`  method added (180)
* This will defragment fragmented symbols. Fragmented symbols are typically caused by frequent small appends (e.g. 1 row every hour) and result in sub-optimal read performance. Reading defragmented symbols can result in significantly improved performance compared to a more fragmented equivalent!

✅ Write API for column-level statistics added. ([253](https://github.com/man-group/ArcticDB/pull/253))
* The structures created using this API will later be used to improve the performance of queries in later versions. As of version 1.2.0 however, the structures are not consumed at query time.

✅ Configuration variables set via environment variables are now case insensitive ([364](https://github.com/man-group/ArcticDB/pull/364))

🐛 Bug fixes

✅ `add_to_snapshot`  and `remove_from_snapshot`  no longer leave unreachable data on disk if a symbol-version pair being removed from a snapshot is the last reference to this version ([253](https://github.com/man-group/ArcticDB/pull/253))

✅ Reads with `QueryBuilder` parameters that contain a `groupby/aggregation`  clauses now only fetch data segments containing columns relevant to the query ([253](https://github.com/man-group/ArcticDB/pull/253))

**Full Changelog**: https://github.com/man-group/ArcticDB/compare/v1.1.0...v1.2.0
---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging:

New Platforms
:white_check_mark: Version 1.1 includes the first official release of the Windows version of ArcticDB! <img src="https://upload.wikimedia.org/wikipedia/commons/4/44/Microsoft_logo.svg" width="16" />
Versions 1.1 includes the following caveats:
* Please have a recent [Visual C++ Redistributable for Visual Studio 2015-2022](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed.
* Windows does not support fixed-width NumPy strings and so will not be able to write fixed-width strings contained within a NumPy array.
* Writing pickled data can fail on machines with limited memory, see 340 for details and a workaround.
* The LMDB store on Windows pre-allocates the max storage size on disk, so we limited the size to 128MB. We will provide an option to increase the size in the future (229).

:white_check_mark: Version 1.1 now supports Python 3.11!

New Features
:white_check_mark: batch_get_descriptor method added (219)
:white_check_mark: batch_get_metadata method added (220)
Both batch methods do not achieve optimal parallelism - we are aiming to address in 1.2.

Changes
:white_check_mark: 1.1 pins to Pandas < 2.0 whilst we address a few remaining compatibility issues :panda_face: (237)
:white_check_mark: 1.1 supports Protobuf V3 as well as V4
:white_check_mark: Out-of-order data updates now raises exceptions should they prevent future indexed reads (203)
:white_check_mark: PCRE is now statically linked (321)
:white_check_mark: `stream id` which is a confusing term has been renamed to `symbol` in error messages and should be better understood (311)
:white_check_mark: Pickled data supports up-to 4GB (260)
:white_check_mark: Debugging toolbox added (209)
:white_check_mark: Introduced a base exception type `arcticdb.exceptions.ArcticException`. Exceptions are exposed in `arcticdb.exceptions` module with the following hierarchy:


RuntimeError
└-- ArcticException
|-- ArcticNativeNotYetImplemented
|-- DuplicateKeyException
|-- MissingDataException
|-- NoDataFoundException
|-- NoSuchVersionException
|-- NormalizationException
|-- PermissionException
|-- SchemaException
|-- SortingException
| └-- UnsortedDataException
|-- StorageException
|-- StreamDescriptorMismatch
└-- InternalException


Next version:

NEW FEATURE: defragment_symbol_data method added (180)

> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging: