Arcticdb

🐛 Fixes

- Fix segmentation fault related to Ray package (by reverting to C++ 17 build) 1542
- Fix get_description date range after using delete_data_in_range 1526
- Fix tail segfault with empty data 1543

---
Please see [v4.4.1](https://github.com/man-group/ArcticDB/releases/tag/v4.4.1) release notes for the previous changes

🐛 Fixes
- Bug fix in Azure Storage, clear `to_delete` list to allow deletion of more than 256 keys in a batch (1482)

---
Please see [v4.4.0](https://github.com/man-group/ArcticDB/releases/tag/v4.4.0) release notes for the main changes from the last release.

🚀 Features

* Prevent writing empty types by default (gives compatibility with v1.6.2 readers) in https://github.com/man-group/ArcticDB/pull/1440
* Improved resilience to external (out of order) replication in https://github.com/man-group/ArcticDB/pull/1355
* Support modifying library options, and introduce enterprise library options in https://github.com/man-group/ArcticDB/pull/1457

You can now modify library options on an existing Arctic library:


from arcticdb.options import LibraryOptions

ac: Arctic = ...
lib = ac.create_library("lib")
ac.modify_library_option(lib, ModifiableLibraryOption.ROWS_PER_SEGMENT, int(1e6))


See `arcticdb/options.py` for a description of the modifiable options.

🐛 Fixes

* Bugfix 902: Cannot filter on nans and nones in string and float columns https://github.com/man-group/ArcticDB/pull/1276
* Bugfix: Empty type in https://github.com/man-group/ArcticDB/pull/1227
* Bugfix 1334: optimise version ref key access https://github.com/man-group/ArcticDB/pull/1345
* Fix version map cache invalidation policies in https://github.com/man-group/ArcticDB/pull/1351
* Fix empty column default type in https://github.com/man-group/ArcticDB/pull/1378
* Bugfix 1388: Correctly check whether we are in ec2 in https://github.com/man-group/ArcticDB/pull/1415
* Bugfix 1423: Raise a meaningful error message when trying to use `QueryBuilder` with sparse data in https://github.com/man-group/ArcticDB/pull/1435
* Bug Fix Windows: Remove lmdb files when `delete_library` is called in https://github.com/man-group/ArcticDB/pull/1437
* Bugfix 1256: reject parallel appends to unsorted data in https://github.com/man-group/ArcticDB/pull/1442
* Bugfix 1209: Consistently return metadata from write, append, update, write_metadata, and batch versions thereof in https://github.com/man-group/ArcticDB/pull/1444
* Bugfix/935/match pandas behviour when aggregating columns with nans in https://github.com/man-group/ArcticDB/pull/1450
* LZ4 decoding empty type: move the segment buffer forward by the compressed data size for empty types in https://github.com/man-group/ArcticDB/pull/1463
* Bugfix 1268: swap out xxhash for grouping in https://github.com/man-group/ArcticDB/pull/1416
* Bugfix 1260: allow broader range of numeric type promotion with dynamic schema https://github.com/man-group/ArcticDB/pull/1426

:floppy_disk: Storage Exception Normalization

We have made storage-related exceptions uniform across different backend storage platforms, despite the fact that the underlying behaviour varies.

* 447 LMDB Exceptions Normalization in https://github.com/man-group/ArcticDB/pull/1285
* 447 Memory Storage Exception Normalization in https://github.com/man-group/ArcticDB/pull/1297
* Adds a MockS3Client which can simulate s3 failures in https://github.com/man-group/ArcticDB/pull/1281
* 447 S3 Storage Exceptions Normalization in https://github.com/man-group/ArcticDB/pull/1304
* Add a `MockAzureClient` which can simulate azure failures in https://github.com/man-group/ArcticDB/pull/1331
* 447 Azure Storage Exceptions Normalization in https://github.com/man-group/ArcticDB/pull/1344
* 447 Exception normalization for RocksDB in https://github.com/man-group/ArcticDB/pull/1360
* Refactor: Move mongo client errors handling into mongo_storage.cpp before normalization in https://github.com/man-group/ArcticDB/pull/1383
* 447 Add a `MockMongoClient` which can simulate mongo failures in https://github.com/man-group/ArcticDB/pull/1395
* 447 Mongo Exceptions Normalization in https://github.com/man-group/ArcticDB/pull/1411
* LMDB Exception Normalization with mock client in https://github.com/man-group/ArcticDB/pull/1414

<details>
<summary>Uncategorized</summary>

* Prevent writing empty types by default (gives compatibility with v1.6.2 readers) in https://github.com/man-group/ArcticDB/pull/1440
* Add a way to enable/disable silencing of errors when deleting a library in https://github.com/man-group/ArcticDB/pull/1271
* Feature flag to use WinInet client not WinHttp in https://github.com/man-group/ArcticDB/pull/1284
* Abstract an S3ClientWrapper out of details-inl.hpp in https://github.com/man-group/ArcticDB/pull/1274
* GitHub Workflows: Make can_merge run for all files to allow any docs changes to be mergeable. in https://github.com/man-group/ArcticDB/pull/1292
* Fix benchmarks in https://github.com/man-group/ArcticDB/pull/1286
* Reduce the hashes that we use to benchmark against in https://github.com/man-group/ArcticDB/pull/1303
* Refactor 1278: Column data dense forward iterator in https://github.com/man-group/ArcticDB/pull/1301
* Add __contains__ to Arctic class, to support lib in arctic in https://github.com/man-group/ArcticDB/pull/1309
* Update BSL table for v4.3.0 in https://github.com/man-group/ArcticDB/pull/1282
* Utility to analyze the size of various key types in a library in https://github.com/man-group/ArcticDB/pull/1291
* Do not compile wheel build on EC2 in https://github.com/man-group/ArcticDB/pull/1318
* Docs: For top level imports use `arcticdb.object` instead of using full path to `object` in https://github.com/man-group/ArcticDB/pull/1323
* conda-build: Adaptations for folly in https://github.com/man-group/ArcticDB/pull/1320
* Align docstring with the behaviour in https://github.com/man-group/ArcticDB/pull/1243
* Added 142 new tests for empty/missing operations in https://github.com/man-group/ArcticDB/pull/1319
* Abstract AzureClientWrapper out of azure_storage.cpp in https://github.com/man-group/ArcticDB/pull/1315
* Stop using ec2 runners in the conda+linux workflow in https://github.com/man-group/ArcticDB/pull/1302
* Adds python tests which simulate s3 storage exceptions in https://github.com/man-group/ArcticDB/pull/1330
* S3 local delete failure raising meaningful error in https://github.com/man-group/ArcticDB/pull/1329
* Fix dynamic strings append to fixed strings issue in https://github.com/man-group/ArcticDB/pull/1346
* Minor improvement on analysis flow in https://github.com/man-group/ArcticDB/pull/1290
* Build time improvements in https://github.com/man-group/ArcticDB/pull/1263
* build: Disable compilers' extensions in https://github.com/man-group/ArcticDB/pull/1335
* maint: Fully specify `fmt::format_to` in https://github.com/man-group/ArcticDB/pull/1333
* Fixed pd_delete_replace + added single tests in https://github.com/man-group/ArcticDB/pull/1342
* Use type-deduced functor for all column iterating functions in https://github.com/man-group/ArcticDB/pull/1347
* Disable ec2 runners on PR in https://github.com/man-group/ArcticDB/pull/1357
* Use 14.39 toolset in https://github.com/man-group/ArcticDB/pull/1359
* Removed pytest dependency from arcticdb in https://github.com/man-group/ArcticDB/pull/1350
* build: Use C++20 in https://github.com/man-group/ArcticDB/pull/1332
* mark test_symbol_list_parallel_stress_with_delete as flaky in https://github.com/man-group/ArcticDB/pull/1368
* Docs: Increase CSS max-wdith and build docs from a branch in https://github.com/man-group/ArcticDB/pull/1363
* Print error msg in ExponentialBackoff exception in https://github.com/man-group/ArcticDB/pull/1365
* Disable missing key warnings when expected in https://github.com/man-group/ArcticDB/pull/1379
* Remove pin on civetweb in https://github.com/man-group/ArcticDB/pull/1380
* Multiple segments within the same block: storage and library refactoring in https://github.com/man-group/ArcticDB/pull/1307
* Not allowing snapshotting tombstoned versions in https://github.com/man-group/ArcticDB/pull/1280
* Clarify Intel/AMD build support in https://github.com/man-group/ArcticDB/pull/1389
* Fix debug formatting in https://github.com/man-group/ArcticDB/pull/1397
* maint: Replace `robin_hood` with `unordered_dense` in https://github.com/man-group/ArcticDB/pull/1390
* Test benchmarking improvements in https://github.com/man-group/ArcticDB/pull/1326
* maint: Remove dependency on some elements of folly in https://github.com/man-group/ArcticDB/pull/1370
* Allow different testing dependency version in pipeline in https://github.com/man-group/ArcticDB/pull/1410
* Add metadata extraction functions in library_tool in https://github.com/man-group/ArcticDB/pull/1375
* Add a global timeout for pytests in https://github.com/man-group/ArcticDB/pull/1381
* Set upper bound for supported protobuf version in https://github.com/man-group/ArcticDB/pull/1421
* 1 year and 1k stars readme banner in https://github.com/man-group/ArcticDB/pull/1425
* read_batch set include_deleted to false by default when reading a version in https://github.com/man-group/ArcticDB/pull/1419
* build: Update to fmt 10 in https://github.com/man-group/ArcticDB/pull/1427
* Make changes for prometheus metrics in https://github.com/man-group/ArcticDB/pull/1418
* maint: Replace use of folly getCurrentThreadId with STL in https://github.com/man-group/ArcticDB/pull/1417
* maint: Ignore the diff of 1263 in https://github.com/man-group/ArcticDB/pull/1340
* Roll back vcpkg version to fix failing abseil build in https://github.com/man-group/ArcticDB/pull/1436
* maint: Remove use of `folly/portability/PThread.h` in https://github.com/man-group/ArcticDB/pull/1447
* maint: Remove use of folly/system/ThreadName.h in https://github.com/man-group/ArcticDB/pull/1446
* Better error messaging around pickling in https://github.com/man-group/ArcticDB/pull/1451
* Update analysis_workflow.yml in https://github.com/man-group/ArcticDB/pull/1455
* Adding update, append and delete asv benchmarks in https://github.com/man-group/ArcticDB/pull/1434
* Support generators for metadata vectors again in https://github.com/man-group/ArcticDB/pull/1456
* Remove the brotli dep in https://github.com/man-group/ArcticDB/pull/1458
</details>

---
> The wheels are on [PyPI](https://pypi.org/project/arcticdb/). Below are for debugging:

🐛 Fixes
- Fix regression in round-tripping empty type for dynamic schema (1313)

---
> The wheels are on [PyPI](https://pypi.org/project/arcticdb/). Below are for debugging:

**Version 4.3.0 was pulled from PyPi and Conda Forge due to a regression. We no longer provide builds for 4.3.0.
Regression is fixed in [4.3.1](https://github.com/man-group/ArcticDB/releases/tag/v4.3.1) release. Please use 4.3.1 instead.**

🚀 Features

- Exposes existing regex filter in `lib.list_symbols` (1123)
python
>>> from arcticdb import Arctic
>>> import pandas as pd
>>> ac.create_library("test")
>>> lib = ac["test"]
>>> lib.write("sym0", pd.DataFrame())
>>> lib.write("sym1", pd.DataFrame())
>>> lib.list_symbols()
['sym0', 'sym1']
>>> lib.list_symbols(regex="1$")
['sym1']

- Introduce jitter in symbol list compaction threshold (1174)
- Sorting speed improvements in SegmentInMemory (1181)
- Reduce log level from warn to debug for "Failed to find segment for key" message where appropriate (1130)
- Speed up writes by parellising `aggregator_set_data` over data segments (1065)
- Support sortedness checks and maintenance with parallel writes and appends (1251)
- 1014 Introduce storage fixtures to easily test ArcticDB against various storage backends. See `arcticdb.storage_fixtures` package. (1054)

🐛 Fixes

- Release the symbol list's storage lock if it has existed for longer than its TTL (1134)
- Ensure that the version chain is always updated atomically (1104)
- Return empty `pd.DataFrame` with `MultiIndex` if originally provided (1126)
- conda-build: Explicitly depend on openssl and libcurl (1244)
- Reintroduce attrs as a runtime dependency (1272)
- Speedup reading wide dataframes that have no empty columns (1225)
- Bugfix 1046: Prevent appending/updating numeric columns with non-identical types with static schema (1205)
- Bugfix 1173: Correctly apply sortedness checks when calling update with date_range argument (1238)
- Fix non-deterministic hashing in Linux conda builds (1261)
- Improve date range returned by get info for unordered and range indexed dataframes (1241)
- Bugfix 1248 and 1249: compact_incomplete reject incomplete segments that overlap each other, or existing segments in the case of append (1255)
- Detailed error in case of S3's libcurl network failure (1265)

<details>
<summary>Uncategorized</summary>

- [Aggregation tests] Replace `non_zero_numeric_type_strategies` with `numeric_type_strategies` (968)
- Fixes reuse_name for azure storage 1061 (1115)
- small getting-started-docs tweaks (1103)
- Improve fixture reliability (1116)
- maint: Define `arcticdb::proto::logger` in `log.hpp` (1117)
- maint: Remove unneeded includes (1113)
- [Column] Move some definitions to cpp file (1100)
- maint: Move implementations in `memory_segment_impl.hpp` to `memory_segment_impl.cpp` (1092)
- Update git blame file (1118)
- Flaky test hypothesis mean agg (496) (1125)
- Use same region for S3 and EC2 to avoid data transfer costs (1128)
- build: Remove attrs from the dependencies (1135)
- Only build on pull request events (1127)
- More fixture robustness improvements (1132)
- Remove releasing docs as they are now in GitHub wiki (1136)
- Update README.md (1141)
- Remove test parellism, and speed up test bottleneck (1143)
- Fix support for shared/unique S3 prefixes (1140)
- maint: Remove headers in `types.hpp` (1121)
- Skip flaky pytests which check log messages (1161)
- Update README.md (1156)
- Refactor: Move DataError method implementaitons into cpp (1155)
- Update .git-blame-ignore-revs for DataError implementation move (1165)
- Add MSVC 2022 preset. Tweak MSVC CMake settings. (1133)
- Build-time improvments: allocator.hpp, log.hpp, buffer.hpp (1152)
- Fix publish.yml workflow (1167)
- README - put third party tools in alphabetical order (1172)
- Fix persistent tests (1147)
- Introduce sorting and merging google benchmarks (1138)
- Skip array type tests due to occsional segfaults (1187)
- build: Remove some adherence to folly (1144)
- Add equity options notebook + data (1178)
- maint: Ignore some references (1190)
- Added equity options notebook to index (1193)
- Use vcpkg for gbench (1189)
- Forward port internal PR 1082 (1180)
- Bugfix 1191: Propagate storage failures in version map batch methods to calling code (1194)
- Link against python explicitly in order to make MSVS builds work (1192)
- Final version of equity opts notebook (1196)
- Bugfix 1182: Unskip test that is no longer flaky (1197)
- Docs that StorageFailureSimulator is not used in all stores (1203)
- Clean and reorganize `OffsetString` and `StringPool` (1137)
- build: Do not depend on protobuf-lite (1212)
- docs: Fix documentation links (1038)
- Fix recurse_segment forward declaration to match the signature of its implementation (1217)
- Update git blame file for `OffsetString` and `StringPool` implementation move (1211)
- Add frequently used items at the top level of arcticdb (1219)
- Switch from `arcticdb` to `adb` in the demo Notebooks (1228)
- Add a way to handle non-string values for index names (1170)
- Pass unmodified argument by const& to FieldCollection::add_field (1234)
- Switch from `arcticdb` to `adb` in python docstrings (1236)
- Remove obsolete test log level environment variable (1231)
- Update incorrect docs for `validate_index` (1233)
- Bugfix 1207: Use pandas.Timestamp.max - 1 day in test_read_ts. Remove pointless snapshot. Improve error message when index key reading fails. (1235)
- Bugfix invalid library name (1206)
- Enhancement/1253/skip temporary allocation when decoding dynamic schema columns (1259)
- Expose headers for consumers via arcticdb_core_static (1257)
- Update WarnVersionTypeNotHandled::warn() warning message (1273)
- Update README correcting spelling (1275)
- build: Adapt protobuf compilation (1199)
- Enable skipped test_partial_write_hashed (1215)
</details>

---
> The wheels are on [PyPI](https://pypi.org/project/arcticdb/). Below are for debugging:

This is a patch release to version [4.2.0](https://github.com/man-group/ArcticDB/releases/tag/v4.2.0) which fixes Issue #1157 regarding the [`defragment_symbol_data`](https://docs.arcticdb.io/latest/api/library/#arcticdb.version_store.library.Library.defragment_symbol_data) method.

🐛 Fixes

- Defragmenting a symbol no longer invalidates previous versions (https://github.com/man-group/ArcticDB/pull/1163)

---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging: