Arcticdb

🚀 Features

- Remove python deps that are no longer needed (1005)
- New `row_range` argument on `read` and `ReadRequest` (864)
python
>>> from arcticdb import Arctic
>>> import pandas as pd
>>> df = pd.DataFrame({"col1": np.arange(10), "col2": np.arange(100, 110)}, index=np.arange(10))
>>> ac = Arctic("lmdb://test")
>>> lib = ac.get_library("test_lib", create_if_missing=True)
>>> lib.write("test_symbol", df)
>>> lib.read("test_symbol", row_range=(3,7)).data
col1 col2
3 3 103
4 4 104
5 5 105
6 6 106


🐛 Fixes

- Symbol list refactor (796)
- Fixed aggregation on sparse grouping columns (1068).
Depending on timestamps being accurate in the symbol list has proved to be troublesome. Instead, we should use the most recent version id known to a client as an indication of the client's view of the world at the time as symbol list entry is written. That way, we can identify and correct symbol list entries that refer to conflicting writes.

Notebooks
- Added AWS blockchain notebook (1040)
- Added AWS blockchain to docs index (1043)
- Add Snapshot + Equity Notebooks (1071)

<details>
<summary>Uncategorized</summary>

- 744 extend real storage tests to run with large lifelike data and all api methods (989)
- Update BSL table for 4.1 (1023)
- Centralise the pytest marks (1024)
- Document the S3 backends that we have tested against and "un-beta" LMDB on Windows (1016)
- Sparse aggregation (1007)
- Docs versioning (1008)
- set-default after deploy so that 'latest' alias can be created first (1029)
- build: Remove old Cython configuration and adaptation (1028)
- Docs workflow fixes. (1030)
- build: Replace `emilib` with `robin_hood` (995)
- hot-fix: Use previous build of libmongocxx to avoid missing symbols (1050)
- Remove unused C++ Wangle dep (1047)
- Bugfix 1055: Unflake test_read_batch_time_stamp (1058)
- Change tag format in docs build (1062)
- Snapshot notebook typos (1088)
- Update vcpkg dep (1091)
- Enhancement/732/processing unit ecs model (960)
- Add xfail to flaky tests (1087)
- Add a mechanism to extend storage transaction lifetime to lifetime of… (975)
- Tweak release docs (1019)
- Change tmpdir to tmp_path (1093)
- Remove unnecessary xfails (1097)
- Add checks to see whether we should be validating version entries during compaction (1099)
- 941 self hosted runners for ci (997)
- Make dependency of pymongo optional in running (1027)
- Add preliminary change for slowdown error test (1064)
- Add mutex to ensure only single thread at pybind->c++ layer (973)
- Issue 1017 Only warn if the "base" LMDB env is opened twice (1022)
- Fix run-cmake action (1034)
- Add a fallback to free GH runners, when there is a problem with the self-hosted ones (1063)
- fix: Empty column handling improvements (1049)
- Pin all our Github actions deps (1090)
</details>

---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging:

🚀 Features

- Remove useless python deps (1005)
- feat: Allow `row_range` to be treated as a clause (864)
python
>>> from arcticdb import Arctic
>>> import pandas as pd
>>> df = pd.DataFrame({"col1": np.arange(10), "col2": np.arange(100, 110)}, index=np.arange(10))
>>> ac = Arctic("lmdb://test")
>>> lib = ac.get_library("test_lib", create_if_missing=True)
>>> lib.write("test_symbol", df)
>>> lib.read("test_symbol", row_range=(3,7)).data
col1 col2
3 3 103
4 4 104
5 5 105
6 6 106


🐛 Fixes

- Symbol list refactor (796)
- Fixed aggregation on sparse grouping columns (1068).
Depending on timestamps being accurate in the symbol list has proved to be troublesome. Instead, we should use the most recent version id known to a client as an indication of the client's view of the world at the time as symbol list entry is written. That way, we can identify and correct symbol list entries that refer to conflicting writes.

Notebooks
- Added AWS blockchain notebook (1040)
- Added AWS blockchain to docs index (1043)
- Add Snapshot + Equity Notebooks (1071)

<details>
<summary>Uncategorized</summary>

- 744 extend real storage tests to run with large lifelike data and all api methods (989)
- Update BSL table for 4.1 (1023)
- Centralise the pytest marks (1024)
- Document the S3 backends that we have tested against and "un-beta" LMDB on Windows (1016)
- Sparse aggregation (1007)
- Docs versioning (1008)
- set-default after deploy so that 'latest' alias can be created first (1029)
- build: Remove old Cython configuration and adaptation (1028)
- Docs workflow fixes. (1030)
- build: Replace `emilib` with `robin_hood` (995)
- hot-fix: Use previous build of libmongocxx to avoid missing symbols (1050)
- Remove unused C++ Wangle dep (1047)
- Bugfix 1055: Unflake test_read_batch_time_stamp (1058)
- Change tag format in docs build (1062)
- Snapshot notebook typos (1088)
- Update vcpkg dep (1091)
- Enhancement/732/processing unit ecs model (960)
- Add xfail to flaky tests (1087)
- Add a mechanism to extend storage transaction lifetime to lifetime of… (975)
- Tweak release docs (1019)
- Change tmpdir to tmp_path (1093)
- Remove unnecessary xfails (1097)
- Add checks to see whether we should be validating version entries during compaction (1099)
- 941 self hosted runners for ci (997)
- Make dependency of pymongo optional in running (1027)
- Add preliminary change for slowdown error test (1064)
- Add mutex to ensure only single thread at pybind->c++ layer (973)
- Issue 1017 Only warn if the "base" LMDB env is opened twice (1022)
- Fix run-cmake action (1034)
- Add a fallback to free GH runners, when there is a problem with the self-hosted ones (1063)
- fix: Empty column handling improvements (1049)
- Pin all our Github actions deps (1090)
</details>

---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging:

⭐ New APIs

In-memory Backend

You can now open ArcticDB with an in-memory backend,


from arcticdb import Arctic
ac = Arctic("mem://")
ac.create_library("test")
assert ac.list_libraries() == ["test"]
Create libraries as normal. Each `Arctic` object manages its own in-memory storage, so the lifetime
of your libraries and data is the same as the lifetime of the `Arctic` instance that owns them.

ac2 = Arctic("mem://")
assert ac2.list_libraries() == [] ac2 is backed by different memory to ac so the "test" library is not returned


Query Builder

We now support a new "count" aggregator. You can invoke it with:


q = QueryBuilder()
q = q.groupby("grouping_column").agg({"a": "count"})


⚠️ Breaking and API Changes

LMDB Backend

This release includes a fix for issue 850 : Ensure that LMDB libraries are readable after being moved to a different location. **The fix means that LMDB libraries created with `arcticdb>=4.1.0` will not be readable by older clients and those clients must update**.

This is because the fix stops us from serializing the LMDB library path (instead we always prefer the one in the Arctic URI), but older clients still expect to see the LMDB path serialized. Older clients reading a new LMDB library will in fact ignore the path passed in to the `Arctic` constructor and instead read the current working directory.

When you exceed the LMDB map size, we now raise a custom exception `arcticdb.exceptions.LmdbMapFullError` that explains how to re-open LMDB with a larger map size, whereas previously we raised a less helpful `arcticdb.exceptions.InternalException`.

🚀 Features

- Support `count` aggregator with groupby (948)
- Warning for LMDB when two Arctic instances open over the same storage (1000)
- Small LMDB Fixes: 2GiB map size for Windows, Validation before delete (918)
- Custom exception when LMDB map is full (1006)
- Memory backed API (860)
- Allow ampersand in symbol names (900)
- Add querybuilder notebook demo into the docs (875)
- Extended testing against real cloud storages (789)
- ASV Benchmarking published [here](https://man-group.github.io/ArcticDB/) (#962 970)
- Preparatory work for RocksDB backend (945)

🐛 Fixes

- Fix LZ4 decoding error issues that occurred with a mix of empty and non-empty columns (964)
- Performance improvement for `read_batch` when called with many symbols (4-5x improvement) (870)
- Convert semimap to switch which has resolved some segmentation fault issues (912)
- Upgrade cUrl to 8.4.0 (977)
- Cache open libraries in the LibraryManager (990)
- Enhancement 914: Improve error messaging when string column encoding fails due to the presence of a non-string object (933)
- Fix storage lock mutex implementation (966)
- Make azure sdk stick to winhttp if possible (851)
- Extra update checks (539)
- maint: Indicate the non-support of PyArrow (882)
- Add pymongo to the list of install dependencies (891)
- conda-build: Run python tests for macos-latest (873)
- fix: Change comparison in `test_hypothesis_{sum,mean}_agg` (931)


<details>
<summary>Uncategorized</summary>

- Fixes IFNDR issue due to mismatching inline/non-inline functions depending on the translation unit (fixes 943) (949)
- docs: Post 4.0.0 release documentation (967)
- Remove Black and pre-commit setup (972)
- Improve string writing performance (969)
- Skip docs in build.yml (984)
- Move api docs from sphinx to mkdocs (897)
- Document support for Mac on intel (982)
- Update PAT for publish to master (988)
- Fix ccache's non-existence exiting the workflow (996)
- Refactor get_descriptions lib methods to be more consistent (994)
- Fail docs build on sphix failure (883)
- docs: Better document publishing release candidates on conda-forge (901)
- conda-build: Unpin some dependencies (888)
- docs: Reword conda-forge section mentioning `libevent-2.1.10` (905)
- Update readme to reflect supported/beta status of Windows PyPi/MacOS conda-forge builds (907)
- Skip flaky Mac test (924)
- docs: Improve section "Building using mamba and conda-forge" (917)
- fix interface in ManualClockVersionStore getter (925)
- docs: Add high-level documentation of abstractions (628)
- Update storage compatibility (916)
- Update copyright notice (939)
- Use new `get_library` argument in ArcticDB_demo_lmdb.ipynb (930)
</details>

---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging:

🚀 Features
- Better backend storage retryable error handling and error message printout 1365
🐛 Fixes
- Stop allowing snapshotting tombstoned versions 1280
- Add a way to enable/disable silencing of errors when deleting a library 1271
<details>


</details>

---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging:

This is a patch release to version 4.0 that backports some changes from master.

🚀 Features

- Add preliminary change for slowdown error test (1070)

🐛 Fixes

- Empty column handling improvements (1079)
- Use previous build of libmongocxx to avoid missing symbols (1083)
- Remove docs publish step so we don't overwrite the docs (1025)
- Remove Black and pre-commit setup (1085)




<details>

</details>

---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging:

This is a patch release to version `4.0` that backports some changes from master.
🐛 Fixes
Bugfix for a deadlock issue when using Python multithreading and `batch_read` (https://github.com/man-group/ArcticDB/pull/1021)

---
> The wheels are on [Pypi](https://pypi.org/project/arcticdb/). Below are for debugging: