Lakefs-spec

⭐ Highlights

Changes the lakeFS-spec transaction to operate on short-lived ephemeral branches (258).

Transaction versioning operations are implicitly scoped to the given repository and transaction branch, so you do not need to manually specify these arguments anymore:

python
will automatically merge into `main` upon successful completion of `tx`.
with fs.transaction("my-repo", "main") as tx:
fs.put("my-file.txt", f"my-repo/{tx.branch.id}/my-file.txt")
tx.commit(message="Add my file")


🚨 Breaking changes

The usage of `LakeFSFileSystem.transaction` was changed. It is now necessary to call it with a repository name to properly initialize the transaction.

What's Changed
* Bump lakeFS service container version to v1.7.0 by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/249
* Replace `fs._open()` with proper `fs.open()` implementation by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/250
* Add integration tests for third-party data science libraries by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/251
* (docs): Add lakeFS-spec and lakeFS-SDK transaction differences by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/252
* Update project metadata by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/256
* Split lakefs-spec test job into multiple jobs by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/261
* Introduce new transaction model based on ephemeral branches by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/258


**Full Changelog**: https://github.com/aai-institute/lakefs-spec/compare/v0.6.1...v0.7.0

⭐ Highlights

This release marks the completion of the lakeFS SDK wrapper migration. All available functionality is now expressed in terms of abstractions from the new `lakefs` Python package.

As a particular consequence, the `LakeFSFile` was removed in favor of lakeFS' own file-like object APIs. For simple file interactions through the file system only (including transactions), this will not result in breaking changes.

What's Changed
* Slim down the bug report and feature request templates by janwillemkl in https://github.com/aai-institute/lakefs-spec/pull/235
* Update dev requirements, fix lakefs v0.2.0 regression by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/239
* Implement more APIs in terms of `lakefs` abstractions by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/240
* Fix `fs.rm()` for the recursive case by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/243
* Add test coverage for commits to repo information tests by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/244
* Remove Placeholder from interface typing, use new ReferenceType hint by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/245
* Add lakeFS package versions to pytest header by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/246
* Update bug-report.yaml by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/247
* Simplify revision parsing in the tutorial notebook by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/248


**Full Changelog**: https://github.com/aai-institute/lakefs-spec/compare/v0.6.0...v0.6.1

⭐ Highlights

- lakeFS-spec is now using the newly release [`lakefs` high-level Python SDK](https://lakefs.io/blog/improved-python-experience/) under the hood (see #231)
- the project now has a logo in all the relevant places 🎨

🚨 Breaking Changes

- Loading `lakectl.yaml` configuration from non-default paths is no longer supported (in line with the `lakefs` Python SDK)

What's Changed
* Correctly set docs version number for release workflow by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/228
* Docs: Update project highlights by janwillemkl in https://github.com/aai-institute/lakefs-spec/pull/229
* Several small test improvements by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/230
* Migrate the package to use the new `lakefs` SDK wrapper library by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/231
* Implement deletes using the lakeFS branch API by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/236
* Add custom lakeFS-spec logo/favicon by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/238

New Contributors
* janwillemkl made their first contribution in https://github.com/aai-institute/lakefs-spec/pull/229

**Full Changelog**: https://github.com/aai-institute/lakefs-spec/compare/v0.5.0...v0.6.0

⭐ Highlights

The v0.5.0 release of lakeFS-spec simplifies working with deferred placeholder entities returned by versioning operations as part of a transaction. These placeholders now work as transparent proxy objects, meaning they expose the same interface as the class they are wrapping (no more `Placeholder.unwrap()` necessary! 🙌🏻 )

The user documentation has been completed, especially the remaining missing pieces of the API reference.

🚨 Breaking Changes

- `LakeFSFileSystem.ls()` no longer defaults to recursive listing, in line with other fsspec implementations. If you require recursive directory listings, you can pass the `recursive=True` argument to `ls()` or consider the `find()`/`glob()`/`walk()` operations.

What's Changed
* Allow Path like inputs in public LakeFSFileSystem methods by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/182
* Create API docstrings for the `LakeFSFileSystem` and `LakeFSFile` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/181
* Docs: Allow Jupytext notebooks for tutorials by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/183
* Add API docs for all remaining smaller modules by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/184
* Record test coverage on fsspec package [ENG-196] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/188
* Modify error translator to raise more informative errors by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/190
* Idempotency of demo notebook by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/189
* Drop `requirements.txt` from package lock files by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/194
* Docs: Fix typos in tx user guide by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/197
* Bump lakeFS version for testing environment to v1.3.0 by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/199
* Docs: Add call to action for early adopters to README and landing page [ENG-186] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/196
* `ls`: Make non-recursive by default, correctly update dircache by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/198
* Introduce issue templates by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/202
* Revert "Introduce issue templates" by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/204
* Improve test coverage on lakefs abstract base classes by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/201
* Introduce templates - eliminate yaml errrors by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/205
* Validate with docstrings with `pydoclint` by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/203
* Add lakeFS server version guard to `LakeFSFilesystem.touch()` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/214
* Overload `LakeFSFile.__del__` to pass statement by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/219
* Remove bug ftr prefix issue templates by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/217
* Add overloads for `LakeFSFileSystem.ls()` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/221
* Docs: Client-side caching for Open-Meteo API in tutorial notebook by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/220
* Run `pre-commit autoupdate` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/223
* Remove dunder methods from API docs by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/222
* Create template pull-request by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/206
* Correctly handle dircache for recursive `ls` calls by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/224
* Simplify transaction `Placeholder` class by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/225
* Fix transaction doc on `Placeholder` values by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/226

New Contributors
* Maciej818 made their first contribution in https://github.com/aai-institute/lakefs-spec/pull/202

**Full Changelog**: https://github.com/aai-institute/lakefs-spec/compare/v0.4.0...v0.5.0

The v0.4.0 release is dominated by the complete overhaul of the user documentation (check it out at https://lakefs-spec.org/) 🚀

The documentation now features a quickstart guide, a full user guide, and an improved tutorial on how to use lakeFS-spec in a data science project. Feedback on and contributions to the documentation are highly appreciated!

🚨 Breaking Changes

- The `LakeFSFileSystem.scope` context manager has been removed from the API in https://github.com/aai-institute/lakefs-spec/pull/175, since its functionality has been superseded

What's Changed
* Make `pyyaml` dependency mandatory by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/143
* Restructure README [ENG-122] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/129
* Run `pre-commit autoupdate`, delete unnecessary gitkeep file by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/144
* Remove file upload performance warning by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/146
* Improve lakeFS transactions by using deque, placeholders for commit SHAs by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/140
* Move demo lakectl to tutorials folder by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/148
* Integrate `.ipynb` to CI and move to docs by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/119
* Quickstart guide [ENG-146] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/142
* Delete stale black config and comments, fix error handling in blockstore puts by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/150
* Add lakeFS file system configuration doc by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/151
* Reword Python environment installation guide by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/152
* Add `exist_ok` flag to all resource-creating client helpers by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/154
* Docs: Landing page [ENG-147] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/155
* Docs: Branding fixes by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/157
* Improvements to `ls` implementation by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/166
* Add file system usage documentation [ENG-148] by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/156
* Docs: Add third-party integrations user guide by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/167
* Allow leading `lakefs://` scheme in input paths by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/169
* Change `fs.info()` logic to recognize non-slash-terminated directories by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/160
* Add transaction usage document [ENG-150] by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/171
* Documentation cleanup [ENG-198] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/170
* Docs: Post-hoc cleanup of third-party integrations user guide [ENG-149] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/172
* Change certain aspects of the data science tutorial notebook by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/158
* Add doc-strings to the client_helpers by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/168
* Add HTTP409 check in `create_repository` for forward compatibility by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/177
* Docs: Render pre-release banner for `main` by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/176
* Adapt `create_tag` client helper to reflect tag immutability by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/179
* Remove `_strip_protocol` overload and usages by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/174
* Remove `LakeFSFileSystem.scope` context manager by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/175
* Add depagination helper by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/180


**Full Changelog**: https://github.com/aai-institute/lakefs-spec/compare/v0.3.0...v0.4.0

This release introduces transactions as the new way of carrying out versioning operations during file transfers. They supersede hooks, which were removed from the package.

Type hints were added to all public `lakefs-spec` interfaces, as well as `fsspec` callback support to file transfers.

This release also marks the transition to the `lakefs-sdk` package as the way of interfacing with the lakeFS server from Python. As such, older lakeFS server versions might not be supported anymore.

What's Changed
* Fix unbound local error by moving up definition of `exists` boolean by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/111
* Change lakeFS package to `lakefs-sdk` in place of `lakefs-client` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/107
* Add README on hack/ folder resources by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/115
* Implement Demo Feedback from [ENG-111] by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/117
* Run `pre-commit autoupdate` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/118
* Add fsspec callback support for `put_file` and `get_file` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/120
* Drop isort hook in favor of ruff plugin by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/122
* Remove stale lakeFS SDK branch statement by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/124
* Minor test cleanups by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/125
* Add type hints to lakeFS file and file system [ENG-123] by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/123
* Keep only single development docs version for main branch by AdrianoKF in https://github.com/appliedAI-Initiative/lakefs-spec/pull/131
* Upgrade to mike to v2.0.0 by AdrianoKF in https://github.com/appliedAI-Initiative/lakefs-spec/pull/132
* Switch out black for ruff format by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/134
* Explicitly check for existence in `LakectlConfig.read()` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/133
* Add ruff-format hook by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/136
* Add lakeFS transactions by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/135
* Defer to super class for `get_file` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/137
* Update demo to use transactions, bump lakefs-spec to HEAD and urllib by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/138
* Remove hooks functionality by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/139


**Full Changelog**: https://github.com/appliedAI-Initiative/lakefs-spec/compare/v0.2.1...v0.3.0