Lakefs-spec

⭐ Highlights

The v0.5.0 release of lakeFS-spec simplifies working with deferred placeholder entities returned by versioning operations as part of a transaction. These placeholders now work as transparent proxy objects, meaning they expose the same interface as the class they are wrapping (no more `Placeholder.unwrap()` necessary! 🙌🏻 )

The user documentation has been completed, especially the remaining missing pieces of the API reference.

🚨 Breaking Changes

- `LakeFSFileSystem.ls()` no longer defaults to recursive listing, in line with other fsspec implementations. If you require recursive directory listings, you can pass the `recursive=True` argument to `ls()` or consider the `find()`/`glob()`/`walk()` operations.

What's Changed
* Allow Path like inputs in public LakeFSFileSystem methods by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/182
* Create API docstrings for the `LakeFSFileSystem` and `LakeFSFile` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/181
* Docs: Allow Jupytext notebooks for tutorials by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/183
* Add API docs for all remaining smaller modules by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/184
* Record test coverage on fsspec package [ENG-196] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/188
* Modify error translator to raise more informative errors by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/190
* Idempotency of demo notebook by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/189
* Drop `requirements.txt` from package lock files by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/194
* Docs: Fix typos in tx user guide by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/197
* Bump lakeFS version for testing environment to v1.3.0 by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/199
* Docs: Add call to action for early adopters to README and landing page [ENG-186] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/196
* `ls`: Make non-recursive by default, correctly update dircache by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/198
* Introduce issue templates by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/202
* Revert "Introduce issue templates" by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/204
* Improve test coverage on lakefs abstract base classes by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/201
* Introduce templates - eliminate yaml errrors by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/205
* Validate with docstrings with `pydoclint` by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/203
* Add lakeFS server version guard to `LakeFSFilesystem.touch()` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/214
* Overload `LakeFSFile.__del__` to pass statement by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/219
* Remove bug ftr prefix issue templates by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/217
* Add overloads for `LakeFSFileSystem.ls()` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/221
* Docs: Client-side caching for Open-Meteo API in tutorial notebook by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/220
* Run `pre-commit autoupdate` by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/223
* Remove dunder methods from API docs by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/222
* Create template pull-request by Maciej818 in https://github.com/aai-institute/lakefs-spec/pull/206
* Correctly handle dircache for recursive `ls` calls by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/224
* Simplify transaction `Placeholder` class by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/225
* Fix transaction doc on `Placeholder` values by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/226

New Contributors
* Maciej818 made their first contribution in https://github.com/aai-institute/lakefs-spec/pull/202

**Full Changelog**: https://github.com/aai-institute/lakefs-spec/compare/v0.4.0...v0.5.0

The v0.4.0 release is dominated by the complete overhaul of the user documentation (check it out at https://lakefs-spec.org/) 🚀

The documentation now features a quickstart guide, a full user guide, and an improved tutorial on how to use lakeFS-spec in a data science project. Feedback on and contributions to the documentation are highly appreciated!

🚨 Breaking Changes

- The `LakeFSFileSystem.scope` context manager has been removed from the API in https://github.com/aai-institute/lakefs-spec/pull/175, since its functionality has been superseded

What's Changed
* Make `pyyaml` dependency mandatory by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/143
* Restructure README [ENG-122] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/129
* Run `pre-commit autoupdate`, delete unnecessary gitkeep file by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/144
* Remove file upload performance warning by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/146
* Improve lakeFS transactions by using deque, placeholders for commit SHAs by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/140
* Move demo lakectl to tutorials folder by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/148
* Integrate `.ipynb` to CI and move to docs by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/119
* Quickstart guide [ENG-146] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/142
* Delete stale black config and comments, fix error handling in blockstore puts by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/150
* Add lakeFS file system configuration doc by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/151
* Reword Python environment installation guide by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/152
* Add `exist_ok` flag to all resource-creating client helpers by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/154
* Docs: Landing page [ENG-147] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/155
* Docs: Branding fixes by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/157
* Improvements to `ls` implementation by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/166
* Add file system usage documentation [ENG-148] by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/156
* Docs: Add third-party integrations user guide by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/167
* Allow leading `lakefs://` scheme in input paths by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/169
* Change `fs.info()` logic to recognize non-slash-terminated directories by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/160
* Add transaction usage document [ENG-150] by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/171
* Documentation cleanup [ENG-198] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/170
* Docs: Post-hoc cleanup of third-party integrations user guide [ENG-149] by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/172
* Change certain aspects of the data science tutorial notebook by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/158
* Add doc-strings to the client_helpers by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/168
* Add HTTP409 check in `create_repository` for forward compatibility by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/177
* Docs: Render pre-release banner for `main` by AdrianoKF in https://github.com/aai-institute/lakefs-spec/pull/176
* Adapt `create_tag` client helper to reflect tag immutability by maxmynter in https://github.com/aai-institute/lakefs-spec/pull/179
* Remove `_strip_protocol` overload and usages by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/174
* Remove `LakeFSFileSystem.scope` context manager by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/175
* Add depagination helper by nicholasjng in https://github.com/aai-institute/lakefs-spec/pull/180


**Full Changelog**: https://github.com/aai-institute/lakefs-spec/compare/v0.3.0...v0.4.0

This release introduces transactions as the new way of carrying out versioning operations during file transfers. They supersede hooks, which were removed from the package.

Type hints were added to all public `lakefs-spec` interfaces, as well as `fsspec` callback support to file transfers.

This release also marks the transition to the `lakefs-sdk` package as the way of interfacing with the lakeFS server from Python. As such, older lakeFS server versions might not be supported anymore.

What's Changed
* Fix unbound local error by moving up definition of `exists` boolean by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/111
* Change lakeFS package to `lakefs-sdk` in place of `lakefs-client` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/107
* Add README on hack/ folder resources by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/115
* Implement Demo Feedback from [ENG-111] by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/117
* Run `pre-commit autoupdate` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/118
* Add fsspec callback support for `put_file` and `get_file` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/120
* Drop isort hook in favor of ruff plugin by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/122
* Remove stale lakeFS SDK branch statement by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/124
* Minor test cleanups by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/125
* Add type hints to lakeFS file and file system [ENG-123] by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/123
* Keep only single development docs version for main branch by AdrianoKF in https://github.com/appliedAI-Initiative/lakefs-spec/pull/131
* Upgrade to mike to v2.0.0 by AdrianoKF in https://github.com/appliedAI-Initiative/lakefs-spec/pull/132
* Switch out black for ruff format by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/134
* Explicitly check for existence in `LakectlConfig.read()` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/133
* Add ruff-format hook by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/136
* Add lakeFS transactions by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/135
* Defer to super class for `get_file` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/137
* Update demo to use transactions, bump lakefs-spec to HEAD and urllib by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/138
* Remove hooks functionality by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/139


**Full Changelog**: https://github.com/appliedAI-Initiative/lakefs-spec/compare/v0.2.1...v0.3.0

This release adds bare-bones user documentation which will be expanded in subsequent releases.

What's Changed
* Move ensure_branch out of _warn_on_[...] check by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/108
* Update demo `.ipynb` to current version and include `rev_parse` [ENG-99] by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/109
* Create bare-bones user documentation [ENG-110] by AdrianoKF in https://github.com/appliedAI-Initiative/lakefs-spec/pull/110


**Full Changelog**: https://github.com/appliedAI-Initiative/lakefs-spec/compare/v0.2.0...v0.2.1

What's Changed
* Add Reproducibility Example to Readme.md by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/93
* Run hooks even if precheck skips fs action [ENG-100] by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/100
* Remove no-context `host` specifications in FS constructor, drop `yaml… by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/101
* Allow LakeFS files to decay standard Blockstore files on request [ENG-92] by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/95
* Small fixes re: blockstore puts by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/102
* Remove setuptools/poetry entrypoint registration by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/104
* Introduce future annotations and guard imports for Python 3.9 compat by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/105
* Add badges to README by AdrianoKF in https://github.com/appliedAI-Initiative/lakefs-spec/pull/106


**Full Changelog**: https://github.com/appliedAI-Initiative/lakefs-spec/compare/v0.1.6...v0.2.0

This release adds the ability to copy files in a repo by implementing `LakeFSFileSystem.cp_file`, as well as a comprehensive demo to explore the features of `lakefs-spec`.

The client helpers module was greatly expanded by maxmynter.

Minor bug fixes include removal of stdout handlers from all loggers.

What's Changed
* Update lakeFS image in local tests and CI, bump ruff by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/87
* Add file upload and download events and corresponding `LakeFSFile` hooks by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/85
* Implement `LakeFSFilesystem.cp_file` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/84
* Implement directory listings caching for `ls` by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/86
* Add new `client_helper` functions by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/88
* Add rev-parse client helper by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/90
* Remove stdout handlers from module loggers by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/91
* Create a Demo for LakeFS-spec and Using LakeFS in a small scale e-2-e example project by maxmynter in https://github.com/appliedAI-Initiative/lakefs-spec/pull/67
* [ENG-96] Run pre-commit autoupdate by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/97
* Update Python development dependencies and lakeFS docker image tags by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/98
* Fix small README mistakes by nicholasjng in https://github.com/appliedAI-Initiative/lakefs-spec/pull/99


**Full Changelog**: https://github.com/appliedAI-Initiative/lakefs-spec/compare/v0.1.5...v0.1.6