Aistore

Core
- [API change] archive and download logs (feature) !6172, !6175
- [API change] dsort: extend input format !6181
- [API change] dsort spec; CLI: print job spec !6204
- [API change] revise request spec (major upd) !6217
- [API change] dsort: is now 'xaction' as well !6253
- (downloader, dsort, ETL): disallow to run when out of space !6235
- handle "DNS lookup fail" as one of the _unreachable_ err types; nlog flush-exit !6164
- when electing new primary; when joining nodes at startup !6165
- k8s: Change prod k8s and docker default to not log all to stderr !6166
- revise GFN !6167
- stats runner is now responsible to periodically flush logs !6170
- core: fail user attempt to abort global rebalance when !6184
- new Go API; assorted fixes !6189
- metasync BMD; up modules !6190
- downloader: return not-found when not found !6196
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- support S3 AWS profiles with alternative creds (feature) !6214
- core: state transition => rebalance => (point of no return) !6216
- amend low-level Go API check-response routine; add error type-code !6228, !6229
- control plane: deserialize original error from call result !6230
- xactions: when checking inactivity ("is idle") !6242 !6243
- primary readiness vs cluster shutdown !6244
- Go API: wait for xaction-related conditions !6245
- assorted tuneups: space cleanup; housekeeping (HK) callback; log !6246
- access control: when copying/transforming/dsorting to non-existing 'ais://' destination !6255
- core: a call to update stats should never block !6257
- core stats: add _fast_ counters !6258 !6259 !6261
- sparsify latency stats !6260
- ETL: refactor and cleanup construction !6267
- deploy/dev: updated minikube scripts !6272
- new option to add Cloud bucket to aistore without checking accessibility !6275, !6277
- un-throttle PUT mirroring; assorted changes !6278
- feature: local generation of global (job) IDs !6280 !6282

Performance
- Add distributed loader scripts and playbooks for using aisloader with multiple hosts !6156
- pyaisloader: usability improvements !6215
- Update Grafana dashboard to include latency statistics !6249
- Reorganize benchmarks and related tools !6254
- aisloader: no need to call `rand` for 100% or 50% read/write workloads !6256
- aisloader-composer: add dashboard for DC network and disk !6266
- aisloader: add an option to randomize gateways !6279
- aisloader-composer: fix output files for GET bench !6283

Python
- sdk: update ETL templates (docker migration) !6168
- sdk: Release version 1.4.1 !6169
- sdk: ETL templates (compress + ffmpeg decode) !6185
- sdk: ETL templates (imagepullpolicy as always) !6191
- sdk: adding keras_transform template !6200
- sdk: ETL templates fix !6201
- sdk: ETL templates (ffmpeg decode transformer) !6205
- sdk: compress ETL template (updated usage) !6211
- sdk: torchvision sample transformer ETL template !6221
- sdk: fix comments (minor) !6240
- sdk: update version !6248
- sdk: increase timeout for torchvision transformer template (large image) !6252
- sdk: updated torchvision transform ETL !6262
- sdk: update dsort job info query and related tests !6265
- sdk: switch ETL init code 'transform_url' boolean flag to 'arg_type' string !6269
- docs: update ETL dev deployment for macOS !6163
- ETL: keras template minor fix !6213
- ETL: remove incorrect reference !6268
- ETL: add 'arg-type=FQN' (new) !6271

Datasets (resize, resort, and shuffle)
- [API change] dsort: extend input format !6181
- dsort input format: iterate list, iterate range !6186 !6187
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- add test scripts; memsys: init gmm only once !6192
- refactoring and renaming !6193
- move/consolidate error types; continued refactoring !6202
- Go API change; add dsort/api.go; CLI: print job spec !6203
- [API change]: dsort spec; CLI: print job spec !6204
- CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
- dsort: continued refactoring (major update) !6208, !6209, !6210
- free sgl on error; feature: _any_ extension !6212
- [API change] revise request spec (major upd) !6217
- create destination on the fly !6218
- record content path to retain full shard name !6219
- output shard size estimation (rewrite) !6223
- add is-compressed; refactor dsort-mem !6227
- compressable shards (major) !6231
- output ext; rcb buffer; fixes !6232
- duplicated records (full coverage & stress); fixes !6233
- fix tests; add stress !6234
- rename subpackage, fix comments, refactor !6237
- remove dsort-context, rewrite initialization !6238
- static/stateless shard readers/writers; refactor and simplify !6239
- two goroutines per each shard-distributing request !6241
- [API change]: dsort: is now 'xaction' as well !6253
- dsort: support generic abort-xaction API !6264
- no need to block when sending shard records !6286

CLI
- archive and download logs (feature) !6180
- clarify "copying" vs "transforming" and "cached" vs "present" !6183
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- dsort: extend inline help, pretty-print job spec; update docs !6206
- dsort: Go API change; add dsort/api.go; CLI: print job spec !6203
- 'archive get' is now a shortcut (an alias) !6222

Build, test, and tools
- add test scripts; memsys: init gmm only once !6192
- tests and tools: cleanup around stop-maintenance, wait-rebalance" !6194
- deployment: update local deployment script to allow target-only deployment with defined primary host !6195
- deployment: optionally, skip deploying primary proxy !6197
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- tools/generate shards: optimize buffer allocation !6224
- deploy/dev: Add ansible deployment scripts for deploying locally on multiple nodes !6199
- aistorage/CI docker image (lzma libraries) !6220
- tests: init with cleanup and without !6226
- CI: Retry stuck Python ETL tests in GitLab CI pipeline !6270
- remove `aisfs` (FUSE) !6273
- dev tools: readers; handle read from corrupted arch or non-arch !6250

Documentation
- update [getting started](/docs/getting_started.md) !6161
- updated python sdk readme !6162
- update ETL dev deployment for macOS !6163
- update documentation with recent ETL changes !6173
- CLI/dsort: extend inline help, pretty-print job spec; update docs !6206

Core
- add `htext` to track restarted state; target run and misc !5966
- cluster rebalance (scenarios) !5969, !5971, !5973, !5974, !5975, !5977, !5980, !5983, !5986, !5987, !5989, !5991, !5992, !5993, !5995, !6002
- add 'cluster-ready' helper; use it to reinforce !5976
- cleanup better when decommissioning; previous BMD at startup !5979
- fs: reliable remove-all !5981, !5982, !5984
- yet another buf pool !5985
- do not modify cluster map when starting up; always skip logging idle disks !5988
- rebalance (scenarios, major update) !5992
- [API change]: core: rebalance (scenarios) !5993
- rebalance (major update); when receiving new cluster map !5995
- up modules; handle housekeeper registration race !5994
- 'not present in the loaded cluster map' and similar startup validation !5996
- shutdown or decommission a node that's already in maintenance !5998
- transport: never establish a streaming connection to the peer that's in maintenance (or will be) !5999
- `metasync` just-in-time; assorted refactoring (minor) !6001
- maintenance mode: pre & post vs keepalive & metasync; CLI: more colored cues !6004
- shutdown is also 'maintenance'; docs: adding-removing intro !6005
- add `meta` package !6006, !6007
- ETL: add arg-type parameter when initializing with code !6008
- archive v2: support empty template (tar entire bucket); atime !6013
- keep poi.atime in nanoseconds !6015
- archive v2: append to arch; refactoring !6017
- archive v2: up modules !6018
- archive v2: part four (major) !6019
- archive v2: detect an empty tar when appending, and handle !6020
- archive v2: part six !6022
- archive v2: mime detection !6024
- archive v2: extend 'append-to-arch' to support tar.gz !6025, !6027
- archive v2: tar and tgz append; fixes !6028
- log filenames; overlapping run vs node-restarted !6029
- archive v2: multi-object append-to-arch !6030
- archive v2: multi-object append-to-arch !6033
- archive v2: multi-object append-to-arch !6034
- cleanup disk utils (minor) !6035
- ios startup: run the command only once !6036
- hide AuthN secret !6038
- archive v2: append to zip !6041
- archive v2: append to msgpack !6043
- add `cmn/archive` package !6044
- archive v2: write and copy via new 'cmn/archive' !6045
- archive v2: append via new 'cmn/archive' !6046
- [API change] archive v2: MIME vs file extensions !6047
- ios: cleanup lsblk cache; CLI: refactor get-node-arg; up modules !6048
- archive v2: remove msgpack; refactor !6051
- archive v2: add '.tar.lz4' serialization (new) !6053
- archive v2: tar.lz4 cont-d !6054
- archive v2: lz4 features; checksum !6055
- s3 compat: run E2E tests with correct HTTP/HTTPS mode !6057
- [API change]: append to arch if exists !6062
- [API change] append to arch if doesn't exist; CLI cont-d !6064
- checksumming and buffering vs reader-from !6074
- core: content-length universally; revise write-json and friends !6075, !6076
- archive v2: [API change] put (files, dirs) with an option to append !6081, !6082
- archive v2: quiesce faster, refine continue-on-error logic !6083
- core: double-check target-in-maintenance, quiesce faster !6084
- archive v2: finalize cmn/archive package !6085
- archive v2: finalize cmn/archive package !6086
- log verbosity: core and modules !6087
- http client: disable compression; core: undefer & micro-optimize !6066
- append to (non-existing) arch: an option to create !6068
- mem-pool alloc/free symmetry: copy/transform & archive !6069
- copy/transform, multi-archive: refactor Rx logic and error handling !6071
- log verbosity: core and modules; remote cluster !6089
- log verbosity: core and modules; remote cluster !6091
- ec: minor refactoring !6092
- archive v2: WD basename; get with extraction; Range !6093
- archive v2: WD basename; get with extraction; Range !6094
- archive v2: tools/archive utils !6095
- archive v2: tools/archive utils !6096
- compile-out asserts; super-verbose logging; log module 'mirror' (ref) !6097
- log verbosity at runtime; log modules; remove glog; unify (major update) !6099
- fields iterator; size converter; log rotation (fixes) !6101
- [API change] get-bucket-info to count remote objects !6102
- [API change] get-bucket-info to count remote objects !6103
- [API change] get-bucket-info (part three); docs and CLI !6106
- list-objects vs buckets: revise and refactor, add validation, clarify !6107
- list-objects: introduce optional args (ref, cleanup) !6108
- list-objects: mem-pool msgpack buffers !6109
- kvdb: remove redundant err-not-found; amend dsort, downloader, authn !6110
- x-lso must idle more time !6111
- log modules (part three) !6112
- add `nlog` (new logger) !6113, !6122, !6124
- do not log perf counters when there's no change; sort the names !6119
- fix disk usage call for clusters on mac OS !6121
- log etl events: spec parsed, pod ready, hpull/hpush !6125
- cleanup `fs.PathError`; add object name validation !6127
- extend dsort to support `.tar.lz4` !6128
- aisloader: cleanup output when running with json option !6129
- xactions to return extended error info !6130
- xactions: add and return errors (major upd) !6131, !6132
- xactions: error handling cont-d; bucket-scope multi-object op-s !6133
- jobs error handling and reporting; tests !6134, !6135, !6136
- use Go 1.20 `join-err`; reduce default pre-election interval !6141
- http request multiplexer !6142
- dsort: use common cmn/archive pkg (major upd) !6143
- archive v2: tar formats (USTAR, etc.) !6153
- archive v2: extend list-objects and GET to operate inside archives !6155
- archive v2: extend list-objects and GET to operate inside archives !6157
- control plane: consistently propagate cluster map !6158
- old EC metadata shouldn't terminate cluster-wide rebalance !6159
- core: when primary goes down it notifies !6160

CLI
- performance tabs: always show 'cluster idle' if idle !5997
- stop-maintenance w/ no rebalance; stats idle-ness (minor) !6000
- add `transform-url` flag (used when initializing ETL with code) !6010
- misc. improvements !6011
- `mountpath` completions; disable/detach; minor ref !6056
- de-spaghettify put handler !6059
- archive multi-object (cont-d) !6060
- archive v2: CLI put, append, alias, docs !6072
- move 'gen-shards' and extend it to support all formats !6073
- archive v2: CLI: APPEND-to-arch is now on-par with multi-PUT !6077
- archive v2: CLI: `dry-run` option is back !6078
- archive v2: CLI: destination naming; dry-run; tips and examples !6079
- list-objects: extended help, template with no ranges !6104
- error and warning verbosity (major update) !6105
- log modules (part two); CLI archive: pre-parse and add tip, advice !6100
- archive help !6126
- archive v2: CLI and docs !6149
- archive v2: multiple CLI updates and improvements; is-archive bit !6150
- remove k8s apimachinery pkg (minor) !6151

Python
- sdk/python: Add option to pre-import modules when initializing ETL with code !5963
- sdk/python: Refactor ETL to provide name once on object creation !5968
- sdk/python: Release version 1.2.0 !5970
- sdk/python: Fix python package links !5978
- sdk/python: improve SDK testing, fix bucket eviction keep_md parameter !5990
- sdk/python: Release version 1.2.2 !6012
- sdk/python: Add dSort support in SDK !6032
- sdk/python: Add wait to dsort abort test !6037
- sdk/python: Fix dsort abort test on faster systems !6039
- sdk/python: Add get_url option to bucket, object group, and objects !6042
- sdk/python: Add interface for AIS sources that can be accessed via list of URLs !6052
- PyTorch: Add new PyTorch DataPipe to iterate over URLs from various AIS sources !6058
- update lint-tests to work with Python 3.11 (pytorch unsupported) !6061
- sdk/python: Include user agent in SDK requests !6067
- sdk/python Use msgpack content type when listing objects in bucket !6070
- sdk/python: Add msgspec to pyproject dependencies !6088
- sdk/python: Release SDK version 1.3.0 !6090
- sdk/python: bucket summary + bucket info !6098
- sdk/python: bck info integration test fix !6115
- sdk/python: bck summary integration test fix !6117
- sdk/python: bck summ + bck info fixes (revert and cleanup) !6120
- sdk/python: pyaisloader !6123
- pyaisloader: bucket utils (minor fixes) !6139
- sdk/python: bucket summary and info (minor fixes) !6140
- `pyaisloader`: total size/count fix !6146
- sdk/python: Release version 1.4.0 !6152

Docs
- add [lifecycle.md](https://storagetarget.com/2023/05/05/lifecycle-management/) !6009
- WebDataset Blog Post part 1: Storing WebDataset in AIS !6014
- WebDataset Blog Post part 2: AIS ETL on WebDataset shards !6016
- update docs/tools and cmd/README !6050
- [WebDataset blog post pt 3 -- PyTorch Datatpipe with ETL](https://aiatscale.org/blog/2023/06/09/aisio-transforms-with-webdataset-pt-3) !6065
- archive v2: CLI put, append, alias, docs !6072
- archive v2: CLI: destination naming; dry-run; tips and examples !6079
- archive v2: CLI and docs !6149

Build, CI
- add local build options to k8s scripts and fix local registry !6021
- build: upgrade all minors !6040
- add `pyaisloader` stage !6148
- k8s: fix deployment scripts for compatibility with the latest `aisnode` image !6154

Table of Contents
- **CLI v1.2**
- **Python SDK v1.1.2**
- **S3 compatibility and [Botocore](https://github.com/NVIDIA/aistore/blob/master/python/aistore/botocore_patch/README.md)**
- **API changes**
- **Tests and Documentation**
- **Core: bug fixes and improvements**
- **Build and Continuous Integration**
- **Extensions: Downloader, dSort, ETL**

See also:

* [Blog: Transforming non-existing datasets](https://aiatscale.org/blog/2023/04/10/tco-any-to-any)
* [Blog: Python SDK: transform and load into PyTorch](https://aiatscale.org/blog/2023/04/03/transform-images-with-python-sdk)

CLI
- show all jobs !5645
- start/stop job/xaction !5660
- refresh rate and countdown; long-running 'show job' and friends !5651
- 'show log node-name' to mimic 'tail -f' !5652, !5654
- add custom duration flag and logic !5655
- 'ais config (cluster|node|cli)', 'ais config reset', and friends !5656
- bucket completions !5657
- set-config to show all updates; tweak iter-fields reflection !5658
- 'show job' to aggregate all categories and support all selections !5661
- transition to using job display names (major) !5663
- start, stop, show jobs and xactions (cont-d) !5665
- amend and restructure jobs !5666
- running xactions (completions) !5672
- tweak config json printout; get-config from memory !5673
- update backend config !5674
- update backend config (part two) !5677
- add footnote, marshal message only once !5678
- remove `xaction` term and subcommand (everything is `job` now) !5692
- suggest (targets, proxies, nodes) !5694, !5696
- revise bash completions script !5697
- remove 'xaction' (term and subcommand) !5698, !5699, !5700
- 'show cluster': separate cluster nodes from all other (tab-tab) completions !5701
- consolidate and refactor cluster map access !5704
- tweak `ais create bucket --props` & `ais bucket props set` !5706
- extend 'job start' to support (resilver, copy-bucket, rename-bucket) !5715
- tweak listed props !5719
- remove (cleanup) download and dsort jobs !5721
- extend 'ais stop' to support --all|--regex !5722
- 'show job' verbose option; unify usage args; ref PUT/APPEND !5723
- rewrite command-not-found logic; add similar commands !5724
- `show jobs` (major) !5726, !5727
- bash autocomplete ordering improvements !5728
- improvements (usability) !5729
- add `bucket cp` alias !5730
- flag printable name; split 'show job' in parts; usability !5736
- further unify stopping, waiting-for, and showing jobs !5744
- revise & amend 'show rebalance' - all permutations !5761
- universal start-end formatting; template refactoring !5762
- jobs grouping by name and, within name, by UUID !5764
- complete `etl-name` transition !5767
- ETL tools, UUID (part one) !5745, !5746, !5749, !5753, !5754, !5763
- fix download/dsort progress !5769
- new table to show target statistics !5788
- 'ais show performance' (new) !5791, !5793, !5800, !5802, !5803, !5809, !5810, !5811, !5812, !5816
- IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5820
- reduce code, simplify, cleanup !5821
- IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5823
- disk stats: add average read/write sizes !5824
- amend existing mountpath tab and add a new one !5833, !5834
- expect node unreachable when iterating '--refresh' !5837
- assorted usability; add 'no-color' config !5839
- 'ais show performance': average (GET, PUT, etc.) sizes on the fly !5840
- support new API to reset stats !5841, !5843
- 'ais show performance': refactor throughput, add latency !5844
- 'ais show performance': finalize latency tab !5847
- 'ais show performance' cont-d !5848
- 'ais show performance': finalize top-level tab !5850
- 'ais show performance': add cluster-level throughput, beautify !5852
- 'ais show performance': alias 'stats' and remove older code !5853
- 'ais show performance': disk table v2 !5855
- 'ais show performance': finalize disk table !5857
- 'ais show performance': new mountpaths/disks/capacity table !5858, !5859
- 'ais show performance': finalize capacity table !5861
- refactor and cleanup multi-object put !5862
- multi-object PUT: source dir, list/range; matching pattern !5865
- fix concatenation logic, refactor progress bar !5867
- copy bucket: support progress bars (copied objects and size) !5870
- consistent timeout management !5871
- copy/transform a list or range of objects: add progress bar !5873
- copy/transform with progress bar: style, reuse !5874
- multi-object PUT !5876
- rogress bar: all multi-object operations; universal 'wait-for' !5879
- PUT multi-object - all flavors !5880
- get multiple objects in one shot (""multi-object GET"") !5884
- GET destination & assorted fixes !5882
- copy-bucket: prepend prefix, command helps, examples !5889, !5891
- more inline help !5892
- assorted improvements (minor) !5900
- fix downloading with progress bar enabled !5903
- how-to text: how to reconfigure remote ais cluster !5932
- add CLI compatibility warning (new CLI vs old cluster and vice versa) !5952
- cluster membership-changing operations (shutdown, decommission, et al): improve usability !5956

Python: SDK library and ETL
- Add unit tests for cluster class !5675
- Add unit test for api client class !5676
- Fix python ETL test workflow !5680
- Add unit tests for Xaction class !5681
- Add unit test for object class, fix object put data from a filepath !5682
- Add unit tests for ETL class !5683
- Add unit tests for sdk utils !5685
- Restructure python subdirectories and containment !5686
- Run python unit tests as part of default python test make option, include python ETL tests in all python test runs !5688
- Only run python ETL tests when python labels are added !5702
- update test utils to support running Python tests on Windows !5716
- Add multi-object functionality !5720
- Add tests and validation to object ranges, support leading zeros !5731
- Add unit test for object group class !5732
- Update documentation for python multi-object ops !5734
- Use aws bucket for python sdk CI tests to test caching functions !5739
- Add string template support to object groups !5741
- Refactor all references from xaction to job !5742
- Update ETL runtime defaults and add new python 3.11 option !5743
- Fix remote bucket tests to avoid collisions !5748
- Increment cloudpickle version and update github action to use 3.11 for tests !5756
- Fix python test dependencies !5780
- Refactor to simplify typing and tests !5789
- Bump python SDK to 1.0.5 !5790
- Update python sdk version !5792
- Update pytorch integration README with compatibility issue !5794
- Patch sdk to support torchdata integration !5795
- Bump aistore package version !5798
- Fix pylint and formatting !5799
- Add ObjAttr type for returning additional object metadata !5805
- Address lint warnings, general improvements !5808
- Add writer option to object get !5813
- Split README for different python projects !5814
- Add PROMOTE functionality to objects !5818
- Standardize pylint version, fix lint errors !5822
- Improve object put behavior and add directory put options !5826
- Use Pathlib over os.path !5827
- Refactor multi-file put to bucket class !5830
- Improve job interface and update promote options !5832
- Update python package build tools, increase version to 1.0.9 !5835
- Improve example documentation !5842
- Improve input validation !5846
- Set up proper logging, update constants !5849
- Add job wait for idle status and fix job bucket filter !5863
- Add multi-object copy !5866
- Fix remote test fixture !5877
- Add multi-object ETL !5878

Table of Contents
- **Remote AIS clusters**
- **List objects**
- **List and summarize buckets**
- **S3 API**
- **CLI**
- **Documentation**
- **Tests**
- **Python SDK & ETL**
- **Core**
- **Refactoring & fixes**
- **Tools**
- **Build & CI**

See also:

* [Blog: AIStore 3.12 Release Notes](https://aiatscale.org/blog/2022/11/13/relnotes-3.12)

Remote AIS clusters
- list remote ais buckets; fix remote namespace containment !5567, !5621
- aliasing and bucket namespaces (major revision) !5622, !5623
- ais gateways to cache remote ais clusters (information) !5630
- extend remote ais errors - add cluster UUID !5631
- remote ais aliasing !5632
- **[API change]**: remote ais clusters (major) !5633, !5634
- **[API change]** add `HealthUptime`; use it to show remote clusters !5636
- remove support for multi-aliasing !5637
- alias vs bucket metadata !5638

List objects
- CLI to show `CACHED` (column) only when it's informative (and not when user asks `ais ls ... --cached`) !5554, !5555
- rename `list-objects` structures; regenerate message pack !5542, !5543
- amend and re-enforce _single target_ rule for remote buckets !5556
- introduce location property, phase-out target URL (property) !5557, !5559
- add custom metadata, regenerate message pack; refactor !5564, !5565
- `list-objects` version 2 (major): !5583, !5584, !5586, !5589, !5590, !5591, !5601, !5602, !5605, !5607
- [backend API change] mem-pool 'list-objects' pages !5604
- `list-objects` flow with a single target listing _remote_ props (name, size, version) !5608
- list remote pages v2: flow controls; ais gateways will now select _single target_ to execute remote call !5609, !5610
- `list-objects` to include (in listing result) custom object properties !5614
- consistency in custom props across: `get`, `put`, `head` object, and `list-objects` !5615, !5616

List & summarize buckets
- easy URL to support listing buckets via `/gs`, `/az`, and `/ais` endpoints !5486
- `list-buckets` and `get-bucket-info` API; CLI summary table !5521, !5522
- `list-buckets` version 2 (major update) !5524
- unify 'list-buckets' and 'bucket-summary' !5530, !5531
- shorter and faster `list-buckets` version: w/ presence but wo/ summary !5535, !5536- **[API change]** add `presence` filter; eliminate bucket-info control structure (is redundant); refactor !5540
- `list-buckets` and `bucket-summary` cont-d !5541
- bucket sizing: always compute two sizes: on-disk and sum objs !5544
- `bucket-summary` on-disk (apparent) vs sum-all-objects, remote vs cached !5545
- `bucket-summary` and `list-buckets` cont-d unification !5546, !5547
- `list-buckets` by target: remote ais; hdfs and http; cloud; other !5567
- rename/move bucket summary; align control structures !5568
- list remote ais buckets; fix remote namespace containment !5621
- **[API change]** `list-buckets` correctness for named bucket queries !5626

S3 API
- multipart upload !5458, !5460, !5467, !5468
- extend s3 API to support remote buckets (major) !5462
- persistent multipart state (major) !5466
- multipart GET !5472
- multipart list active uploads and upload parts (major) !5474
- return xml-formatted errors (compliance) !5480
- multipart: refine and simplify !5482
- revise `list-buckets` to include all [providers](https://github.com/NVIDIA/aistore/blob/master/docs/providers.md) !5485
- always set header `Server` when redirecting !5495
- amend [s3 compatibility](https://github.com/NVIDIA/aistore/tree/master/docs/s3compat.md) readme; add multipart example !5496
- multipart: sha256 fix; md5 not checked but computed; `aws s3api` example !5500
- xml-format errors (parts two and three) !5501
- revise `HEAD object` implementation; multipart `ETag`; miscellaneous fixes !5504
- compatibility: configurable root '/' access to AIS cluster !5572, !5573, !5574
- compatibility: assorted error codes !5620

CLI
- 'ais etl' to replace extension in transformed objects !5418
- standardize error and warning coloring !5493
- don't log and don't show backend secret !5506
- improve unknown-command messaging, provide hints !5512
- add `ais object ls` command; add `present` flags !5514
- support updated HEAD(bucket) API; 'ais ls' and permutations to show obj props !5517
- static linkage with `cgo` disabled !5526
- rename aws, et al. backend provider constants !5528, !5529
- option to display sizes either in bytes or KiB, MiB etc. units !5532
- rename sub-packages; show-unmatched & hide-header !5534
- bucket `display-name` helper (and consistent usage) !5539
- flip bucket listing default to 'present' (was 'all accessible') !5550
- option `--all` to list both buckets and objects (in remote buckets) !5551
- multi-feature completions; action-warn and action-done !5575
- show config subsections in JSON; multi-feature completions !5576
- amend config JSON printout !5577
- extend and amplify: (config | section | json) !5578
- is-time-not-set, is-object-not-cached (for consistency) !5611
- `atime` & `ctime` !5612
- throughput must be averaged over an interval of time since the previous request !5643

Documentation
- amend [main readme](https://github.com/NVIDIA/aistore/tree/master/README.md); add Reuse section !5375, !5376, !5378
- blog: [Python SDK: Getting Started](https://aiatscale.org/blog/2022/07/20/python-sdk) !5373
- add `curl` example to set backend bucket property !5380
- python/sdk: ETL !5397
- amend [main readme](https://github.com/NVIDIA/aistore/tree/master/README.md): add PyTorch integration section !5406
- remote backend; native integration !5407
- revise `ulimit`, reference checked-in version of `limits.conf` !5429
- update [CONTRIBUTING.md](https://github.com/NVIDIA/aistore/tree/master/CONTRIBUTING.md) !5459
- blog: [AIStore: Data Analysis w/ DataFrames](https://aiatscale.org/blog/2022/08/15/dask-data-analysis) demo !5469, !5473
- update [s3cmd](https://github.com/NVIDIA/aistore/tree/master/docs/s3cmd.md) readme - add multipart example !5483
- restructure and extend [getting-started](/docs/getting_started.md); revise and amend `Local Playground` option !5484
- easy URL: add readme with extended examples and comments !5487
- [REST API](https://github.com/NVIDIA/aistore/tree/master/docs/http_api.md) querying information !5641
- [REST API](https://github.com/NVIDIA/aistore/tree/master/docs/http_api.md) listing buckets (remote namespace, presence and existence) !5644
- update ETL [main readme](https://github.com/NVIDIA/aistore/tree/master/docs/etl.md), [CLI readme](https://github.com/NVIDIA/aistore/tree/master/docs/cli/etl.md), and [examples](https://github.com/NVIDIA/aistore/tree/master/sdk/python/examples) !5471
- amend [s3 compatibility](https://github.com/NVIDIA/aistore/tree/master/docs/s3compat.md) readme, add multipart example !5496
- [main readme](https://github.com/NVIDIA/aistore/tree/master/README.md): Installing from release binaries (add section) !5640

Tests
- python ETL !5410, !5461, !5476
- python `IO communicator` !5415
- tests: skipping short (minor) !5416
- pyhton/sdk: ETL stress test !5420
- add s3 multipart test !5482
- add ETL test for streaming with python runtime !5513
- re-enable and amend object-properties tests across all providers !5515
- amend `list-buckets`: add cases; up CLI !5569, !5570
- reinforce multi-object tests, fix `bucket-exists` helper !5606
- amend and fix multi-object range/list copy test !5619
- amend `list-buckets` test that checks expected result !5631

Python SDK & ETL
- ETL API part 1 !5370
- sdk version !5374
- code (re)formatting and misc !5381
- reorg Makefile dependencies !5390
- ETL APIs part 2 !5394
- bucket ETL transform API !5405
- bump sdk version to 1.0.3 !5408
- fix `IO communicator` fixes !5413
- fix transform bucket !5421, !5428
- add examples !5441
- add streaming runtime w/ refactoring and cleanup !5444
- streaming `init-code` with chunk-size param (major) !5446, !5447
- with temp debug !5448
- `init_code` changes for before and after functions !5449
- remove debug w/ minor ref !5450
- functions `filter` (new), `transform`, and `before`/`after` !5451
- runtime v2 fixes !5455
- bootstrapper; by reference !5457
- ETL API fixes !5464
- update readme (etl init_code) !5475
- extend `hpush` yaml with bitwise flags !5479
- remove unused ETL functions; add test for streaming with python runtime !5513
- update Jupyter notebook to reflect sdk changes !5548

Core
- global rebalance: initiating new vs receiving delayed ACKs !5426
- etl md: remove sgl, read-all !5470
- error handling: introduce type codes (major upgrade) !5477, !5481
- easy URL: support listing buckets via `/gs`, `/az`, and `/ais` endpoints !5486
- easy URL: add readme with extended examples and comments !5487
- user agent, internet browser: pretty-print JSON response !5492
- add `User-Agent` to http request headers !5494
- backward compatibility; remote backends !5498
- consolidate http headers; add type-cast fail !5503
- introduce error types unsupported and not-implemented-yet !5508
- refactor GET(object) !5507
- add `lom.rename-file` w/cleanups !5509
- simplify `lom.persist` & `recache/refresh` !5510
- update `atime` on PUT !5511
- HEAD(bucket): extend the API in re: cluster metadata !5516
- **[API change]** add `api.GetBucketInfo` !5519
- enumerate and unify bucket and object 'presence' vs 'existence' !5520
- `fs`: add size-bucket and refactor x-bucket-summary !5523
- backend providers as simple KVs (minor) !5525
- on the fly and (new) upon request remote bucket creation/addition !5538
- object props: remove get-node, add location and helpers !5561, !5562, !5563
- creating/adding buckets on the fly: further clarifications !5566
- debug pkg: assert-msg is redundant; minor ref !5579
- `cos-assert`; align (minor ref) !5587
- periodic stats log: tweak idleness logic !5588
- `x-archive` to always close data mover !5594
- key-value driver; probabilistic filter !5595, !5596
- embed stop channel !5597
- transport: let individual streams define max header size !5598
- intra-cluster transport: grow receive header, simplify `rx-loop` !5599
- intra-cluster transport: within-read retries !5603
- aws backend: rewrite `list-objects` versioning part, speed-up working with versioned buckets !5612, !5613
- consistency in custom props across: `get`, `put`, `head` object, and `list-objects` !5615, !5616
- simplify bucket `try-init` logic to add buckets on the fly !5628
- notifications, control plane (ref) !5629
- cluster and node 'started-up' is now uptime (was `bool`) !5635
- **[API change]** add `HealthUptime` and use it to `ais show` remote clusters !5636

Refactoring & fixes
- make deploy: mark aws secret optional to unblock startup !5379
- consolidate `xaction`, move `xs` package !5383
- move `dbdriver`, simplify !5384
- GKE, cluster restart: amend false-positive SIE (storage integrity error) detection !5412
- defer freeing `etl-md` sgl !5414
- `fieldalignment` analyzer !5431, !5432, !5433, !5434, !5435, !5437, !5438, !5440, !5442
- fix missing `curl` in aisnode k8s image !5427
- http REST: parsing etc. helpers !5488
- rename aws, etc. provider constants !5527
- `gofmt` to replace `interface{}` with `any` !5533
- refactor and revise `objwalk` pkg !5551, !5553
- delete object vs 503 status; duplicated `trname` (transport endpoint name) !5614

Tools
- scripts: revise github-release: tar.gz binaries, sha256(resulting tar.gz); naming convention !5411
- add `devtools/trand` for tests; use strong randomization for product !5465
- tools: revise `get-primary`, `wait-for-cluster-state`, `wait-smap-sync` !5571, !5580, !5581
- `aisloader`: writing non-ais buckets !5600
- tools: add `buckets-contain` helper !5627
- scripts: amend local multi-cluster deployment to `--override-backends` and more !5617, !5618
- scripts: add [install_from_binaries](https://github.com/NVIDIA/aistore/tree/master/deploy/scripts/install_from_binaries.sh) !5640

Build & CI
- github workflow: setting go version to 1.18.3 !5369
- undo lint and Go version downgrades !5371, !5372
- add `.dockerignore` !5382
- cluster-minimal image: use alpine latest, remove azure !5385
- deploy/k8s: fixes; minimal minikube deployment !5386
- pull latest minikube image in GitHub workflow !5387, !5388
- fully transition to Ubuntu 22; apt-get changes !5389
- add `limits.conf` to all docker images !5391
- deploy/prod: docker Makefile !5392
- gitlab pipeline to build and push docker images !5393
- add github workflow to build docker images !5395
- two-stage Dockerfile to produce `aisnode` for minikube !5398
- add Makefile for dev/k8s scripts !5399
- add short CI stage to the k8s yaml template !5401
- cleanup [deploy/dev/k8s](/deploy/dev/k8s) readme and scripts !5402
- add cross-compile option !5409
- upgrade all minors !5417
- rebuild and use new docker images !5436
- fix latest lint issues !5423
- fix gitlab Dockerfile !5425
- `golangci`: re-enable `nolintlint`, run go1.17, comment-out disabled !5439
- run python/etl manually (was part of test-short) !5502
- `cli` module to have its own Makefile !5505
- github-release; never-nil (minor) !5422
- lint: re-enable `nolintlint` !5424
- update docker github workflow !5430
- pylint: fix warnings !5453
- `golangci` v.49 !5489
- fully transition to Go 1.19 (part two) !5491
- docker/cluster-minimal: configure aws and gcp backends !5499
- gitlab/CLI: force back e2e test suite; upgrade and build !5552
- upgrade all minor versions !5560
- add early return (linter) !5582
- update pylint settings to skip generated files !5642

- PyTorch: add support for `etl-name` in `AISDataset` and `AISFileLoaderIterDataPipe` !5957

S3 compatibility and Botocore
- Add botocore monkey patch alongside python SDK !5684
- Move botocore and pytorch packages to python top-level, separate from sdk !5691
- Add s3 compatibility testing with boto3 !5703
- list-objects vs HEAD !5705
- compute multipart md5 and set etag !5708
- Improve s3 compat test documentation and update validated tests !5747
- Pass S3 delete a list of objects !5854
- fix infinite loop when listing objects; add bucket name to list object response !5864
- return bucket creation date in UTC !5869
- new flag `NoRecursion` to support S3 delimiter feature !5930

API changes and new APIs
- add API to query multiple xactions via any IC proxy !5670, !5671
- yet another API to query xactions (new) !5687
- list-objects API: tweak listed props !5718
- [API change] flatten `xaction-snap` control structure (major upd) !5740
- [API change] ETL: tools, UUID (part nine) !5754
- [API change] ETL: new query parameter to specify transform name !5755, !5765
- [API change] init/start ETL to return xaction ID !5768
- [API change] GET(object) !5770, !5772, !5773, !5775
- [API change] remove xaction 'query-msg' (deprecated) !5783, !5784, !5785
- get-object API: amend comments !5769
- refactor `api` package (major) !5776
- [new API] query metric names and kinds ('counter', 'latency', 'throughput', et al.) !5796
- [API change] get node status !5811
- [API change] core stats: consistency between stats-querying APIs !5817
- [API change] PUT(object) !5828
- [API change] amend capacity-disks-filesystems control !5831
- [API change] add API to reset stats !5841
- [API change] copy bucket: add `prefix` option !5894
- [API change] copy/transform bucket: add 'prefix' option !5895
- [API change] bucket summary: add 'prefix' option (major) !5896, !5898
- [API change] return comma-separated list of job UUIDs !5927
- [API change] extend & refactor system info, add load averages !5951

Core
- xactions: display name, global & bucket scope, abort by kind (core) !5662
- xaction snaps V2 (major) !5667
- tweak reversing-to-remais logic !5668
- bucket summary !5678
- bucket summary: handle failure to start !5679
- offline access to remote ais cluster buckets !5674
- refactor `cluster.Target` interface !5689
- unify time formatting; log: current date periodically !5705
- fix HEAD(remote-ais-bucket) when the latter is not _present_ !5707
- xaction registry: cleanup list-objects separately, revise house-keeping !5725
- copy-bucket & transform-bucket: revise generic stats !5730
- global rebalance: fix cleanup upon (re)transmit failures; handle `err-stream-terminated` !5733
- extend xaction snap: add source and destination buckets !5735
- always return atime and checksum when listing 'cached' content !5738
- assorted fixes and ref (cont-d) !5759
- add `has-active-peers`; refactoring !5760
- special circumstance to retry `delete-object`; lesser assert !5782
- consolidate xaction API messages !5786
- rebalance upon cold reboot, refactor join/kalive !5787
- GET(object): add version-changed stats counter !5801
- core stats: lockless (counter, gauge, special) !5804
- core stats: revise error counting (major) !5806, !5807
- core stats: introduce `KindSize` !5815
- IEC and IS units of measurement (ref) !5819
- core: micro-optimizations and cleanup !5825
- rename mountpath and filesystem (ref pass) !5829
- stats: serialize to SGL !5834
- stats: rename for consistency !5836
- config features: add fsync-put; don't destroy bucket when creating one !5845
- core: micro-optimize BMD update !5851
- core: refactor 'ios' !5856
- extend the minimal/default subset of list-objects queried props !5868
- refactor packages: 'nl', 'api', 'xact' !5872
- add 'idles-before-finishing' state !5875
- parse IEC and SI units !5882
- multi-object GET (part two) !5885
- streaming-x: multi-bucket matching, transport name !5886
- long timeout to query _all_ xactions !5887
- copy-bucket: prepend prefix !5889
- LOM as a remote data provider (DP) !5905
- copy/transform remote objects that may or may not be present (major) !5906
- copy/transform remote objects that may or may not be present !5908
- copy/transform remote objects (part three) !5909
- end-to-end protection: compute a given checksum type only once !5912
- copy/transform remote objects (part four) !5914
- core/list-objects: always primary !5915
- copy/transform remote objects (part four) !5917
- canonical bucket/object name, and more !5918
- copy/transform remote objects (part five); canonical names !5919, !5921
- store cleanup: delay deleting bucket directories when in doubt !5923
- copy/transform remote objects (part six) - tests !5924
- remote ais cluster: fix API forwarding logic; transform/copy test !5926
- when node is in maintenance don't try to query it's state-and-status !5956
- revise waiting/polling logic, reset probe frequency !5960

Build and Continuous Integration (CI)
- add workflow job to build `ais-init` image !5646
- build: move CI (image, make); move/update testing-in-docker content !5710
- build: upgrade all minors !5711
- Update minikube deploy script, remove from test run !5774
- Update docker/ci image to include docker io !5778
- Add uuid dependency to docker/ci image !5779
- build: clear `golang/x/net` security alerts (two commits) !5838
- up modules; cleanup redundant err helpers, simplify; objhead !5907
- lint with new linter !5929
- upgrade linter !5931
- build: remove 3rd party atomic package (use standard) !5934, !5935, !5936
- math/rand vs crypto/rand ref !5937
- build: transition to Go 1.20 !5938
- up modules; ""end-to-end protection"" and other comments !5939
- add k8s pipeline to run manually !5941, !5943

Tests and Documentation
- tests: amend all-xaction-status; golangci-lint deadline !5781
- rename 'ais/tests => ais/test' !5860
- tests: fix copy/transform eviction logic, and misc !5928
- extend ETL tests, add from/to permutations !5940
- docs: 'show job' !5946
- docs: 'ais job' !5947
- docs: list objects and buckets, copy buckets, and more; add help.md !5953
- tests: fix `test-promote` !5763
- tests: remote bucket versioning !5766
- main readme, cluster.md, storage.md, config.md updated, and more...

Extensions: Downloader, dSort, ETL
- downloader: do not renew xaction !5645
- amend and refactor `dloader` package (formerly, 'downloader') !5647, !5648, !5649
- downloader and dsort: list-active !5650
- downloader: cluster-wide xaction ID (major) !5659
- downloader: continued refactoring !5660
- move ais extensions: `etl`, `downloader`, `dsort` !5664
- k8s: remove unused ExecCmd !5712
- default bucket props vs cluster config !5713
- dsort: add support for EKM with JSON structure !5717
- ETL: tools, UUID (part one) !5745
- ETL: tools, UUID (part two) !5746
- ETL: tools, UUID (parts three & four) !5749
- ETL: CLI, tools, UUID (parts three & four) !5751
- ETL: use (logical) "etl_name" instead of (unique) "etl_id" and "uuid" (major update) !5752
- ETL: tools, UUID (part eight) !5753
- Add pod health API endpoint for ETL !5771
- ETL: assorted renames for consistency !5777
- ETL stats; xaction base `notif` abstraction (minor) !5902
- downloader: reduce "wrapping" and deferring !5904