Aistore

- stress test for copy w/ '--sync' !6552
- fix `pylint` to pass !6556
- test multi-object copy with '--sync' flag !6567
- fix black formatter issues in github CI !6582
- github-CI lint - follow up !6583
- support range read (offset, length) !6588
- update common requirements !6609
- bump SDK version !6610
- lint: add more !6454

Bench
- `aisloader-composer`: install docker alongside latest `cri-o` on CentOS !6436
- `aisloader-composer`: fix install-docker and update OCI inventory !6446, !6449
- `aisloader-composer`: update OCI inventory; avoid using reserved variables in playbooks !6452
- `aisloader-composer`: update dashboard with k8s only networking visualization !6453
- `aisloader`: support `latest-version` !6581
- `aisloader`: add '--cached' flag !6623

Build, CI
- refactor common 'k8s' package; up cli mod; docs !6434
- build/minikube: skip making cli !6437
- gitlab-CI: scheduled pipeline changes !6442
- upgrade OSS packages !6443
- lint: enable `gocritic` "huge-param" !6457
- lint: add `gosec` linter !6462
- gitlab: add `etl` label & rule !6488
- github-CI: publish `pypi` package for aistore !6492
- build: upgrade all minors !6501
- rename 'cluster' package !6514
- 'api' package not to import 'core' !6515
- tests, tests, and more tests !6530
- CI: fix HDFS docker image !6566
- CI: remove HDFS build and tests !6572
- deployment: add `jq` to init container for parsing JSON in Bash scripts !6577
- CI: update tgt cnt for test short !6579
- gitlab CI: add short test for cloud providers and long test for Azure !6584
- build: new linter !6624
- add github issue templates !6630
- build: release candidate 4 (rc4) !6640
- build: rc7; fixes !6658

Documentation
- blog: [aistore fast tier cache](https://aiatscale.org/blog/2023/11/27/aistore-fast-tier) !6444
- blog: [maximizing cluster bandwidth with multihoming](https://aiatscale.org/blog/2024/02/16/multihome-bench) !6642
- document aistore environment variables !6459
- aistore environment variables !6461
- packages (`meta`, `ais`, `cmn`); docs !6463
- in-cluster K8s environment: prune and cleanup, comment, and document !6471
- erasure coding: remove all gitlab references from the docs !6477
- update `bucket.md`, `out_of_band.md`, and CLI docs !6527
- fix LRU config docs !6598
- add blob downloader (readme) !6604
- update `etl-init` usage instructions !6611
- update main readme !6644
- blob-downloader, Prefetch !6645
- new arch/block diagrams !6655

- add support for self-signed certificates with or without verification !6465
- add 'latest' flag for GET !6536
- latest flag for prefetch and copy !6542

Highlights
- [**Blob downloader**](https://github.com/NVIDIA/aistore/blob/main/docs/blob_downloader.md)
- [**Multi-homing**: support multiple user-facing network interfaces](https://aiatscale.org/blog/2024/02/16/multihome-bench)
- **Versioning and remote sync**
- execute in presence of out-of-band changes/deletions
- support _latest version_: the capability to check in-cluster metadata and, possibly, `GET`, download, prefetch, and/or copy the latest remote (object) version
- _remote synch_: same as above, plus: remove in-cluster object if its remote counterpart is not present (any longer)
- both _latest version_ and _remote sync_ are supported in a variety of APIs (including `GET` primitive) and tools (CLI, `aisloader`)
- **Intra-cluster n-way mirroring**
- to withstand a loss of node(s) erasure coding is now optional
- **AWS S3 (frontend) API**
- multipart V2 (major upgrade); other productization
- listing very large S3 datasets
- support [presigned S3](https://github.com/NVIDIA/aistore/blob/main/docs/s3compat.md) requests (beta)
- **List objects** (job): show diff: in-cluster vs. remote
- **Prefetch** (job): V2 (major upgrade)
- **Copy/transform** (jobs): V2 (major upgrade)
- AWS S3: migrate AWS backend to AWS SDK V2
- Azure Blob Storage: transition to latest stable native SDK

> See also: [aistore features](https://github.com/NVIDIA/aistore?tab=readme-ov-file#features) and brief [overview](https://github.com/NVIDIA/aistore/blob/main/docs/overview.md#at-a-glance).

Core
- NVMe multipathing: pick alternative block-stats location !6432
- rotate logs; remove redundant interfaces, other refactoring !6433
- cold `GET`: add stats !6435
- http(s) clients: unify naming, construction; reduce code !6438, !6439
- don't escape URL paths; up cli !6441
- `dsort`: sort records (minor) !6445
- core: micro-optimize copy-buffer !6447
- `list-objects` utilities and helpers; rerun `list-objects` code-gen: refactor and optimize; cleanup !6450, !6451
- intra-cluster transport: zero-copy header !6455
- Go API: (object, multi-object): ref !6456
- add 'read header timeout'; docs: aistore environment variables !6459
- core: support target multi-homing - comma-separated IPs (part one) !6464
- package 'ais': continued refactoring; up cli !6466
- support multiple user-facing network interfaces (multi-homing) !6467, !6468
- when setting backend two (or more) times a row !6469
- core: (begin, abort, commit) job - corner cases !6470
- in-cluster K8s environment: prune and cleanup, comment, and document !6471
- multi-object `PUT` - variations !6473, !6474
- unify `PUT` and `PROMOTE` destination naming !6475
- `APPEND` (verb) to append if exists; amend metadata (major) !6476
- EC: refactor and simplify erasure-coding datapath; docs: remove all gitlab references !6477
- `list-objects`: enforce intra-cluster access, validate !6480
- EC: remove redundant state; simplify !6481
- Go API `get-bmd`; follow-up !6483
- EC: cleanup manager: remove rlock and unused map - micro-optimize !6490
- copy bucket: extend the command to sync remote bucket !6491
- extend 'copy bucket' to sync remote !6494, !6495, !6497, !6498, !6499
- don't compare checksums of different (checksum) types !6496
- when deleting non-present (remote) object !6502
- move transform/copy-bucket from 'mirror' package to 'xs' !6503
- don't create data mover in a single-node cluster !6504
- multi-object transform/copy (job): add missing cleanup !6506
- multi-object transform & copy !6507
- core: abort all (jobs) of a given kind; CLI 'ais stop'; strings: Damerau-Levensthein !6508
- revamp target initialization !6509
- copy/transform remote, non-present !6510
- revamp target initialization !6512, !6513
- [API change] get latest version (feature) !6516
- amend Prefetch; flush `atime` cache when shutting down !6517
- amend metadata cache flushing logic (`atime`, `prefetch`, `is-dirty`) !6518
- core: remote reader to support 'latest version' !6519
- extend config ROM; follow-up !6520
- Prefetch v2 !6521
- backend error formatting; `notification-listener` name !6522
- [API change] Prefetch v2; multi-object operations !6523
- Prefetch v2; cold-get stats; put size !6524
- [config change] versioning vs remote version changed or deleted !6525, !6526
- add 'remote-deleted' stats counter; Prefetch: test more !6528
- AWS backend `not-found`; job status; other cleanup !6529
- core: refactor 'copy-object' interface, prep to sync remote => in-cluster !6531
- [Cluster Config change] versioning vs remote version: remote _changed_, _deleted_ !6532
- copy/transform (bucket | multi-object); intra-cluster notifications !6533
- revise/simplify 'is-not-exist' check; `ldp.reader` to honor `sync-remote` option !6537
- pre-parse (`log-modules`, `log-level`); micro-optimize !6538
- amend error handling: `not-found` vs list iterator; OOS !6539
- jobs ("xactions"): add and log non-critical errors; `join(error)` and fiends !6540
- [API change] list-objects to report 'version-changed' (new) !6541
- `list-objects` to report 'version-changed' (new) !6543, !6545
- `list-objects` to report: 'version-changed', 'deleted' !6546
- `list-objects` to support (in-cluster <=> remote) diff !6547, !6548
- copy/transform with an option to sync remote: prune destination !6549
- copy/transform --sync: add stress test, extract "pruning" logic !6550
- revise and refine _object write transaction_ (OWT) !6554, !6555
- Go API: amend 'wait-idle' helper method !6558
- copy/transform '--sync': use probabilistic filtering !6559
- refactor `list-range-prefix` iterator !6560
- multi-object copy/transform with '--sync' option !6561
- S3 API (on the front): fix `list-objects` !6562, !6563
- multi-object copy/transform with '--sync' option !6564
- core: reset idle timer; xaction names (micro-optimizations) !6565
- core: `ETag` in response headers !6569
- S3 API (frontend): validate object names; multipart pathnames !6570
- copy/transform with '--sync' option: add scripted test !6571, !6573
- backend: special case to return 404 instead of 403 !6575
- productize Azure backend !6576, !6578, !6580
- S3 multipart: write-through all parts !6585
- multipart upload: write-through all parts !6586
- multipart upload: add extended error message; add stress test !6587
- all supported backends: revisit range read (make it consistent across) !6589
- introduce blob downloader (new) !6592
- xaction (job) descriptor: remove unused specifiers !6593
- blob downloader: add dedicated (non-generic) control path !6595
- blob downloader (new) !6596, !6599, !6603
- multipart upload: fix `s3cmd` to run elsewhere !6600, !6601
- blob downloader (new) !6605, !6606, !6608
- blob downloader (new); remote AIS cluster !6613
- silent `HEAD(bucket)` !6614
- leverage erasure coding to provide intra-cluster mirroring (new) !6615, !6616
- blob downloader (new) !6618
- S3 (frontend): support presigned S3 requests (new) !6621
- intra-cluster mirroring: add integration test (no limit) !6622
- blob downloader (new) !6628, !6629, !6631, !6632, !6633, !6639
- add target's `get-cold-blob` interface; refactoring !6634
- AWS backend: `nil` client !6636
- Prefetch via blob-downloader: add 'blob-threshold' option !6637, !6638
- blob-downloader: user abort; expected checksum !6646
- Azure: `ETag` as object version; build !6647
- Azure: transition from preview to stable 1.x (major) !6648
- AWS backend: use `sync.Map` instead !6649, !6651
- (AWS, GCP) backend: log extended error info; RC5 !6653
- S3: presigned S3 requests; bucket config: add `max-page-size` !6657

Python

Highlights
- cold GET: extract and micro-optimize the flow
- sync Cloud bucket
- leverage [validate-warm-GET](https://github.com/NVIDIA/aistore/blob/master/docs/validate_warm_get.md) bucket config, and
- extend it to support non-versioned Cloud buckets, and
- optionally, delete (remotely deleted) objects
- bucket sizing and counting:
- support very large buckets that are not necessarily _present_ in the cluster;
- unify `ais ls --summary` and `ais storage summary` to utilize the same control message and flags
- list, _summarize_, and lookup the properties of remote buckets without adding them to cluster's BMD
- HTTPS:
- support TLS configuration to authenticate clients
- switch cluster from HTTP to HTTPS, and vice versa
- optimize metadata cache
- optimize capacity management
- bug fixes, performance improvements

Core
- set `prime-time` to amend local generation of globally unique IDs !6325
- multi-object (archive, copy, transform) jobs: transport endpoint !6326
- core: (maintenance, decommission, shutdown) transition w/ rebalancing !6327
- core: (maintenance, decommission, shutdown) transition w/ rebalancing !6328
- intra-cluster transport: make receive-side stats optional !6329
- intra-cluster transport: reduce receive side contention !6330
- fix _channel full_ condition; rebalance-cluster; transport !6331
- feature flags: add `limited-coexistence`; transport: track closed endpoints !6334
- fix `prime-time`: add `caller-is-primary`; up cli module !6335
- switch existing cluster between HTTPS and HTTP !6336
- Go 1.21: use built-in `min` and `max` functions !6337
- `list-objects`(remote-bucket-and-only-remote-props); Go 1.21 `clear` built-in !6339
- Go 1.20: use typed atomic pointer, remove unsafe !6343
- core: assorted micro-optimizations; remove read locks !6346
- tweak multi-error `join-err`, remove error channel (minor) !6347
- [API change] capacity management !6348
- xxhash; field-align `vol` package !6349
- bucket: new-query help; silent GET; test tools !6350
- etl: adding `fqn` param to spec templates !6351
- low-level control structs: `bucket`, `namespace` !6352
- etl: Keras template fix !6355
- etl: fix hello-world ais-etl tests !6356
- core: don't recompute `uname` hash !6359
- repackage HRW methods !6361
- core: lom cache v2 (major update) !6362
- refactor: downloader's diff resolver; control plane (receive BMD) !6363
- core: lom metadata cache (cont-ed) !6365
- `dsort`: error handling, assorted cleanups, more scripted tests !6366
- core transactions: concurrency !6368
- downloader: throttle; wait !6369
- optimize cold GET !6370
- global rebalance: log; minor edits !6373
- core: update backend 'get-reader' API (all supported backends) !6374
- core: `validate-warm-get` to support non-versioned buckets, and more !6375
- `validate-warm-get` to support non-versioned buckets !6376
- [API change] silent `HEAD(object)` request !6378
- core: add `load-unsafe` (the faster way to load local metadata) !6382
- total disk size: compute at startup, recompute on change !6383
- [API change] new bucket summary; unify `list-objects` and `summary` !6384, !6386, !6387
- add `config.Rom` to consolidate assorted "read-mostly" config values; refactor and unify !6388
- [API change] new bucket summary (major update) !6390
- mountpath jogger: support bucket query !6392
- backend providers: do not include (`checksum`, `version`) if not asked to !6394
- python: updated bucket info API !6395
- feature flags: don't-add-remote & don't-head-remote; log: add s3 module; verbosity; !6398
- support listing remote buckets without adding them to cluster's BMD !6399
- concurrent HEAD(object) vs evict/create bucket - fix the race !6400
- [API change] list and _summarize_ remote buckets without adding remote buckets to cluster's BMD !6401
- datapath query (`dpq`) !6402
- Go-based API: response header to error message !6403
- [API change] new bucket summary !6405, !6406
- downloader: streamline and cleanup initialization sequence !6409
- HTTPS: support TLS configuration !6410, !6411, !6412, !6413, !6414, !6415, !6416
- assorted minor fixes !6417, !6418
- core: cold GET: fast path & slow path !6419
- cluster configuration: flip `validate-cold-get` !6420
- downloader (major update); [API change]: xaction registry !6422
- `validate-warm-get`: add scripted test utilizing remote ais cluster !6423
- core: cold GET: fast path & slow path !6424, !6427
- feature flags: add `disable-fast-cold-get`; `show performance latency`; up cli module !6425
- refactor ais/utils !6429

Bench: `aisloader` and `aisloader-composer`
- skip list objects for 100% put load !6332
- `composer`: add playbook and script for intial `aisloader` copy !6333
- `composer`: add support for `aisloader --filelist` option !6345
- default value for duration should be infinite if num-epochs value is defined !6353
- `composer`: add epochs option for GET workloads !6354
- `composer`: add cluster name prefix to `netdata` sources for easier filtering !6357
- new bucket not to be listed; usability !6358

CLI
- typed `does-not-exist` error; misc !6358
- always print `dsort` job description !6367
- `show cluster` to report total num disks !6371
- `show performance`: usability fixes, improvements !6379
- `show performance` not to filter regex-selected _zero_ columns !6380
- attempt to copy/transform an empty remote bucket !6393
- new bucket summary; evict multiple buckets in one shot; pretty print !6396, !6397
- `ais show bucket` with an option to add remote bucket to cluster's BMD (effectively, create bucket) !6404
- `ais search`: CLI command search results to include idiomatic extensions !6428

Build, CI, Deployment
- tests: upon node shutdown: wait for the node to stop (tcp) listening !6338
- CI: add gather-logs template for K8s tests !6340
- deploy: ais with HTTPS in minikube !6364
- build: bump `urllib` version !6372
- tests: `validate-warm-get` (scripted) !6423
- K8s playbooks: update kill `aisloader` command !6385
- docs: validate-warm-get; assorted !6377
- docs: add performance.md; inline help; rm all-columns flag (redundant) !6381
- build: upgrade all minors !6389
- CI: add `checkmarx` scan !6391
- build: upgrade `golangci-lint`, add linters !6407, !6408

Core
- tweak stop-maintenance logic; rebalance: cleanup log messages; assorted minor fixes !6288
- do not timestamp `err-aborted` message !6290
- [API change] dsort: remove _extended metrics_; add new counters; revise and refactor !6297
- list-objects; house-keeper; `aisloader`, logger (assorted fixes) !6298
- core stats: remove mutex and work channel - speed up !6299
- slab allocator: remove stats mutex, do not sort !6300
- consolidate and revise OOM handling !6301
- ETL: require admin access to create & delete; add feature flag !6302
- remove unused heartbeat tracker w/ minor ref !6308
- reimplement keep-alive mechanism (major) !6309
- keep-alive v2 (major update) !6312
- keep-alive v2: remove timeout stats (control structure and code) !6317
- keep-alive v2: add fast path !6320
- micro-optimize get-all-running (jobs); atomic heard-from/timed-out !6321
- node-restarted: remove 'lsof', use net dialer; fix node-decommissioning tests !6322

Tools and tests
- CI: update _fspath_ (aka _mountpath_) config for minikube-based aistore deployments !6289
- `aisloader`: list and read s3 buckets _directly_ !6291
- `aisloader`: list, read, and write s3 buckets directly !6292
- tests: K8s long tests (EchoGolang) fix !6293
- `aisloader`: fix cleanup option for s3 bucket benchmarks !6294
- `aisloader`: reimplement direct get from s3 - use SDK !6295
- `aisloader`: show progress when listing s3 directly !6296
- CLI: add show details param to etl !6304
- tools: add check for ais etl deployment !6305
- tools: add `ETL_NAME` var for CLI tests !6310
- `aisloader-composer`: add playbook and script for clearing Linux Page Cache on all AIS targets !6311
- `aisloader-composer`: add playbook for copying aws credentials !6314
- tools: update check for aistore Kubernetes deployment !6315
- CI: update github action version (all modules) !6316
- CLI/ETL: support enumerated `arg-type` !6287, !6323

Build
- upgrade all OSS packages (minor versions) !6313
- transition to Go 1.21 !6318