Quilt

Python API
* [Added] Declared compatibility with `jsonschema==4.*`.
* [Added] `--host` and `--port` parameters for `quilt3 catalog`.
* [Added] `--no-browser` parameter for `quilt3 catalog`.
* [Changed] `quilt3 catalog` now requires `quilt3` to be installed with `catalog` extra dependency (`pip install quilt3[catalog]`).
* [Changed] Dependencies on `flask` and `dnspython` are dropped.
* [Removed] Deprecated passing subpackage as part of package name for `Package.install()`, use `path` parameter instead.
* [Removed] Deprecated calling of `Package.resolve_hash()` without specifying `name` parameter.
* [Removed] Deprecated `PackageEntry.physical_keys`, use `PackageEntry.physical_key` instead.

CLI
* [Removed] Deprecated passing subpackage as part of package name for `quilt3 install`, use `--path` parameter instead.

Catalog, Lambdas
* [Changed] pkgselect: make directly invocable, always use execution role ([2560](https://github.com/quiltdata/quilt/pull/2560))
* [Changed] Move package listing / querying to GraphQL ([2552](https://github.com/quiltdata/quilt/pull/2552))
* [Changed] Fix performance of `countPages` in thumbnail lambda ([2616](https://github.com/quiltdata/quilt/pull/2616)).
* [Changed] `pkgpush` lambda now uses dedicated lambda for hashing files to push larger packages faster from catalog.
* [Changed] Local-mode-specific adjustments to landing page and catalog UI ([2611](https://github.com/quiltdata/quilt/pull/2611))
* [Changed] PDF preview: count pages again ([2621](https://github.com/quiltdata/quilt/pull/2621))
* [Changed] Make Vega download data files from S3 via proxy to avoid CORS issues ([2631](https://github.com/quiltdata/quilt/pull/2631))
* [Changed] Don't support packages in out-of-stack-buckets ([2641](https://github.com/quiltdata/quilt/pull/2641))
* [Fixed] Improve upload performance and stability, fix some hashing-related errors ([2532](https://github.com/quiltdata/quilt/pull/2532))
* [Added] Echarts renderer ([2382](https://github.com/quiltdata/quilt/pull/2382))
* [Added] Set height for `quilt_summarize.json` files ([2474](https://github.com/quiltdata/quilt/pull/2474))
* [Added] Add a "transcode" lambda for previewing video files ([2366](https://github.com/quiltdata/quilt/pull/2366/))
* [Added] Add object-level metadata editor and move package metadata editor to popup ([2510](https://github.com/quiltdata/quilt/pull/2510/))
* [Added] Video previews ([2540](https://github.com/quiltdata/quilt/pull/2540))
* [Added] Audio previews ([2547](https://github.com/quiltdata/quilt/pull/2547))
* [Added] Powerpoint (`.pptx`) preview ([2598](https://github.com/quiltdata/quilt/pull/2598), [#2626](https://github.com/quiltdata/quilt/pull/2626))

Python API
* [Added] Validation of package names with `handle_pattern` in [workflows](advanced-features/workflows.md) config.
* [Added] Validation of package entries with `entries_schema` in [workflows](advanced-features/workflows.md) config.

Python API
* [Added] Size of each manifest record is now limited by 1 MB. This constraint is added to ensure that S3 select, Athena and downstream services work correctly. This limit can be overridden with `QUILT_MANIFEST_MAX_RECORD_SIZE` environment variable. ([2114](https://github.com/quiltdata/quilt/pull/2114))
* [Changed] Decrease size of `Package` in-memory representation ([1943](https://github.com/quiltdata/quilt/pull/1943))

CLI
* [Added] `--workflow` argument to `push` command ([2279](https://github.com/quiltdata/quilt/pull/2279))

Catalog, Lambdas
* [Added] Prepopulate today date for metadata ([2121](https://github.com/quiltdata/quilt/pull/2121))
* [Added] Limit and offset parameters in pkgselect lambda ([2124](https://github.com/quiltdata/quilt/pull/2124))
* [Added] File listing: "load more" button to fetch more entries from S3 ([2150](https://github.com/quiltdata/quilt/pull/2150))
* [Added] Voila Notebooks format support ([2163](https://github.com/quiltdata/quilt/pull/2163))
* [Added] Ability to add files from S3 while revising a package ([2171](https://github.com/quiltdata/quilt/pull/2171))
* [Added] Lambdas for pushing an existing package/creation of package ([2147](https://github.com/quiltdata/quilt/pull/2147), [#2180](https://github.com/quiltdata/quilt/pull/2180))
* [Added] Custom navbar link configurable via admin UI ([2192](https://github.com/quiltdata/quilt/pull/2192))
* [Added] Adding S3 objects to packages from configurable source buckets while revising ([2193](https://github.com/quiltdata/quilt/pull/2193))
* [Added] Add Athena SQL queries initial viewer and runner ([2197](https://github.com/quiltdata/quilt/pull/2197))
* [Added] Managing per-bucket permissions via admin UI ([2228](https://github.com/quiltdata/quilt/pull/2228))
* [Added] Deep search indexing for Excel ([2247](https://github.com/quiltdata/quilt/pull/2247))
* [Added] Deep search indexing for PDF ([2256](https://github.com/quiltdata/quilt/pull/2256))
* [Added] Add Vega/JSON view switcher for Vega files ([2236](https://github.com/quiltdata/quilt/pull/2236))
* [Added] Subpackage download ([2258](https://github.com/quiltdata/quilt/pull/2258))
* [Added] Per-bucket deep indexing settings ([2290](https://github.com/quiltdata/quilt/pull/2290))
* [Added] Embed: IPC, more debug features, docs ([2314](https://github.com/quiltdata/quilt/pull/2314))
* [Added] Download button on Summary page ([2367](https://github.com/quiltdata/quilt/pull/2367))
* [Added] Validation of package names and entries, pattern for default package name ([2364](https://github.com/quiltdata/quilt/pull/2364))
* [Changed] New DataGrid-based file listing UI with arbitrary sorting and filtering ([2097](https://github.com/quiltdata/quilt/pull/2097))
* [Changed] Item selection in folder-to-package dialog ([2122](https://github.com/quiltdata/quilt/pull/2122))
* [Changed] Don't preview .tif (but keep .tiff), preview .results as plain text ([2128](https://github.com/quiltdata/quilt/pull/2128))
* [Changed] Sort packages by modification time by default ([2126](https://github.com/quiltdata/quilt/pull/2126))
* [Changed] Resolve logical keys in summaries and vega inside packages ([2140](https://github.com/quiltdata/quilt/pull/2140))
* [Changed] Embed: load polyfills, bring back prefix filtering, load more ([2153](https://github.com/quiltdata/quilt/pull/2153))
* [Changed] Scan more bytes (first 128 KiB) when trying to detect if a JSON file is a Vega visualization ([2229](https://github.com/quiltdata/quilt/pull/2229))
* [Changed] Use GraphQL for fetching and editing buckets ([2240](https://github.com/quiltdata/quilt/pull/2240))
* [Changed] Use registry for search requests ([2242](https://github.com/quiltdata/quilt/pull/2242))
* [Changed] Enhance `quilt_summarize.json` format, support title, description and multi-column layout ([2245](https://github.com/quiltdata/quilt/pull/2245))
* [Changed] PDF preview: don't count pages ([2276](https://github.com/quiltdata/quilt/pull/2276))
* [Changed] Default bucket icon changed from Quilt logo to more neutral ([2287](https://github.com/quiltdata/quilt/pull/2287))
* [Changed] Cachebust revision list request ([2298](https://github.com/quiltdata/quilt/pull/2298))
* [Changed] Wrap wide Vega charts with horizontal scroll ([2304](https://github.com/quiltdata/quilt/pull/2304))
* [Changed] Unify package creation and update dialogs (support adding S3 files in both) ([2302](https://github.com/quiltdata/quilt/pull/2302))
* [Changed] Warmer chart colors ([2329](https://github.com/quiltdata/quilt/pull/2329), [#2338](https://github.com/quiltdata/quilt/pull/2338))
* [Fixed] `UnicodeDecodeError` in indexer and pkgselect lambdas ([2123](https://github.com/quiltdata/quilt/pull/2123))
* [Fixed] Catch and display package-related errors properly ("no such package" and "bad revision") ([2160](https://github.com/quiltdata/quilt/pull/2160))
* [Fixed] Crashing `pkgselect` lambda's folder view on an empty manifest ([2147](https://github.com/quiltdata/quilt/pull/2147))
* [Fixed] Infinite spinner on logout ([2232](https://github.com/quiltdata/quilt/pull/2232))
* [Fixed] Dismiss error page when navigating from it ([2291](https://github.com/quiltdata/quilt/pull/2291))
* [Fixed] Avoid crash on non-existent logical keys in pkgselect detail view ([2307](https://github.com/quiltdata/quilt/pull/2307)
* [Fixed] Error while rendering a preview inside a package ([2328](https://github.com/quiltdata/quilt/pull/2328))
* [FIxed] Bring back missing username at Admin/Users table ([2339](https://github.com/quiltdata/quilt/pull/2339))

Python API
* [Added] `QUILT_TRANSFER_MAX_CONCURRENCY` environment variable ([2092](https://github.com/quiltdata/quilt/pull/2092))
* [Added] `QUILT_DISABLE_CACHE` environment variable ([2091](https://github.com/quiltdata/quilt/pull/2091))
* [Added] Support for callable `dest` parameter in `Package.push()` ([2095](https://github.com/quiltdata/quilt/pull/2095))
* [Changed] Removed unused dependency on `packaging` ([2090](https://github.com/quiltdata/quilt/pull/2090))
* [Fixed] Possible downloading of truncated manifests ([1977](https://github.com/quiltdata/quilt/pull/1977))
* [Fixed] `TypeError` on import when running with `PYTHONOPTIMIZE=2` ([2102](https://github.com/quiltdata/quilt/pull/2102))

Catalog, Lambdas
* [Added] Support for EventBridge S3 events to es/indexer ([1987](https://github.com/quiltdata/quilt/pull/1987))
* [Added] Generate and resolve Quilt package URIs ([1935](https://github.com/quiltdata/quilt/pull/1935))
* [Added] Buttons for copying canonical package URIs ([1990](https://github.com/quiltdata/quilt/pull/1990))
* [Added] Additional validation for package name ([1998](https://github.com/quiltdata/quilt/pull/1998))
* [Added] Populate package name with username prefix ([2016](https://github.com/quiltdata/quilt/pull/2016))
* [Added] Link from bucket overview page to bucket settings ([2022](https://github.com/quiltdata/quilt/pull/2022))
* [Added] Folder to package dialog ([2040](https://github.com/quiltdata/quilt/pull/2040))
* [Added] Search lambda: `freeform` action API and UI ([2087](https://github.com/quiltdata/quilt/pull/2087), [#2088](https://github.com/quiltdata/quilt/pull/2088))
* [Added] Spreadsheets Drag'n'Drop to Metadata ([2094](https://github.com/quiltdata/quilt/pull/2094))
* [Changed] Tree view for files in package update dialog ([1989](https://github.com/quiltdata/quilt/pull/1989))
* [Changed] Lambda indexing retry logic to not fail content extraction ([2007](https://github.com/quiltdata/quilt/pull/2007))
* [Changed] Number of retries per ES and S3 failure in indexing Lambda ([1987](https://github.com/quiltdata/quilt/pull/1987))
* [Changed] Handle delete markers in ES ([1997](https://github.com/quiltdata/quilt/pull/1997), [#2000](https://github.com/quiltdata/quilt/pull/2000), [#2003](https://github.com/quiltdata/quilt/pull/2003), [#2017](https://github.com/quiltdata/quilt/pull/2017), [#2023](https://github.com/quiltdata/quilt/pull/2023))
* [Changed] Two-column layout for package dialogs ([2001](https://github.com/quiltdata/quilt/pull/2001))
* [Changed] Show Schema validation errors in text mode ([2010](https://github.com/quiltdata/quilt/pull/2010))
* [Changed] Toolchain: use webpack5 ([2036](https://github.com/quiltdata/quilt/pull/2036)) and TypeScript ([#2043](https://github.com/quiltdata/quilt/pull/2043)), ditch babel
* [Changed] Use polyfill.io instead of bundled polyfills ([2043](https://github.com/quiltdata/quilt/pull/2043))
* [Changed] Render JSON with custom tree-like viewer ([2037](https://github.com/quiltdata/quilt/pull/2037))
* [Changed] Set JSON Schema defaults ([2053](https://github.com/quiltdata/quilt/pull/2053))
* [Changed] Don't upload unmodified files while revising a package ([2080](https://github.com/quiltdata/quilt/pull/2080))
* [Fixed] Bug that caused search to miss delete object and delete package events ([1987](https://github.com/quiltdata/quilt/pull/1987))
* [Fixed] lambda previews for time series `AICSImage` data (potential `IndexError` if odd number of time points) ([1945](https://github.com/quiltdata/quilt/pull/1945))
* [Fixed] Handle folders in search results ([1992](https://github.com/quiltdata/quilt/pull/1992), [#1994](https://github.com/quiltdata/quilt/pull/1994))
* [Fixed] Use EncodingType=url for S3 list requests to handle special chars in keys / prefixes ([2026](https://github.com/quiltdata/quilt/pull/2026))
* [Fixed] Empty response from `pkgselect` folder view for packages with all non-string logical_keys or physical_keys ([1947](https://github.com/quiltdata/quilt/pull/1947))
* [Fixed] "Download Directory" bug that caused 502 or failed downloads when any files contained the "+" character ([2067](https://github.com/quiltdata/quilt/pull/2067/))
* [Added] Ability to log in with Microsoft Azure Active Directory
via OIDC ([2089](https://github.com/quiltdata/quilt/pull/2089))

Python API
* [Added] Metadata quality API ([1855](https://github.com/quiltdata/quilt/pull/1874)). For details see this [section](advanced-features/workflows.md).
* [Changed] Improved formatting of package load progress bar ([1897](https://github.com/quiltdata/quilt/pull/1897))
* [Fixed] Crash during load of package manifest with unicode symbols with non-unicode locale set ([1931](https://github.com/quiltdata/quilt/pull/1931))

Catalog, Lambdas
* [Added] Ad hoc package updates ([1856](https://github.com/quiltdata/quilt/pull/1856))
* [Added] Copy packages from one bucket to another ([1932](https://github.com/quiltdata/quilt/pull/1932))
* [Added] Enhanced JSON editor and schema validation for package metadata ([1867](https://github.com/quiltdata/quilt/pull/1867))
* [Added] Preview .pdbqt files as plain text ([1855](https://github.com/quiltdata/quilt/pull/1855))
* [Added] Retry logic for failed queries, minimize load on ES for sample, images overviews ([1864](https://github.com/quiltdata/quilt/pull/1864/))
* [Added] Buttons to download packages and directories as .zip archives ([1868](https://github.com/quiltdata/quilt/pull/1868/))
* [Added] Search help dropdown for the index landing page ([1838](https://github.com/quiltdata/quilt/pull/1838))
* [Changed] Get package revisions from ElasticSearch, not S3 ([1851](https://github.com/quiltdata/quilt/pull/1851))
* [Changed] Render vega specs smaller than 20 MiB right away, render larger ones after pressing a button ([1873](https://github.com/quiltdata/quilt/pull/1873))
* [Changed] Prefix filtering for directory view similar to AWS Console ([1876](https://github.com/quiltdata/quilt/pull/1876), [#1880](https://github.com/quiltdata/quilt/pull/1880))
* [Changed] Preview `*notes` files as plain text ([1896](https://github.com/quiltdata/quilt/pull/1896))
* [Changed] Default search operator to "AND" (was "OR") for more precise searches ([1924](https://github.com/quiltdata/quilt/pull/1924))
* [Changed] `top_hash`-based package routes (timestamp routes are still supported in the same way) ([1938](https://github.com/quiltdata/quilt/pull/1938))
* [Fixed] Incomplete package stats for empty packages in es/indexer Lambda ([1869](https://github.com/quiltdata/quilt/pull/1869))
* [Fixed] Slow parquet preview rendering (and probably other occurrences of JsonDisplay) ([1878](https://github.com/quiltdata/quilt/pull/1878))

Python API
* [Performance] 2X to 5X faster multi-threaded hashing of S3 objects ([1816](https://github.com/quiltdata/quilt/issues/1816), [#1788](https://github.com/quiltdata/quilt/issues/1788))
* [Fixed] Bump minimum required version of tqdm. Fixes a crash (`UnseekableStreamError`) during upload retry. ([1853](https://github.com/quiltdata/quilt/issues/1853))

CLI
* [Added] `--meta` argument to `push` ([1793](https://github.com/quiltdata/quilt/issues/1793))
* [Fixed] Crash in `list-packages` ([1852](https://github.com/quiltdata/quilt/issues/1852))

Catalog, Lambdas
* [Added] Ability to preview larger Jupyter notebooks; warning when cells are elided ([1823](https://github.com/quiltdata/quilt/issues/1823), [#1822](https://github.com/quiltdata/quilt/issues/1822))
* [Added] Object size to package browsing experience in catalog ([1744](https://github.com/quiltdata/quilt/issues/1744))
* [Added] Total number of packages to catalog Overview tab ([1808](https://github.com/quiltdata/quilt/issues/1808))
* [Added] PDF and other file formats, including .gz ones, to catalog Overview tab
* [Added] Drag-n-drop package creation ([1786](https://github.com/quiltdata/quilt/pull/1786))
* [Added] Glacier support ([1794](https://github.com/quiltdata/quilt/pull/1794), [#1796](https://github.com/quiltdata/quilt/pull/1796))
* [Added] Show package metadata ([1806](https://github.com/quiltdata/quilt/pull/1806))
* [Added] Search facets help ([1828](https://github.com/quiltdata/quilt/pull/1828/))
* [Added] Admin/buckets: re-index and repair ([1824](https://github.com/quiltdata/quilt/pull/1824))
* [Changed] Case-insensitive package filtering ([1807](https://github.com/quiltdata/quilt/pull/1807))
* [Changed] Show PDFs in bucket overviews ([1811](https://github.com/quiltdata/quilt/pull/1811))
* [Changed] Admin/buckets: adjust SNS ARN input, make it possible to not subscribe to a topic ([1824](https://github.com/quiltdata/quilt/pull/1824))
* [Changed] Show "push package" button when there's no packages in a bucket ([1843](https://github.com/quiltdata/quilt/pull/1843))
* [Fixed] Bugs involving bad or missing package stats during S3 Select calls ([1829](https://github.com/quiltdata/quilt/issues/1829))
* [Fixed] Overly aggressive 40X retry logic in es/indexer ([1804](https://github.com/quiltdata/quilt/issues/1804))
* [Fixed] Semantic bugs in ElasticSearch timeouts (queries now time out properly) ([1803](https://github.com/quiltdata/quilt/issues/1803))
* [Fixed] Missing Helvetica issues for PDFs ([1792](https://github.com/quiltdata/quilt/issues/1792))
* [Fixed] Bulletproof file downloads via HTTP header override ([1787](https://github.com/quiltdata/quilt/pull/1787))
* [Fixed] Previews not rendering in global search ([1787](https://github.com/quiltdata/quilt/pull/1787))