Meltano

✨ New

- [8944](https://github.com/meltano/meltano/issues/8944) Suggest verifying stream selection after `meltano config ... test` fails
- [8917](https://github.com/meltano/meltano/issues/8917) Bumped the default `elt.buffer_size` to 100 MiB
- [8913](https://github.com/meltano/meltano/issues/8913) Include the Python version and machine type in system debug log message
- [8894](https://github.com/meltano/meltano/issues/8894) Fields marked as `unsupported` in the catalog are now marked as such in the output of `meltano select --list --all`
- [8889](https://github.com/meltano/meltano/issues/8889) The `args` attribute of plugin commands is now optional
- [8786](https://github.com/meltano/meltano/issues/8786) Added an `--all` flag to `meltano state clear` to delete all state IDs -- _**Thanks fauzxan!**_
- [8735](https://github.com/meltano/meltano/issues/8735) Add Content-Type to S3 state file uploads
- [8844](https://github.com/meltano/meltano/issues/8844) Support an environment variable for the `--full-refresh` flag -- _**Thanks MutuT2!**_
- [8618](https://github.com/meltano/meltano/issues/8618) Python 3.13 is now supported
- [8737](https://github.com/meltano/meltano/issues/8737) Added `exceptions` to the log message when using the JSON formatter
- [8626](https://github.com/meltano/meltano/issues/8626) Added a `--run-id` option to `meltano el[t]`
- [8494](https://github.com/meltano/meltano/issues/8494) Added a `callsite_parameters` argument to the builtin log formatters, which adds source path, line number and function name fields to the emitted log

🐛 Fixes

- [8949](https://github.com/meltano/meltano/issues/8949) Add newline at the end of plugin lock files
- [8914](https://github.com/meltano/meltano/issues/8914) Ask for confirmation in `meltano config ... reset`
- [8869](https://github.com/meltano/meltano/issues/8869) Adding User friendly loader error when the loader is missing from the Meltano run. -- _**Thanks NishitSingh2023!**_
- [8852](https://github.com/meltano/meltano/issues/8852) Using the `--force` flag of `meltano install` no longer causes plugin installation to crash when using the `uv` venv backend
- [8828](https://github.com/meltano/meltano/issues/8828) Show the correct value in the "Current value is still" warning for `meltano config set` -- _**Thanks ReubenFrankel!**_
- [8815](https://github.com/meltano/meltano/issues/8815) A mapper is now auto-installed when one of its mappings is used in a command -- _**Thanks ReubenFrankel!**_

⚙️ Under the Hood

- [8851](https://github.com/meltano/meltano/issues/8851) Decoupled the `JobState` ORM model of the system db from state store implementations
- [8898](https://github.com/meltano/meltano/issues/8898) Use `importlib.resources` to read package files and use `importlib.metadata` to detect "editable" installations (via PEP 610)
- [8830](https://github.com/meltano/meltano/issues/8830) Dropped support for EOL Python 3.8

📚 Documentation Improvements

- [8950](https://github.com/meltano/meltano/issues/8950) Document Git requirement and add `uv` section
- [8937](https://github.com/meltano/meltano/issues/8937) Use tabs for code blocks with OS-specific instructions in Getting Started guide
- [8934](https://github.com/meltano/meltano/issues/8934) Document a logging example to log at DEBUG level but excluding extractor/loader stdout
- [8909](https://github.com/meltano/meltano/issues/8909) Update part1.mdx
- [8829](https://github.com/meltano/meltano/issues/8829) Update Azure state backend connection documentation for DefaultAzureCredential -- _**Thanks acarter24!**_
- [8809](https://github.com/meltano/meltano/issues/8809) Bumped Docusaurus to 3.1, fixed broken links and started printing warnings on broken anchors
- [8752](https://github.com/meltano/meltano/issues/8752) Added 2024-Q3 changelog
- [8808](https://github.com/meltano/meltano/issues/8808) Fixed broken anchor links in settings page

📦 Packaging changes

- [8956](https://github.com/meltano/meltano/issues/8956) Bump the runtime-dependencies group with 3 updates
- [8947](https://github.com/meltano/meltano/issues/8947) Bump the runtime-dependencies group with 3 updates
- [8943](https://github.com/meltano/meltano/issues/8943) Bump the runtime-dependencies group with 2 updates
- [8931](https://github.com/meltano/meltano/issues/8931) Bump the runtime-dependencies group with 6 updates
- [8907](https://github.com/meltano/meltano/issues/8907) Bump the runtime-dependencies group with 4 updates
- [8910](https://github.com/meltano/meltano/issues/8910) Bump aiohttp from 3.10.10 to 3.10.11
- [8883](https://github.com/meltano/meltano/issues/8883) Bump the runtime-dependencies group with 4 updates
- [8873](https://github.com/meltano/meltano/issues/8873) Bump the runtime-dependencies group with 4 updates
- [8860](https://github.com/meltano/meltano/issues/8860) Bump the runtime-dependencies group with 3 updates
- [8849](https://github.com/meltano/meltano/issues/8849) Bump the runtime-dependencies group with 2 updates
- [8841](https://github.com/meltano/meltano/issues/8841) Bump boto3 from 1.35.41 to 1.35.42 in the runtime-dependencies group

🐛 Fixes

- [8802](https://github.com/meltano/meltano/issues/8802) No longer fail on uploading state to S3 when the object does not initially exist
- [8800](https://github.com/meltano/meltano/issues/8800) Failure message is now correctly printed when a plugin can not be removed from the system database

🐛 Fixes

- [8784](https://github.com/meltano/meltano/issues/8784) Inheriting plugins are no longer always auto-installed -- _**Thanks ReubenFrankel!**_
- [8785](https://github.com/meltano/meltano/issues/8785) Install plugins correctly given multiple levels of inheritance -- _**Thanks ReubenFrankel!**_

📚 Documentation Improvements

- [8779](https://github.com/meltano/meltano/issues/8779) Fix layout of troubleshooting page

🐛 Fixes

- [8770](https://github.com/meltano/meltano/issues/8770) Redact secret settings by default in `meltano compile` -- _**Thanks holly-evans!**_
- [8731](https://github.com/meltano/meltano/issues/8731) Valid options are now printed for the `--interval` option of the schedule subcommand

⚙️ Under the Hood

- [8762](https://github.com/meltano/meltano/issues/8762) Remove `record-flattening` capability in favour of `schema-flattening` -- _**Thanks ReubenFrankel!**_
- [8748](https://github.com/meltano/meltano/issues/8748) Move `install_plugins` to `meltano.core.plugin_install_service`
- [8732](https://github.com/meltano/meltano/issues/8732) Use `uv venv` instead of `uv virtualenv` alias

📚 Documentation Improvements

- [8764](https://github.com/meltano/meltano/issues/8764) Migrate to Docusaurus v3
- [8756](https://github.com/meltano/meltano/issues/8756) Updated docs to reference `meltano-dbt-ext` PyPI distribution

🐛 Fixes

- [8689](https://github.com/meltano/meltano/issues/8689) Emit a clearer error message when trying to set/unset an unknown setting in `.env`
- [8658](https://github.com/meltano/meltano/issues/8658) List only the state IDs within the specified prefix in the S3 URI
- [8698](https://github.com/meltano/meltano/issues/8698) `meltano config ... test` now uses the `elt.buffer` setting
- [8699](https://github.com/meltano/meltano/issues/8699) Add missing `project_readonly` setting to JSON schema
- [8691](https://github.com/meltano/meltano/issues/8691) Addressed `structlog` warning by removing `format_exc_info` from the processor chain

📚 Documentation Improvements

- [8721](https://github.com/meltano/meltano/issues/8721) Added instructions for installing Meltano with uv
- [8695](https://github.com/meltano/meltano/issues/8695) Fixed line highlighting and handling of `==` in code blocks

✨ New

- [8633](https://github.com/meltano/meltano/issues/8633) Added a `--only-install` option to commands that can auto-install plugins
- [8482](https://github.com/meltano/meltano/issues/8482) Plugins are now auto-installed when commands require them -- _**Thanks ReubenFrankel!**_
- [8620](https://github.com/meltano/meltano/issues/8620) Sensitive values in `meltano config <plugin>` are now hidden by default
- [8580](https://github.com/meltano/meltano/issues/8580) Added CLI `--refresh-catalog` option and extractor extra `use_cached_catalog` to ignore the cached source catalog

🐛 Fixes

- [8648](https://github.com/meltano/meltano/issues/8648) Listing Cloud (S3, GCS, etc.) state IDs no longer crashes if there is a file at the root of the bucket, and files in GCS buckets are only listed within the prefix specified in the `state_backend_uri` -- _**Thanks jx2lee!**_
- [8636](https://github.com/meltano/meltano/issues/8636) Values with `$` can now be escaped instead of trying to expand them from environment variables
- [8590](https://github.com/meltano/meltano/issues/8590) `meltano config ... test` now also checks for `BATCH` messages

⚙️ Under the Hood

- [8265](https://github.com/meltano/meltano/issues/8265) Use timezone-aware datetime objects

📚 Documentation Improvements

- [8646](https://github.com/meltano/meltano/issues/8646) Documented hot to escape setting values with `$`
- [8627](https://github.com/meltano/meltano/issues/8627) Fixed a broken link in the plugin concept page
- [8621](https://github.com/meltano/meltano/issues/8621) Update to current year and recommendations
- [8610](https://github.com/meltano/meltano/issues/8610) Remove Meltano Cloud docs
- [8609](https://github.com/meltano/meltano/issues/8609) Redirect Meltano Cloud docs to Arch docs
- [8608](https://github.com/meltano/meltano/issues/8608) Meltano Cloud has been shut down in favor of Arch -- _**Thanks hulet!**_
- [8505](https://github.com/meltano/meltano/issues/8505) Added 2024-Q2 changelog
- [8589](https://github.com/meltano/meltano/issues/8589) Recommend adding `namespace` key to custom plugins when migrating to Meltano 3
- [8578](https://github.com/meltano/meltano/issues/8578) Update `environments.md` to fix link
- [8570](https://github.com/meltano/meltano/issues/8570) Added a few dbt-postgres troubleshooting notes to the Getting Started guide
- [8566](https://github.com/meltano/meltano/issues/8566) Added a note about state IDs when using `meltano schedule run`