Meltano

✨ New

- [5892](https://github.com/meltano/meltano/issues/5892) Add `--interactive` CLI flag in `meltano config set`.
- [6640](https://github.com/meltano/meltano/issues/6640) Add rich exception formatting, configurable from `logging.yaml`.

🐛 Fixes

- [6694](https://github.com/meltano/meltano/issues/6694) Regression in `meltano remove` with error stating the plugin is missing the `settings_with_extras` attribute.
- [6652](https://github.com/meltano/meltano/issues/6652) Catch permission errors when creating a new project -- _**Thanks jakegut!**_
- [6676](https://github.com/meltano/meltano/issues/6676) Prevent warnings/errors when running `meltano config <plugin> test` -- _**Thanks ReubenFrankel!**_

📚 Documentation Improvements

- [6706](https://github.com/meltano/meltano/issues/6706) Add troubleshooting page -- _**Thanks jared-rimmer!**_
- [6688](https://github.com/meltano/meltano/issues/6688) Add docs about environment variable expansion.
- [6671](https://github.com/meltano/meltano/issues/6671) Add example meltano projects.
- [6674](https://github.com/meltano/meltano/issues/6674) Add docs for schedule `env` key.
- [6680](https://github.com/meltano/meltano/issues/6680) Fix feature flag name for strict env var mode.
- [6672](https://github.com/meltano/meltano/issues/6672) Add tutorial on how to debug a custom extractor -- _**Thanks jared-rimmer!**_

BREAKING CHANGES

- [6582](https://github.com/meltano/meltano/issues/6582) Make `default_environment` a top-level setting

Running `meltano environment add dev` as a one-time fix will resolve the issue, even if no other changes are made and no environment-specific settings are added.
_Note: This breaking change was added inadvertently in this version. We do not intend to release breaking changes in minor versions and have added this notice retroactively. _

✨ New

- [6596](https://github.com/meltano/meltano/issues/6596) Support glob paths in file plugin `update` extra
- [6578](https://github.com/meltano/meltano/issues/6578) Add `--no-install` option to `meltano add`

🐛 Fixes

- [6656](https://github.com/meltano/meltano/issues/6656) Correct STRICT_ENV_VAR_MODE behavior
- [6627](https://github.com/meltano/meltano/issues/6627) Raise exception when meltano.yml is empty
- [6643](https://github.com/meltano/meltano/issues/6643) Address `alter_column` warning for MSSQL
- [6636](https://github.com/meltano/meltano/issues/6636) Use `docs.meltano.com` instead of `meltano.com/docs`
- [6601](https://github.com/meltano/meltano/issues/6601) raise error on migration fail
- [6609](https://github.com/meltano/meltano/issues/6609) `meltano state list` change pattern from argument to option
- [6592](https://github.com/meltano/meltano/issues/6592) Validate CRON expressions
- [6543](https://github.com/meltano/meltano/issues/6543) Register legacy `DiscoveryFile` object in Ruamel YAML instance
- [6593](https://github.com/meltano/meltano/issues/6593) Docker workflow
- [6584](https://github.com/meltano/meltano/issues/6584) Avoid `pyproject.toml` Poetry parsing issue

⚙️ Under the Hood

- [6621](https://github.com/meltano/meltano/issues/6621) Make `ProjectInitService.init` generate/set a `project_id`
- [6611](https://github.com/meltano/meltano/issues/6611) Use `closing` for DB sessions & add docstrings to `meltano.core.db`
- [6610](https://github.com/meltano/meltano/issues/6610) Replace `StaleJobFailer` class with `fail_stale_jobs` function

⚡ Performance Improvements

- [6644](https://github.com/meltano/meltano/issues/6644) Improve performance when handling large yaml files

📚 Documentation Improvements

- [6663](https://github.com/meltano/meltano/issues/6663) update custom extractor tutorial to use pip_url
- [6648](https://github.com/meltano/meltano/issues/6648) logging configuration guide
- [6650](https://github.com/meltano/meltano/issues/6650) Update Custom Extractor tutorial to use an editable install
- [6631](https://github.com/meltano/meltano/issues/6631) Add "Pytest Best Practices" section to contributing docs
- [6602](https://github.com/meltano/meltano/issues/6602) Update `Docker` definition in `glossary.yml`

✨ New

- [5983](https://github.com/meltano/meltano/issues/5983) Add support for env var inheritance across config layers
- [!6167](https://github.com/meltano/meltano/pull/6167) Add systemdb support for MS SQL Server
- [6494](https://github.com/meltano/meltano/issues/6494) **core**: Report how `send_anonymous_usage_stats` was configured
- [6116](https://github.com/meltano/meltano/issues/6116) Ignore setting values when selecting store location and `store=auto`

🐛 Fixes

- [6383](https://github.com/meltano/meltano/issues/6383) Test DB Creation when meltano install is ran
- [6360](https://github.com/meltano/meltano/issues/6360) Lock plugins using the latest Hub definition if plugin has no variant name and no lock file exists
- [!6557](https://github.com/meltano/meltano/pull/6557) Remove broken unused method `PluginsTrackingContext.append_plugin_context`
- [6532](https://github.com/meltano/meltano/issues/6532) **core**: Timeout Snowplow requests
- [5982](https://github.com/meltano/meltano/issues/5982) **core**: Update env var tests to be in line with correct reading of spec
- [6189](https://github.com/meltano/meltano/issues/6189) Validating Mapper and Mappings names and alerting if necessary
- [6240](https://github.com/meltano/meltano/issues/6240) **ui**: When feature flag `enable_uvicorn` is true require write lock when accessing Project.meltano attribute
- [6469](https://github.com/meltano/meltano/issues/6469) **deps**: Pin werkzeug to <2.2.0

📚 Documentation Improvements

- [!6535](https://github.com/meltano/meltano/pull/6535) Update broken doc link in `meltano.schema.json`
- [!6456](https://github.com/meltano/meltano/pull/6456) Update for Windows and for using pipx --python version
- [6491](https://github.com/meltano/meltano/issues/6491) Update style doc to note that MyPy is not run in CI
- [6107](https://github.com/meltano/meltano/issues/6107) Document YAML syntax for Hub plugins
- [!6465](https://github.com/meltano/meltano/pull/6465) Move 'v2 migration guide' under 'guides'

✨ New

- [6150](https://github.com/meltano/meltano/issues/6150) meltano now surfaces full stderr output for catalog discovery failures
- [3419](https://github.com/meltano/meltano/issues/3419) Change default meltano config behavior to --no-environment
- [2133](https://github.com/meltano/meltano/issues/2133) Preserve comments and flow style in `meltano.yml`
- [6207](https://github.com/meltano/meltano/issues/6207) Add env as first-class attribute of MeltanoFile and ProjectPlugin classes (not an extra)

🐛 Fixes

- [6188](https://github.com/meltano/meltano/issues/6188) Allow `meltano run` to run without providing subcommand
- [6380](https://github.com/meltano/meltano/issues/6380) **core**: Extras of kind `object` not honored in inherited plugins
- [3322](https://github.com/meltano/meltano/issues/3322) Extras of kind `object` not honored in environments
- [6301](https://github.com/meltano/meltano/issues/6301) Swap add_related for add_required in api v1 plugins
- [6359](https://github.com/meltano/meltano/issues/6359) Do not attempt to lock inherited plugins when using `meltano lock`
- [6136](https://github.com/meltano/meltano/issues/6136) clarify job/task relation to airflow
- [6234](https://github.com/meltano/meltano/issues/6234) Emit `RuntimeWarning` if a setting uses the default value
- [6357](https://github.com/meltano/meltano/issues/6357) typo in `meltano-basics` example link
- [6354](https://github.com/meltano/meltano/issues/6354) removed run preview references

⚙️ Under the Hood

- [6324](https://github.com/meltano/meltano/issues/6324) update to automatically fire start AND end events

📚 Documentation Improvements

- [6283](https://github.com/meltano/meltano/issues/6283) Document overriding capabilities and extending settings
- [6420](https://github.com/meltano/meltano/issues/6420) Windows installation old gitlab link
- [6367](https://github.com/meltano/meltano/issues/6367) Fix typo etl -> elt

---

New

- [2830](https://github.com/meltano/meltano/issues/2830) Add support for Python 3.10.
- [3452](https://gitlab.com/meltano/meltano/-/issues/3452) Add `copy` and `move` commands to `meltano state`

Changes

- [3174](https://gitlab.com/meltano/meltano/-/issues/3174) We now recommend pipx over manually created virtual environments.
- [3450](https://gitlab.com/meltano/meltano/-/issues/3450) Added installation instructions for Windows.
- [3367](https://gitlab.com/meltano/meltano/-/issues/3367) Windows now covered in CI for some basic cases.
- [3135](https://github.com/meltano/meltano/issues/3135) Formalize settings alias behavior and disallow conflicting values between setting name and aliases.
- [2035](https://github.com/meltano/meltano/issues/2035) Rename `job` table and `job_id` column in backend db.

Fixes

- [5985](https://github.com/meltano/meltano/issues/5985) - Address warning in Airflow plugin version check by using recommended `packaging.version.Version`.

---

New

- [5961](https://github.com/meltano/meltano/issues/5961) - Add `meltano lock` command to create or update plugin lock artifacts.
- [6199](https://github.com/meltano/meltano/pull/6199/) - Restore project_id from analytics.json & improve analytics.json handling.
- [6203](https://github.com/meltano/meltano/pull/6203) - Fire exit event exactly once & include detailed exception context.
- [6141](https://github.com/meltano/meltano/pull/6141) - Add tracking for job command.
- [6137](https://github.com/meltano/meltano/pull/6137) - Add telemetry for meltano config.
- [6114](https://github.com/meltano/meltano/pull/6114) - Add meltano lock.

Changes

- [3473](https://gitlab.com/meltano/meltano/-/issues/3473) - Python Virtual Environments advanced concept added to docs by splitting it out from the Installation instructions.

Fixes

- [6216](https://github.com/meltano/meltano/pull/6216) - Replace all `jobId` with `stateId` to fix pipeline running bug.