Meltano

---

New

- [2355](https://gitlab.com/meltano/meltano/-/issues/2355) Add `meltano elt` `--dump` option with possible values `catalog`, `state`, `extractor-config`, and `loader-config` to dump content of pipeline-specific generated file

Fixes

- [2358](https://gitlab.com/meltano/meltano/-/issues/2358) Don't unintentionally deselect all attributes other than those marked `inclusion: automatic` when using extractor `select_filter` extra or `meltano elt`'s `--select <entity>` option

---

New

- [2291](https://gitlab.com/meltano/meltano/-/issues/2291) Add `catalog` extractor extra to allow a catalog to be provided manually
- [2291](https://gitlab.com/meltano/meltano/-/issues/2291) Add `--catalog` option to `meltano elt` to allow a catalog to be provided manually
- [2289](https://gitlab.com/meltano/meltano/-/issues/2289) Add `state` extractor extra to allow state file to be provided manually
- [2289](https://gitlab.com/meltano/meltano/-/issues/2289) Add `--state` option to `meltano elt` to allow state file to be provided manually

Fixes

- [2352](https://gitlab.com/meltano/meltano/-/issues/2352) `meltano elt` `--select` and `--exclude` no longer unexpectedly select entities for extraction that match the wildcard pattern but weren't selected originally.

---

New

- [2279](https://gitlab.com/meltano/meltano/-/issues/2279) Populate primary setting env var and aliases when invoking plugin or expanding env vars
- [2349](https://gitlab.com/meltano/meltano/-/issues/2349) Allow plugin setting (default) values to reference pipeline plugin extras using generic env vars, e.g. `MELTANO_EXTRACT__<EXTRA>`
- [2281](https://gitlab.com/meltano/meltano/-/issues/2281) Allow plugin extra (default) values to reference plugin name, namespace, profile using generic env vars, e.g. `MELTANO_EXTRACTOR_NAMESPACE`
- [2280](https://gitlab.com/meltano/meltano/-/issues/2280) Allow plugin extra (default) values to reference plugin settings using env vars, e.g. `target_schema: $PG_SCHEMA`
- [2278](https://gitlab.com/meltano/meltano/-/issues/2278) Read setting values from `<PLUGIN_NAME>_<SETTING_NAME>` env vars, taking precedence over `<PLUGIN_NAMESPACE>_<SETTING_NAME>` but not custom `env`
- [2350](https://gitlab.com/meltano/meltano/-/issues/2350) Add `MELTANO_TRANSFORM_*` transform pipeline env vars for transformer (configuration) to access
- [2282](https://gitlab.com/meltano/meltano/-/issues/2282) Add new extractor extra `load_schema` and use it as default loader `schema` instead of namespace
- [2284](https://gitlab.com/meltano/meltano/-/issues/2284) Add new loader extra `dialect` and use it as default dbt `target` and Meltano UI SQL dialect instead of namespace
- [2283](https://gitlab.com/meltano/meltano/-/issues/2283) Add new loader extra `target_schema` and use it as default dbt `source_schema` instead of loader `schema`
- [2285](https://gitlab.com/meltano/meltano/-/issues/2285) Add new transform extra `package_name` and use it in dbt's `dbt_project.yml` and `--models` argument instead of namespace

Changes

- [2279](https://gitlab.com/meltano/meltano/-/issues/2279) Fall back on setting values from `<PLUGIN_NAME>_<SETTING_NAME>` and `<PLUGIN_NAMESPACE>_<SETTING_NAME>` env vars if a custom `env` is defined but not used
- [2278](https://gitlab.com/meltano/meltano/-/issues/2278) Stop unnecessarily prepopulating `env` on a newly added custom plugin's settings definitions
- [2208](https://gitlab.com/meltano/meltano/-/issues/2208) Standardize on setting env vars prefixed with plugin name, not namespace or custom `env`

---

New

- [2340](https://gitlab.com/meltano/meltano/-/issues/2340) Print full error message when extractor catalog discovery fails
- [2223](https://gitlab.com/meltano/meltano/-/issues/2223) Print clear error message when `meltano invoke` or `meltano elt` attempts to execute a plugin that hasn't been installed yet
- [2345](https://gitlab.com/meltano/meltano/-/issues/2345) Make `tap-csv` `files` a known setting
- [2155](https://gitlab.com/meltano/meltano/-/issues/2155) Add `select_filter` extractor extra to allow extracting a subset of selected entities
- [2155](https://gitlab.com/meltano/meltano/-/issues/2155) Add `--select` and `--exclude` options to `meltano elt` to allow extracting a subset of selected entities

Changes

- [2167](https://gitlab.com/meltano/meltano/-/issues/2167) Include extractor and loader name in autogenerated `meltano elt` job ID
- [2343](https://gitlab.com/meltano/meltano/-/issues/2343) Automatically delete generated plugin config files at end of `meltano elt` and `meltano invoke`

Fixes

- [2167](https://gitlab.com/meltano/meltano/-/issues/2167) Make sure autogenerated `meltano elt` job ID matches in system database and `.meltano/{run,logs}/elt`
- [2347](https://gitlab.com/meltano/meltano/-/issues/2347) Have `meltano config <plugin> set --store=dotenv` store valid JSON values for arrays and objects
- [2346](https://gitlab.com/meltano/meltano/-/issues/2346) Correctly cast environment variable string value when overriding custom array and object settings

Breaks

- [2344](https://gitlab.com/meltano/meltano/-/issues/2344) Move `meltano elt` output logs from `.meltano/run/elt` to `.meltano/logs/elt`
- [2342](https://gitlab.com/meltano/meltano/-/issues/2342) Store pipeline-specific generated plugin config files (`tap.config.json`, `tap.properties.json`, etc) under `.meltano/run/elt/<job_id>/<run_id>` instead of `.meltano/run/<plugin_name>`. Users who were explicitly putting a catalog file at `.meltano/run/<plugin_name>/tap.properties.json` should use `.meltano/extractors/<plugin_name>/tap.properties.json` instead.

---

New

- [2210](https://gitlab.com/meltano/meltano/-/issues/2210) Print documentation and repository URLs when adding a new plugin to the project
- [2277](https://gitlab.com/meltano/meltano/-/issues/2277) Add `tap-bing-ads` as a known extractor
- [2328](https://gitlab.com/meltano/meltano/-/issues/2328) Add new `upcase_string` setting `value_processor` that will convert string values to uppercase

Changes

- [2216](https://gitlab.com/meltano/meltano/-/issues/2216) Add stream properties defined in an extractor's `schema` extra to catalog if they do not exist in the discovered stream schema yet

Fixes

- [2338](https://gitlab.com/meltano/meltano/-/issues/2338) Once again change `target-csv` to use `singer-io/target-csv` instead of the Meltano fork
- [2235](https://gitlab.com/meltano/meltano/-/issues/2235) Make embed links accessible when not authenticated
- [2328](https://gitlab.com/meltano/meltano/-/issues/2328) Always convert `target-snowflake` `schema` setting value to uppercase before passing it to plugin

---

New

- [!1820](https://gitlab.com/meltano/meltano/-/merge_requests/1820) Add 'tap-spreadsheets-anywhere' as an extractor

Changes

- [2309](https://gitlab.com/meltano/meltano/-/issues/2309) Pretty print `meltano schedule list --format=json` output
- [2312](https://gitlab.com/meltano/meltano/-/issues/2312) Don't unnecessarily run discovery and generate catalog when running `meltano invoke <extractor> --help`, making it less likely to fail