Mage-ai

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.11 -->


🎉 Exciting New Features

✨ Base path configuration for Mage

There's a new environment variable in town— `MAGE_BASE_PATH`! 🤠

Mage now supports adding a prefix to a Mage URL, i.e. `localhost:6789/my_prefix/`

_by dy46 in <https://github.com/mage-ai/mage-ai/pull/3141>_

✨ Support for `DROP TABLE` in raw SQL blocks

Raw SQL blocks can now drop tables! 🎉

_by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3184>_

✨ MongoDB connection string support

Our MongoDB users will be happy about this one— MongoDB can now be accessed via a connection string, for example: `mongodb+srv://{username}:{password}{host}`

Doc: <https://docs.mage.ai/integrations/databases/MongoDB#add-credentials>

_by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3188>_

✨ Salesforce destination

Data integration pipelines just got _another_ great destination— Salesforce!

_by Luishfs in <https://github.com/mage-ai/mage-ai/pull/2772>_

🐛 Bug Fixes

* Fix circular dependency and repo config issues by dy46 in <https://github.com/mage-ai/mage-ai/pull/3168>
* Fix an issue in the bigquery init method by csharplus in <https://github.com/mage-ai/mage-ai/pull/3171>
* Fix base path frontend_dist creation by dy46 in <https://github.com/mage-ai/mage-ai/pull/3182>
* Fix the `bool` data type conversion issue with the ClickHouse exporter by csharplus in <https://github.com/mage-ai/mage-ai/pull/3172>
* Do not start pipeline run immediately after creation by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3173>

💅 Enhancements & Polish

* Add block runtime chart for all blocks by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3156>
* Use Playwright for web UI test automation by erictse in <https://github.com/mage-ai/mage-ai/pull/3075>
* Auto-scroll to newest logs by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3170>
* Add `DISABLE_TERMINAL` environment variable by juancaven1988 in <https://github.com/mage-ai/mage-ai/pull/3174>
* Consolidate AWS region variables by dy46 in <https://github.com/mage-ai/mage-ai/pull/3152>
* Adding test connection to MongoDB destination by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3159>
* Adding test connection to clickhouse destination by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3158>
* Adding snowflake table name option and regex by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3186>
* Remove streaming S3 sink timer error by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3190>
* Include `frontend_dist_base_path_template` in package by dy46 in <https://github.com/mage-ai/mage-ai/pull/3178>
* New options to pass Azure Blob `upload_kwargs` to allow overwriting an existing file or use any other options by sumanshusamarora in <https://github.com/mage-ai/mage-ai/pull/3148>

🎉 New Contributors

* Radu-D made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3179>
* juancaven1988 made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3174>

**Full Changelog**: <https://github.com/mage-ai/mage-ai/compare/0.9.10...0.9.11>

<img src="https://github.com/mage-ai/mage-ai/assets/59450879/5ea17901-b85b-4205-8f35-9cbfedfed99c" width="500" />

What's Changed
🎉 Exciting New Features

🤖 Create blocks and documentation using LLMs

Block Creation

<img src="https://github.com/mage-ai/mage-ai/assets/59450879/bd7aaf82-a162-4546-9bef-f4eb1d2b6892" width="600" />

Document Generation

<img src="https://github.com/mage-ai/mage-ai/assets/1066980/112790d5-ef16-45c1-a537-a47d678b952d" width="600" />

From the following PRs:
* Generate block using AI by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3095
* Generate documentation using AI by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3105
* Generate block template based on block description by matrixstone in https://github.com/mage-ai/mage-ai/pull/3029
* Create model and API endpoint for global data products by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3135

❄️ Enable batch upload for Snowflake destination

Leveraging `write_pandas` in the `snowflake-connector-python` library, this feature enhances the speed of batch uploads using Snowflake destinations 🤯 _by csharplus in https://github.com/mage-ai/mage-ai/pull/2896_

Auto-delete logs after retention period

Now, Mage can auto-remove logs after your retention period expires!
Configure retention_period in logging_config:
yml
logging_config:
retention_period: '15d'


Run command to delete old logs:
bash
mage clean-old-logs k8s_project


_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3139_

MongoDB destination support (data integration)

MongoDB is now supported as a destination! 🎉 _by Luishfs in https://github.com/mage-ai/mage-ai/pull/3084_

Pipeline-level concurrency

It's now possible to configure concurrency at the pipeline level:

yml
concurrency_config:
block_run_limit: 1
pipeline_run_limit: 1

Doc: https://docs.mage.ai/design/data-pipeline-management#pipeline-level-concurrency
_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3112_

🧱 New add-block flow



Mage's UI has been improved to feature a _new_ add-block flow! _by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3094, https://github.com/mage-ai/mage-ai/pull/3074, & https://github.com/mage-ai/mage-ai/pull/3106_

Custom k8s executors

Mage now support custom k8s executor configuration:

yml
k8s_executor_config:
service_account_name: mageai
job_name_prefix: "{{ env_var('KUBE_NAMESPACE') }}"
container_config:
image: mageai/mageai:0.9.7
env:
- name: USER_CODE_PATH
value: /home/src/k8s_project


_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3127_

Custom s3 `endpoint_url` in logger

You can now configure a custom `endpoint_url` in s3 loggers, allowing you to customize how messages are displayed!

yml
logging_config:
type: s3
level: INFO
destination_config:
bucket: <bucket name>
prefix: <prefix path>
aws_access_key_id: <(optional) AWS access key ID>
aws_secret_access_key: <(optional) AWS secret access key>
endpoint_url: <(optional) custom endpoint url>


_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3137_

Render text/html from block output

Text and HTML from block output is now rendered!


_by dy46 in https://github.com/mage-ai/mage-ai/pull/3079_

Clickhouse data integration support

Clickhouse is now supported as a integrations destination! _by Luishfs in https://github.com/mage-ai/mage-ai/pull/3005_

Custom timeouts for ECS tasks

You can now set custom timeouts for all of your ECS tasks! _by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3144_

Run multiple Mage instances with the same PostgreSQL databases

_A single_ Postgres database can now support multiple Mage instances ✨ _by csharplus in https://github.com/mage-ai/mage-ai/pull/3070_

🐛 Bug Fixes
* Clear pipeline list filters when clicking Defaults by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3092
* Fetch Snowflake role by mattppal in https://github.com/mage-ai/mage-ai/pull/3100
* Add service argument to OracleDB data loader by mattppal in https://github.com/mage-ai/mage-ai/pull/3032
* Fix `pymssql` dependency by dy46 in https://github.com/mage-ai/mage-ai/pull/3114
* Include runs without associated triggers in pipeline run count by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3118
* Fix positioning of nested flyout menu 4 levels deep by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3125
* Batch git sync in `pipeline_scheduler` by dy46 in https://github.com/mage-ai/mage-ai/pull/3102
* Only try to interpolate variables if they are in the query by dy46 in https://github.com/mage-ai/mage-ai/pull/3142
* Fix command for running streaming pipeline in k8s executor. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3093
* Fix block flow bugs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3094
* Catch timeout exception for test_connection by dy46 in https://github.com/mage-ai/mage-ai/pull/3143
* Fix showing duplicate templates for v1 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3128
* `check_status` method bug fix to look at pipelines ran between specific time period by sumanshusamarora in https://github.com/mage-ai/mage-ai/pull/3115
* Fix db init on start by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3080
* Fix logic for making API requests for block outputs or analyses that were throwing errors due to invalid `pipeline_uuids` by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3090

💅 Enhancements & Polish
* Decrease variables response size by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3097
* Cancel block runs when pipeline run fails by dy46 in https://github.com/mage-ai/mage-ai/pull/3096
* Clean up code in `VariableResource` method by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3099
* Import block function from file for test execution by dy46 in https://github.com/mage-ai/mage-ai/pull/3110
* Misc UI improvements by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3133
* Polish custom templates by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3120
* Throw correct exception about `io-config` by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3130
* Clarify input for dbt profile target if a default target is not configured by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3077
* Support nested sampling on output by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3086
* Add API endpoint for fetching OAuth access tokens by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3089

New Contributors
* sumanshusamarora made their first contribution in https://github.com/mage-ai/mage-ai/pull/3115

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.8...0.9.10

<!-- Release notes generated using configuration in .github/release.yml at master -->


What's Changed

🎉 Exciting New Features

🔥 Custom block & pipeline templates

With this release, Magers now have the option to create _pipeline_ templates, then use those to populate new pipelines.


Additionally, you may now browse, create, and use _custom_ block templates in your pipelines. 🎉


_by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3064>, <https://github.com/mage-ai/mage-ai/pull/3042> and <https://github.com/mage-ai/mage-ai/pull/3065>_

🛟 Save pipelines list filters and groups

Your pipeline filters and groups are now sticky— that means setting filters/groups will persist through your Mage session.


_by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3059>_

Horizontal scaling support for schedulers

You can now run the web server and scheduler in separate containers, allowing for horizontal scaling of the scheduler! Read more in [our docs](https://docs.mage.ai/production/deploying-to-cloud/architecture#scale-web-server-and-schedulers).

Run scheduler only:

bash
/app/run_app.sh mage start project_name --instance-type scheduler


Run web server only:

bash
/app/run_app.sh mage start project_name --instance-type web_server


_by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3016>_

🎏 Run parallel integration streams

Data integration streams may now be executed in parallel!


_by dy46 in <https://github.com/mage-ai/mage-ai/pull/1474>_

Update secrets management backend

The secrets management backend can now handle multiple environments!

_by dy46 in <https://github.com/mage-ai/mage-ai/pull/3000>_

📆 Include interval datetimes and intervals in block variables

Interval datetimes and durations are now returned in block variables. Check out [our docs](https://docs.mage.ai/guides/schedule-pipelines#extra-runtime-variables-from-pipeline-run) for more info!


_by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3058>_ and <https://github.com/mage-ai/mage-ai/pull/3068>

🐛 Bug Fixes

* Persist error message popup when selecting stream by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3028>
* Add google oauth2 scopes to bigquery client by mattppal in <https://github.com/mage-ai/mage-ai/pull/3023>
* enable backfills `delete` and add documentation by mattppal in <https://github.com/mage-ai/mage-ai/pull/3012>
* Show warning if no streams displayed by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3034>
* Fix SLA alerting as soon as trigger run starts by dy46 in <https://github.com/mage-ai/mage-ai/pull/3036>
* Display error when fetching data providers by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3038>
* Add `/files` route to backend server and fix TypeError in FileEditor by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3041>
* Create dynamic block runs in block executor by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3035>
* Fix block settings not updating after saving by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3021>
* Catch redis error and fix logging by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3047>
* Fix roles query by entity bug by dy46 in <https://github.com/mage-ai/mage-ai/pull/3049>
* API quickstart for rapid development by mattppal in <https://github.com/mage-ai/mage-ai/pull/3017>
* Fix pipelines list filters and group bys not removing by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3060>
* Fix deleting block by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3011>
* Temporary pymssql fix v2 by dy46 in <https://github.com/mage-ai/mage-ai/pull/3015>

💅 Enhancements & Polish

* Add `Updated At` column to Pipelines list page by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3050>
* Default error log to Errors tab when opening log detail by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3052>
* Add timeout for fetching files for git by dy46 in https://github.com/mage-ai/mage-ai/pull/3051
* Encode URI component for file path on files page by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3067>
* When clicking a file in file browser, select block if in pipeline by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3024>

New Contributors

* Hikari made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3037>

**Full Changelog**: <https://github.com/mage-ai/mage-ai/compare/0.9.4...0.9.8>


🎉 Features

Azure Data Lake streaming pipeline support

************Docs:************ https://docs.mage.ai/guides/streaming/destinations/azure_data_lake

Mage now supports Azure Data Lake as a streaming destination!


Pipeline Tags

Tags can now be applied to pipelines. Users can leverage the pipeline view to apply filters or group pipelines by tag.



Support for custom k8s executor job prefixes

You can now prefix your k8s executor jobs! Here’s an example k8s executor config file:

yaml
k8s_executor_config:
job_name_prefix: data-prep
resource_limits:
cpu: 1000m
memory: 2048Mi
resource_requests:
cpu: 500m
memory: 1024Mi
service_account_name: default


See [the documentation](https://docs.mage.ai/production/configuring-production-settings/compute-resource#kubernetes-executor) for further details.

Removed data integration config details from logs

Mage no longer prints data integration settings in logs: a big win for security. 🔒


💅 Other bug fixes & polish

Cloud deployment
- Fix k8s job deletion error.
- Fix fetching AWS events while editing trigger.
- Fixes for Azure deployments.
- Integrate with Azure Key Vault to support `azure_secret_var` syntax: [docs](https://docs.mage.ai/production/configuring-production-settings/secrets#azure-key-vault).
- Pass resource limits from main ECS task to dev tasks.
- Use network configuration from main ECS service as a template for dev tasks.

Integrations
- Fix Postgres schema resolution error— this fixes schema names with characters like hyphens for Postgres.
- Escape reserved column names in Postgres.
- Snowflake strings are now casted as `VARCHAR` instead of `VARCHAR(255)`. The MySQL loader now uses `TEXT` for strings to avoid truncation.
- Use AWS session token in io.s3.
- Fix the issue with database setting when running ClickHouse in SQL blocks

Other
- Fix multiple upstream block callback error. Input variables will now be fetched one block at a time.
- Fix data integration metrics calculation.
- Improved variable serialization/deserialization— this fixes kernel crashes due to OOM errors.
- User quote: "A pipeline that was taking ~1h runs in less than 2 min!"
- Fix trigger edit bug— eliminates bug that would reset fields in trigger.
- Fix default filepath in ConfigFileLoader (Thanks Ethan!)
- Move `COPY` step to reduce Docker build time.
- Validate env values in trigger config.
- Fix overview crashing.
- Fix cron settings when editing in trigger.
- Fix editing pipeline’s executor type from settings.
- Fix block pipeline policy issue.

🗣️ Shout outs

- ethanbrown3 made their first contribution in https://github.com/mage-ai/mage-ai/pull/2976 🎉
- erictse made their first contribution in https://github.com/mage-ai/mage-ai/pull/2977 🥳


Workspace management

You can use Mage with multiple workspaces in the cloud now. Mage has a built in workspace manager that can be enabled in production. This feature is similar to the **[multi-development environments](https://docs.mage.ai/developing-in-the-cloud/cloud-dev-environments/overview)**, but there are settings that can be shared across the workspaces. For example, the project owner can set workspace level permissions for users. The current additional features supported are:

- workspace level permissions
- workspace level git settings

Upcoming features:

- common workspace metadata file
- customizable permissions and roles
- pipeline level permissions

Doc: https://docs.mage.ai/developing-in-the-cloud/workspaces/overview



**Pipeline monitoring dashboard**

Add "Overview" page to dashboard providing summary of pipeline run metrics and failures.


**Version control application**

Support all Git operations through UI. Authenticate with GitHub then pull from a remote repository, push local changes to a remote repository, and create pull requests for a remote repository.

Doc: https://docs.mage.ai/production/data-sync/github


N**ew Relic monitoring**

- Set the `ENABLE_NEW_RELIC` environment variable to enable or disable new relic monitoring.
- User need to follow new relic guide to create configuration file with license_key and app name.

Doc: https://docs.mage.ai/production/observability/newrelic



Authentication

**Active Directory OAuth**

Enable signing in with Microsoft Active Directory account in Mage.

Doc: https://docs.mage.ai/production/authentication/microsoft


LDAP

https://docs.mage.ai/production/authentication/overview#ldap

- Update default LDAP user access from editor to no access. Add an environment variable `LDAP_DEFAULT_ACCESS` so that the default access can be customized.

**Add option to sync from Git on server start**

There are two ways to configure Mage to sync from Git on server start

- Toggle `Sync on server start up` option in Git settings UI
- Set `GIT_SYNC_ON_START` environment variable (options: 0 or 1)

Doc: https://docs.mage.ai/production/data-sync/git#git-settings-as-environment-variables


Data integration pipeline

M**ode Analytics Source**

Shout out to [Mohamad Balouza](https://github.com/mohamad-balouza) for his contribution of adding the Mode Analytics source to Mage data integration pipeline.

Doc: https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/mode/README.md

**OracleDB Destination**

Doc: https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/oracledb/README.md

**MinIO support for S3 in Data integrations pipeline**

Support using S3 source to connect to MinIO by configuring the `aws_endpoint` in the config.

Doc: https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/amazon_s3/README.md

Bug fixes and improvements

- **Snowflake:** Use `TIMESTAMP_TZ` as column type for snowflake datetime column.
- **BigQuery:** Not require key file for [BigQuery](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/bigquery) source and destination. When Mage is deployed on GCP, it can use the service account to authenticate.
- **Google Cloud Storage:** Allow authenticating with [Google Cloud Storage](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/google_cloud_storage/README.md) using service account
- **MySQL**
- Fix inserting DOUBLE columns into MySQL destination
- Fix comparing datetime bookmark column in MySQL source
- Use backticks to wrap column name in MySQL
- **MongoDB source:** Add authSource and authMechanism options for [MongoDB source](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/mongodb/README.md).
- **Salesforce source:** Fix loading sample data for Salesforce source
- Improve visibility into non-functioning "test connection" and "load sample data" features for integration pipelines:
- Show unsupported error is "Test connection" is not implemented for an integration source.
- Update error messaging for "Load sample data" to let user know that it may not be supported for the currently selected integration source.
- Interpolate pipeline name and UUID in data integration pipelines. Doc: https://docs.mage.ai/data-integrations/configuration#variable-names

SQL block

**OracleDB Loader Block**

Added OracleDB Data Loader block


Bug fixes

- **MSSQL:** Fix MSSQL sql block schema. `Schema` was not properly set when checking table existence. Use `dbo` as the default schema if no schema is set.
- **Trino:** Fix inserting datetime column into Trino
- **BigQuery:** Throw exception in BigQuery SQL block
- **ClickHouse:** Support automatic table creation for ClickHouse data exporter

DBT block

DBT ClickHouse

Shout out to [Daesgar](https://github.com/Daesgar) for his contribution of adding support running ClickHouse DBT models in Mage.


**Add DBT generic command block**

Add a DBT block that can run any generic command


Bug fixes and improvements

- Fix bug: Running DBT block preview would sometimes not use sample limit amount.
- Fix bug: Existing upstream block would get overwritten when adding a dbt block with a ref to that existing upstream block.
- Fix bug: Duplicate upstream block added when new block contains upstream block ref and upstream block already exists.
- Use UTF-8 encoding when logging output from DBT blocks.

Notebook improvements

- Turn on output to logs when running a single block in the notebook



- When running a block in the notebook, provide an option to only run the upstream blocks that haven’t been executed successfully.


- Change the color of a custom block from the UI.


- Show what pipelines are using a particular block
- Show block settings in the sidekick when selecting a block
- Show which pipelines a block is used in
- Create a block cache class that stores block to pipeline mapping


- Enhanced pipeline settings page and block settings page
- Edit pipeline and block executor type and interpolate
- Edit pipeline and block retry config from the UI
- Edit block name and color from block settings



- Enhance dependency tree node to show callbacks, conditionals, and extensions


- Save trigger from UI to code



Cloud deployment

- Allow setting service account name for [k8s executor](https://docs.mage.ai/production/configuring-production-settings/compute-resource#kubernetes-executor)
- Example k8s executor config:

yaml
k8s_executor_config:
resource_limits:
cpu: 1000m
memory: 2048Mi
resource_requests:
cpu: 500m
memory: 1024Mi
service_account_name: custom_service_account_name


- Support customizing the timeout seconds in [GCP cloud run config](https://docs.mage.ai/production/configuring-production-settings/compute-resource#gcp-cloud-run-executor).
- Example config

yaml
gcp_cloud_run_config:
path_to_credentials_json_file: "/path/to/credentials_json_file"
project_id: project_id
timeout_seconds: 600


- Check ECS task status after running the task.

S**treaming pipeline**

- Fix copy output in streaming pipeline. Catch deepcopy error (`TypeError: cannot pickle '_thread.lock' object in the deepcopy from the handle_batch_events_recursively`) and fallback to copy method.

Spark pipeline

- Fix an issue with setting custom Spark pipeline config.
- Fix testing Spark DataFrame. Pass the correct Spark DataFrame to the test method.

Other bug fixes & polish

- Add json value macro. Example usage: `"{{ json_value(aws_secret_var('test_secret_key_value'), 'k1') }}"`
- Allow slashes in block_uuid when downloading block output. The regex for the block output download endpoint would not capture block_uuids with slashes in them, so this fixes that.
- Fix renaming block.
- Fix user auth when disable notebook edits is enabled.
- Allow JWT_SECRET to be modified via env var. The `JWT_SECRET` for encoding and decoding access tokens was hardcoded, the fix allows users to update it through an [environment variable](https://docs.mage.ai/development/environment-variables).
- Hide duplicate shortcut items in editor context menu
- Before (after running the block a few times and removing/adding block connections):


- After (after following the same steps and running the block a few times and removing/adding block connections):


- When changing the name of a block or creating a new block, auto-create non-existent folders if the block name is using nested block names.
- Fix trigger count in pipeline dashboard
- Fix copy text for secrets
- Fix git sync `asyncio` issue
- Fix Circular Import when importing `get_secret_value` method
- Shorten branch name in the header. If branch name is greater than 21 characters, show ellipsis.
- Replace hard-to-read dark blue font in code block output with much more legible yellow font.
- Show error popup if error occurs when updating pipeline settings.
- Update tree node when block status changes
- Prevent sending notification multiple times for multiple block failures


Conditional block

Add conditional block to Mage. The conditional block is an "Add-on" block that can be added to an existing block within a pipeline. If the conditional block evaluates as False, the parent block will not be executed.

Doc: https://docs.mage.ai/development/blocks/conditionals/overview




Download block output

For standard pipelines (not currently supported in integration or streaming pipelines), you can save the output of a block that has been run as a CSV file. You can save the block output in Pipeline Editor page or Block Runs page.

Doc: https://docs.mage.ai/orchestration/pipeline-runs/saving-block-output-as-csv



Customize Pipeline level spark config

Mage supports customizing Spark session for a pipeline by specifying the `spark_config` in the pipeline `metadata.yaml` file. The pipeline level `spark_config` will override the project level `spark_config` if specified.

Doc: https://docs.mage.ai/integrations/spark-pyspark#custom-spark-session-at-the-pipeline-level

Data integration pipeline

Oracle DB source

Doc: https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/oracledb/README.md

Download file data in the API source

Doc: https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/api

Personalize notification messages

Users can customize the notification templates of different channels (slack, email, etc.) in project metadata.yaml. Hare are the supported variables that can be interpolated in the message templates: `execution_time` , `pipeline_run_url` , `pipeline_schedule_id`, `pipeline_schedule_name`, `pipeline_uuid`

Example config in project's metadata.yaml

yaml
notification_config:
slack_config:
webhook_url: "{{ env_var('MAGE_SLACK_WEBHOOK_URL') }}"
message_templates:
failure:
details: >
Failure to execute pipeline {pipeline_run_url}.
Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
Test custom message."



Doc: https://docs.mage.ai/production/observability/alerting-slack#customize-message-templates

Support **MSSQL and MySQL as the database engine**

Mage stores orchestration data, user data, and secrets data in a database. In addition to SQLite and Postgres, Mage supports using MSSQL and MySQL as the database engine now.

MSSQL docs:

- https://docs.mage.ai/production/databases/default#mssql
- https://docs.mage.ai/getting-started/setup#using-mssql-as-database

MySQL docs:

- https://docs.mage.ai/production/databases/default#mysql
- https://docs.mage.ai/getting-started/setup#using-mysql-as-database

Add MinIO and ****Wasabi**** support via S3 data loader block

Mage supports connecting to MinIO and Wasabi by specifying the `AWS_ENDPOINT` field in S3 config now.

Doc: https://docs.mage.ai/integrations/databases/S3#minio-support

Use dynamic blocks with replica blocks

To maximize block reuse, you can use dynamic and replica blocks in combination.

- https://docs.mage.ai/design/blocks/dynamic-blocks
- https://docs.mage.ai/design/blocks/replicate-blocks


Other bug fixes & polish

- The command `CREATE SCHEMA IF NOT EXISTS` is not supported by MSSQL. Provided a default command in BaseSQL -> build_create_schema_command, and an overridden implementation in MSSQL -> build_create_schema_command containing compatible syntax. (Kudos to [gjvanvuuren](https://github.com/gjvanvuuren))
- Fix streaming pipeline `kwargs` passing so that RabbitMQ messages can be acknowledged correctly.
- Interpolate variables in streaming configs.
- Git integration: Create known hosts if it doesn't exist.
- Do not create duplicate triggers when DB query fails on checking existing triggers.
- Fix bug: when there are multiple downstream replica blocks, those blocks are not getting queued.
- Fix block uuid formatting for logs.
- Update WidgetPolicy to allow editing and creating widgets without authorization errors.
- Update sensor block to accept positional arguments.
- Fix variables for GCP Cloud Run executor.
- Fix MERGE command for Snowflake destination.
- Fix encoding issue of file upload.
- Always delete the temporary DBT profiles dir to prevent file browser performance degrade.