Mage-ai

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.59 -->

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/50c5084f-86fe-4543-9f40-e5bfeaa55330" width="60%"/>
</div>

What's Changed
🎉 Exciting New Features

**Note:** many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/2af3f074-aeb5-4fda-b3cf-4cead6d945fd" width="60%"/>
</div>

🎮 Multi-project Platform [BETA]

We've reworked our support for mulitple projects with a new multi-project platform! tommydangerous is back at it again with this *huge* feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/80a88de4-615c-431a-9d8f-1d9e61af523a

If you'd like to try out the multi-project platform, you can check out [this repo](https://github.com/mage-ai/platform_template) for a sample structure. Head over to your Mage settings to enable

* Support nested projects and custom code paths by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4161
* Trigger and run pipelines across projects by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4186
* Split pipeline scheduler and schedule models for project platform by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4233
* Configure root project preferences and settings by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4234
* Fix pipeline schedule creation and repo name by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4247
* Don't try to use file source if not exist by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4196
* Fix the way we store pipelines by type in the cache by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4278

🚀 Command Center [BETA]

Another new & exciting feature this week— the Mage **Command Center**. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/a01a7f11-a0c8-43f1-9a44-253202659c75

Enable the Command Center in settings and give it a spin today!

* Command center by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4249
* Add command center models by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4254

🪣 Bitbucket Version Control

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/337f50eb-9728-472c-9781-64e00f5fdce2" width="55%"/>
</div>

Shout out to dy46 for continuing to crush the version control integrations!

You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more [here](https://docs.mage.ai/getting-started/setting-up-git).

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/efae5852-7c77-4d1f-8756-fae19621bca1" width="55%"/>
</div>

*by dy46 in https://github.com/mage-ai/mage-ai/pull/4176*

◔ Qdrant integration

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/3ee8f951-f2c4-44be-b790-9a7c85b06878" width="55%"/>
</div>

Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more [here](https://docs.mage.ai/integrations/databases/Qdrant).

*by matrixstone in https://github.com/mage-ai/mage-ai/pull/4081*

🧱 dbt DX v2 + dbt Upgrade [BETA]

This release contains a _huge_ dbt overhaul 🤯

Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!

Here's a quick demo:

https://github.com/mage-ai/mage-ai/assets/59450879/c6518d08-fd32-4bbb-b271-9aa28409b41d

* Upgrade dbt to 1.7 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4244
* dbt v2 browser UI by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4200
* Add block browser modal to notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4246
* Add dbt cache by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4193
* Add fields to dbt code block 2.0 for manual entry by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4331
* Use absolute paths in dbt block by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4307
* Add custom code block tags by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4250
* Collapse or expand folders by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4203

🐛 Bug Fixes

SQL blocks
* Remove double quotes for postgres by dy46 in https://github.com/mage-ai/mage-ai/pull/4170
* Escape BigQuery project name in SQL block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4294
* Pass in `query_vars` as a dict by dy46 in https://github.com/mage-ai/mage-ai/pull/4280

Data integration
* Fix syncing MySQL `TIME` type by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4275
* Bump Google Ads version from 14 to 15 by Luishfs in https://github.com/mage-ai/mage-ai/pull/4289
* Fix incremental sync bug with missing arg by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4315
* Fix executing data integration block with ecs executor by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4322
* Wrap MSSQL table name with double quotes in data integration pipeline by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4290

Trigger and scheduling
* Use UTC date for trigger start date by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4283
* Display correct default start datetime when editing trigger/backfill by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4292
* Try preventing creating duplicate pipeline runs in scheduler by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4311
* Wrap block run initialization logic with lock by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4296

dbt
* Fix dbt profiles interpolation by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4225
* Lazy import for dbt files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4251
* Re-work dbt project path for `yaml` files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4207
* Error handling for "project not found" in dbt by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4334
* Adjust error behavior when adding dbt files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4191

File browser
* Fix files page not opening files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4222
* File browser bug bash by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4237
* Fix bug when deleting block from file browser by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4243
* Fix `requestIdleCallback` not supported on Safari by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4284
* Fix file browser not refreshing by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4223

Dynamic blocks
* Fix dynamic block + dynamic child blocks spawning other blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4295
* Fix reduce output bug by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4326

Git
* Add actions to `GitBranchPolicy` by dy46 in https://github.com/mage-ai/mage-ai/pull/4213
* Fix Git bugs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4264
* Fix Git submodule sync by dy46 in https://github.com/mage-ai/mage-ai/pull/4316
* Fix missing Git module by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4206

Other
* Fix silent errors from global hooks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4173
* Fix block sorting bug by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4175
* Avoid saving error details in block run DB by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4179
* Reset page after applying pipeline filters by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4183
* Restrict opentelemetry package versions by dy46 in https://github.com/mage-ai/mage-ai/pull/4208
* Fix dataframe validation by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4255
* Fix add new button tooltips by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4263
* Save `statistics.json` in correct execution partition folder by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4271
* Fix dashboard resizing by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4282
* Only set the schema in the DB when the server is started by dy46 in https://github.com/mage-ai/mage-ai/pull/4293
* Fix several bugs on Pipelines dashboard by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4300
* Improve UI in several areas by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4309
* Fix dragging and dropping file in browser by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4312
* Fix interpolating mage secret in project metadata.yaml by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4319
* Remove undefined request by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4324
* Fix triple layout after width by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4325
* Fix cannot unpack non-iterable NoneType object by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4327
* Add small fixes by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4329
* Bug fixes and minor improvements on version control app by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4267
* Fix a ton of bugs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4214
* Fix escaping quotes in R block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4285


💅 Enhancements & Polish

File browser
* Improve folder performance by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4221
* View or hide hidden files in file browser by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4235
* Re-use file browser and editor in notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4231

Raw SQL enhancements
* Disable sample query when using raw SQL by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4252
* Add execute raw SQL without preprocessing to all loaders by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4303

Workspace management
* Add default values for workspace creation by dy46 in https://github.com/mage-ai/mage-ai/pull/4219
* Use current pod image as the default by dy46 in https://github.com/mage-ai/mage-ai/pull/4299
* Update url logic for workspace manager

Other
* Update Git settings when environment variables are set in https://github.com/mage-ai/mage-ai/pull/4154 in https://github.com/mage-ai/mage-ai/pull/4166
* Global hooks for platform by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4236
* Decrease image size by removing Qdrant libraries by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4277
* Update Git to set username and email when using version control by dy46 in https://github.com/mage-ai/mage-ai/pull/4262
* Dynamic dynamic child and replicas by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4308
* Speed up pipeline run count query by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4321
* Rewrite schedule queries by dy46 in https://github.com/mage-ai/mage-ai/pull/4185
* Update custom block template by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4177
* Support serialize Polars dataframe by dy46 in https://github.com/mage-ai/mage-ai/pull/4281
* Add `queue_subscription` support for NATS Jetstream streaming source by atjhoendz in https://github.com/mage-ai/mage-ai/pull/4169
* Added URL protocol option to RabbitMQ streaming by Luishfs in https://github.com/mage-ai/mage-ai/pull/4228
* AWS S3 Destination: Added support for IAM role-based authentication by eofs in https://github.com/mage-ai/mage-ai/pull/4320
* Added `role_arn` as an optional config for AWS DynamoDB source by robreinhold in https://github.com/mage-ai/mage-ai/pull/4256
* SQL block 1.5 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4302
* Add retry info in callbacks and retry function by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4332
* Add custom 404 page by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4149

New Contributors
* aliavni made their first contribution in https://github.com/mage-ai/mage-ai/pull/4181
* atjhoendz made their first contribution in https://github.com/mage-ai/mage-ai/pull/4169
* robreinhold made their first contribution in https://github.com/mage-ai/mage-ai/pull/4256

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.50...0.9.59

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/68865221-095c-4e6a-b5bd-01f680756a46" width="65%"/>
</div>

What's Changed
🎉 Exciting New Features
🌊 [Data Integration] Dremio Source

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/d4c781f0-433c-461f-931f-5ee51b5ecd0e" width="65%"/>
</div>

</br>

🥳 Dremio users, rejoice! Mage now supports Dremio as a [data integration source](https://docs.mage.ai/data-integrations/overview), meaning you can now build data integration pipelines pulling from data lakes and more!

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/4072_

🏃‍♂️ Manually run pipeline once in same trigger

This update, courtesy of our frontend engineer, Johnson, is a big one!

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/78053898/7cd95dee-c0be-4a8e-9c19-4a62fa9bd075" width="65%"/>
</div>

</br>

First, he added a new button to the Trigger Detail page for running pipelines once in the same trigger! 👀

Next, some quality of life improvements:
- For `once` triggers, the trigger does not need to be `active` status before running the pipeline once using the `Runonce` button in the Trigger Detail page. _Many_ users have told us this is confusing... No more!
- The trigger will automatically be updated to `active` status. However, if the trigger is NOT an `once` trigger (e.g. a recurring interval or API trigger), the trigger must be set to `active` status before manually running the pipeline once from the Trigger Detail page.

Finally, Johnson renamed the "start/pause" trigger on Trigger Detail page to "enable/disable" to be a bit more clear and communicative. Again, we've heard this is a bit misleading, so we did something about it! 🗣️

_by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4133_

[Streaming] ActiveMQ Sink

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/bb49f0eb-2a20-4c5d-807d-c31e3fdad961" width="65%"/>
</div>

</br>

Shruti continues her _epic_ tear of contributing _magical_ ✨ pipelines. In this PR, she's added an ActiveMQ streaming sink to Mage. Apache ActiveMQ is an open source message broker written in Java... and now you can write data there via Mage! 💫

_by shrutimantri in https://github.com/mage-ai/mage-ai/pull/4141_

🐛 Bug Fixes
* Fix stale pipeline message not appearing by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4138
* Fixed Salesforce Source not running sync by Luishfs in https://github.com/mage-ai/mage-ai/pull/4048
* Fixed Salesforce Destination Upsert action by Luishfs in https://github.com/mage-ai/mage-ai/pull/4130
* Fix external cloud storage logs for k8s blocks by dy46 in https://github.com/mage-ai/mage-ai/pull/4128
* Update DownloadPolicy to allow downloading pipeline zip files by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4148
* Fix pagination by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4152
* Add tests and improve logs table by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4156
* Add tags to pipeline cache by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4162
* Disable error and UI limiting dynamic blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4164
💅 Enhancements & Polish
* Trigger global hooks on pipeline execution by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4147
* Close modal after saving by dy46 in https://github.com/mage-ai/mage-ai/pull/4131
* Speed up pipelines list API operation by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4132
* Add resource parent in the input data for hooks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4144
* Optimize pipeline schedules LIST api by dy46 in https://github.com/mage-ai/mage-ai/pull/4058
* Fix block runs page when there are a large number of block runs by dy46 in https://github.com/mage-ai/mage-ai/pull/4109
* Dynamic blocks 2.0 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4157
* Add more tests for pipeline execute global hook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4159
* Update git settings when environment variables are set by dy46 in https://github.com/mage-ai/mage-ai/pull/4154
* Update url logic for workspace manager by dy46 in https://github.com/mage-ai/mage-ai/pull/4166

New Contributors
* Abiodun made their first contribution in https://github.com/mage-ai/mage-ai/pull/4125

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.48...0.9.50

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/f932803b-c696-4231-84c3-d963d1b487fc">
</div>

What's Changed
🎉 Exciting New Features

Incremental data integration in batch pipelines

🥳 Data integrations in batch pipelines now support incremental replication! You can read more [here](https://docs.mage.ai/data-integrations/batch-pipelines) to get started!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4068_

[Streaming] RabbitMQ Destination

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/ffe32fe8-bf74-4e4f-8da3-7c9189e2613a" width="40%">
</div>
</br>

_Another_ community PR from shrutimantri adds support for RabbitMQ as a streaming data sink. 🔥

Check it out today with your favorite streaming sources! You can find the configuration reference [here](https://docs.mage.ai/streaming/destinations/rabbitmq).

_by shrutimantri in https://github.com/mage-ai/mage-ai/pull/4041_

Chroma integration

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/75615fb2-e569-4eb9-9541-019d23467f26" width="40%">
</div>
</br>

Mage now has a ChromaDB IO Class, meaning you can use data loaders and exporters in your batch pipelines to read/write from Chroma sources. You can read more about configuration [here](https://docs.mage.ai/integrations/databases/Chroma) or visit [Chroma's site](https://www.trychroma.com/) to learn more about their vector database.

_by matrixstone in https://github.com/mage-ai/mage-ai/pull/4017_

Bookmark overrides

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/7585e2ac-650d-4881-8b3c-2d599474a987" width="70%">
</div>
</br>

🎊 If you're creating a trigger on a data integration, you can now override bookmarks with your own custom values!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4073_

SQL Block environment variable interpolation

For our fans of SQL blocks, you can now interpolate environment variables directly in your queries!

sql
SELECT
'{{ env_var("ENV") }}' AS test
, '{{ variables("test") }}' AS test2
, {{ test }} AS test3


This should allow for _much_ greater flexibility in pipelines with SQL!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4076_

Additional upstream dependencies for dynamic children

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/ddb80a64-a094-4da4-a087-a34fa8d96a4e" width="50%">
</div>
</br>

Love dynamic blocks? 🤔 They dynamic children can now have additional upstream dependencies!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4104_

Support caching block output in memory

Previously, pipelines with large Spark DataFrames faced out of heap space errors when persisting block outputs to disk. This PR allows the user to disable persisting output. The feature is only supported in standard batch pipeline (without dynamic blocks) for now.

yaml
cache_block_output_in_memory: true
run_pipeline_in_one_process: true


_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4127_

🐛 Bug Fixes
* Backend API for getting information about bookmarks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4070
* Support different operators when comparing bookmark properties by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4075
* Update backfill statuses by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3994
* Reduce block at any level UI by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4067
* Catch execption of empty integration streams in pipeline scheduler by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4083
* Backfill's date-picker date value mismatch by edmondwinston in https://github.com/mage-ai/mage-ai/pull/3972
* Gracefully access dictionaries in the Oauth Policy by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4086
* Pass tolerations to job pod by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4089
* Fix load sample data for integration pipelines by dy46 in https://github.com/mage-ai/mage-ai/pull/4034
* Default to using environment variables for git and workspace settings by dy46 in https://github.com/mage-ai/mage-ai/pull/4088
* Fixed Google Ads Source by Luishfs in https://github.com/mage-ai/mage-ai/pull/4099
* Fix chromadb dependency by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4107
* Fix chromadb in all package by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4108
* Update local timezone project setting from header by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4111
* Fix runtime variables not showing when creating new trigger by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4116
* Fix executing conditional blocks with pipeline executor by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4120
* Remove itertools groupby by dy46 in https://github.com/mage-ai/mage-ai/pull/4103
* Updates/nats add stream fixes by mfreeman451 in https://github.com/mage-ai/mage-ai/pull/4113
* Update `opentelemetry-exporter-prometheus` package version by dy46 in https://github.com/mage-ai/mage-ai/pull/4101
* Fix postgres streaming sink when there are no messages by shrutimantri in https://github.com/mage-ai/mage-ai/pull/4074

💅 Enhancements & Polish
* Add any runtime variables by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4071
* Include `message_events_json` in Postmark messages_outbound stream by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4085
* Hide "Unique" and "Key" columns for certain data integration destination blocks by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4096
* Add top padding to file code editor by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4098
* Display error in UI when variables directories configured incorrectly by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4091
* Improve the interface for Chroma class by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4110
* Add `column_header_format` option by dy46 in https://github.com/mage-ai/mage-ai/pull/4118
* Allow configuring Amplitude host by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4060

New Contributors
* andrewgetzdata made their first contribution in https://github.com/mage-ai/mage-ai/pull/4078
* suvhotta made their first contribution in https://github.com/mage-ai/mage-ai/pull/4097

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.46...0.9.48

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/b21c4aaa-bf39-4d31-8193-5e5ce31075b4"/>
</div>

What's Changed

🎉 Exciting New Features

⚡️ Spark UI/UX 2.0 for AWS EMR

It's finally here! Mage now comes with a completely revamped custom Spark UI/UX for our AWS EMR users! This is a _huge_ update that comes with a complete overhaul of every element possible for managing your Spark cluster!

Check out the PR for more screenshots and get started today (docs coming soon)!

<div align="center">
<img src="https://user-images.githubusercontent.com/1066980/283299347-f0777e74-1500-4e37-9ef7-b40329a58baa.png" width="70%"/>
<img src="https://user-images.githubusercontent.com/1066980/283295208-34924d4d-94a4-4295-853c-01ce3e4fb14f.png" width="70%"/>
</div>

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3997_

🌊 Streaming: ActiveMQ Source

🔥 shrutimantri is on fire! In _another_ community PR, she adds streaming support for ActiveMQ as a source! If you're an ActiveMQ user, give it a shot today!

<div align="center">
<img src="https://user-images.githubusercontent.com/6594483/282857969-42da0e5c-f51a-494c-9bff-1a23762bc811.png" width="70%"/>
</div>

_by shrutimantri in https://github.com/mage-ai/mage-ai/pull/3978_

📁 Download Files & Pipelines via the UI

_Another_ community PR, this one from PopaRares, allows you to download files and pipelines via the right-click menu in the Mage UI.

This will be a game changer for collabrative projects and importing/exporting data from Mage!

<div align="center">
<img src="https://user-images.githubusercontent.com/33767023/276976985-98a256f1-1694-43e2-bf48-617a0ab9b43e.gif" width="70%"/>
</div>

_by PopaRares in https://github.com/mage-ai/mage-ai/pull/3813_

✨ Streaming: NATS JetStream Source

A big shoutout to community member mfreeman451 for adding the NATS JetStream message broker as a Streaming Source in Mage!

<div align="center">
<img src="https://user-images.githubusercontent.com/1821930/283172350-1e8852cc-1c1c-4ecf-b2be-6fcc6363733b.png" width="70%"/>
</div>

_by mfreeman451 in https://github.com/mage-ai/mage-ai/pull/3985_

🎏 Data integration: Kafka destination

Luishfs is back at it with another destination— this one for data integration. You can now write DI outputs to a Kafka topic. We can't wait to see what y'all cook up with this one!

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/05204fab-7fef-4468-b65b-91be7ce06745" width="70%"/>
</div>

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/3976_

🛟 Auto-save triggers in code

Last, but certainly not least, Mage is now able to auto-save triggers as code. That means (when enabled) you can update triggers and have them auto-save to your Mage project. WThis should help you keep track of your trigger changes across projects.

<div align="center">
<img src="https://user-images.githubusercontent.com/1066980/283650404-fcdc2f17-d7e3-4993-89b6-ec418b132eef.png" width="70%"/>
</div>

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4009_

🐛 Bug Fixes

* Add `pyarrow-hotfix` to `requirements.txt` by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3993
* Fix workspace user management fetching by dy46 in https://github.com/mage-ai/mage-ai/pull/3992
* Handle hidden block positions when split view by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4005
* Don’t show SSH tunnel option unless kernel is PySpark by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4006
* Fix Monoco Editor for base path by dy46 in https://github.com/mage-ai/mage-ai/pull/4012
* Update OAuth sign on and fix OAuth sign on with `REQUIRE_USER_PERMISSIONS` by dy46 in https://github.com/mage-ai/mage-ai/pull/4007
* Clarification when applying bookmark to all streams by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4036 and https://github.com/mage-ai/mage-ai/pull/4049
* Allow using block configuration when `run_pipeline_in_one_process` is true by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4046
* Only show Backfills in vertical nav for standard (python) pipelines by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4050
* Fix files not being selected in notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4051

💅 Enhancements & Polish

* Added application/gzip support to API source by Luishfs in https://github.com/mage-ai/mage-ai/pull/3990
* Re-enqueue the job if queue is empty by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3996
* Add better error message for API source by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4016
* Pass envFrom to job pod by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4031

New Contributors

* mfreeman451 made their first contribution in https://github.com/mage-ai/mage-ai/pull/3985

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/a20e6f84-9cae-46d1-b849-52bbdc845935" width="65%">
</div>

What's Changed

🎉 Exciting New Features

🔐 New SSO/OAuth providers

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/9e8f95c6-5d5b-4869-b5f3-8b2e30d8d645" width="30%"/>
</div>
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/46488e77-2e61-4375-befd-7a5bd74d270c" width="30%"/>
</div>
<br/>

With our latest release, Mage now supports SSO/OAuth from not one, but _two_ providers— Okta & Google. Our engineers also thought ahead, laying the groundwork for supporting more providers in the future, too! Check out the docs— [Google](https://docs.mage.ai/production/authentication/google) & [Okta](https://docs.mage.ai/production/authentication/okta).

_by dy46 in https://github.com/mage-ai/mage-ai/pull/3921_

🔥 Compute management for Apache Spark blocks

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/e77d1df7-8ec8-4a08-803a-aa806e10fb81" width="65%">
</div>

Tommy is back at it with another _massive_ PR, this one adding full support for compute management in Apache Spark blocks. For those of you who leverage Spark, this PR will allow you fine-grained control over your compute. Keep an eye out for completely revamped EMR functionality in the near future!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3883_

🤗 HuggingFace AI Client

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/c4d00071-d453-4e64-a50c-9824f1e2ff97" width="75%">
</div>

Mage now supports using different AI models for interfaces within the applicaiton, not just OpenAI! The first we've added is a HuggingFace client... You can now use HuggingFace with Mage's AI functionality! Read more about getting started [here](https://docs.mage.ai/guides/ai/ai-client).

_by matrixstone in https://github.com/mage-ai/mage-ai/pull/3850 and https://github.com/mage-ai/mage-ai/pull/3919_

🧱 Azure Databricks Delta Lake Destination

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/3452a517-7806-47d4-a506-da7ffe9bf245" width="75%">
</div>

🎉 Mage now supports Azure Databricks Delta Lake as a destination for data integration pipelines! That means you can write all of your favorite sources to the open, parquet-based storage system on Microsofts cloud infrastructure!

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/3932_

📊 Prometheus Metrics

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/4ee96ea1-b3d0-4b5a-8c19-4f2ecf425a96" width="75%">
</div>

Prometheus style metrics are a vendor neutral standard based on a pull model. Prometheus-enabled servers output time-series data on a route (usually /metrics), which can be scraped. Being an open standard, most monitoring tools know how to interface with Prometheus metrics (Open Telemetry support them too).

You can learn more about Prometheus here:

- https://prometheus.io/docs/concepts/data_model/
- https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md

This PR enables the basic built-in metrics, which cover the Tornado server (http metrics) and the Python runtime. More metrics can be added in the future! Check out our docs [here](https://docs.mage.ai/integrations/observability/prometheus#prometheus-metrics).

_by adelcast in https://github.com/mage-ai/mage-ai/pull/3910_

🐛 Bug Fixes

* Inject global and pipeline variables in the keyword arguments for extension blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3917
* Fixed Stripe `INCREMENTAL` run and updated tap by Luishfs in https://github.com/mage-ai/mage-ai/pull/3835
* Break `while` loop when not raising error on failure for pipeline triggered by code by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3933
* Make 2nd argument of lambda function optional by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3941
* Update git settings permissions by dy46 in https://github.com/mage-ai/mage-ai/pull/3935
* Fix saving tokens when creating triggers with code by dy46 in https://github.com/mage-ai/mage-ai/pull/3938
* Misc workspace changes by dy46 in https://github.com/mage-ai/mage-ai/pull/3931
* Fix MongoDB destination and add unit test by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3944
* Fix SQL destination reserved words by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3951
* Fix notebook block ordering for upstream blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3955
* Fix Dockerfile and API when using EMR by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3960
* Fix unit tests warnings and errors by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3961
* Clean column name when using batch load in Snowflake destination by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3968
* Serialize Snowflake dataframe `dict` column to `json` if column type is `string` by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3969
* Move dbt seed logic to downstream block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3953
* Fix project dashboard overview count formatting by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3980
* Fix pipeline scheduler for integration pipelines by dy46 in https://github.com/mage-ai/mage-ai/pull/3981
* Fix roles getting overwritten when updating profile by dy46 in https://github.com/mage-ai/mage-ai/pull/3982
* Fix tree for data integration pipeline by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3986
* Minor bug fix in Pinot config on `io_config.yaml` by shrutimantri in https://github.com/mage-ai/mage-ai/pull/3970

💅 Enhancements & Polish

* Support pipeline level EMR config by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3922
* Add keyboard shortcuts for inserting new scratchpad cell by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3926
* Add colour code for pipeline backfills by edmondwinston in https://github.com/mage-ai/mage-ai/pull/3904
* Show dependency graph zoom options by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3899
* Update how global data products are run by dy46 in https://github.com/mage-ai/mage-ai/pull/3872
* Consistent run status colors across tables by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3940
* Enable users to cancel in progress runs when disabling a pipeline trigger by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3905
* Allow getting instance type from environment variable by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3949
* Include Monaco Editor in build to avoid fetching from CDN by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3916
* Interpolate variables and upstream block output in dbt commands by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3945
* Update `duckdb` version by dy46 in https://github.com/mage-ai/mage-ai/pull/3959
* Update backfill variables by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3963
* Always show "Overwrite global variables" setting when editing a trigger by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3973
* Add pipeline run limit for a pipeline by dy46 in https://github.com/mage-ai/mage-ai/pull/3868
* Use personal access token if available by dy46 in https://github.com/mage-ai/mage-ai/pull/3974
* Add exception failure message in callbacks by dy46 in https://github.com/mage-ai/mage-ai/pull/3952
* Support override `assignPublicIp` and `enableExecuteCommand` in EcsConfig by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3966
* Add kafka `api_version` to the data loader and data exporter templates by shrutimantri in https://github.com/mage-ai/mage-ai/pull/3967

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.43...0.9.45

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/9191e2ba-fc9e-45e6-b10b-c1a15533051c" width="60%">
</div>

What's Changed
🎉 Exciting New Features