Mage-ai

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/b21c4aaa-bf39-4d31-8193-5e5ce31075b4"/>
</div>

What's Changed

🎉 Exciting New Features

⚡️ Spark UI/UX 2.0 for AWS EMR

It's finally here! Mage now comes with a completely revamped custom Spark UI/UX for our AWS EMR users! This is a _huge_ update that comes with a complete overhaul of every element possible for managing your Spark cluster!

Check out the PR for more screenshots and get started today (docs coming soon)!

<div align="center">
<img src="https://user-images.githubusercontent.com/1066980/283299347-f0777e74-1500-4e37-9ef7-b40329a58baa.png" width="70%"/>
<img src="https://user-images.githubusercontent.com/1066980/283295208-34924d4d-94a4-4295-853c-01ce3e4fb14f.png" width="70%"/>
</div>

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3997_

🌊 Streaming: ActiveMQ Source

🔥 shrutimantri is on fire! In _another_ community PR, she adds streaming support for ActiveMQ as a source! If you're an ActiveMQ user, give it a shot today!

<div align="center">
<img src="https://user-images.githubusercontent.com/6594483/282857969-42da0e5c-f51a-494c-9bff-1a23762bc811.png" width="70%"/>
</div>

_by shrutimantri in https://github.com/mage-ai/mage-ai/pull/3978_

📁 Download Files & Pipelines via the UI

_Another_ community PR, this one from PopaRares, allows you to download files and pipelines via the right-click menu in the Mage UI.

This will be a game changer for collabrative projects and importing/exporting data from Mage!

<div align="center">
<img src="https://user-images.githubusercontent.com/33767023/276976985-98a256f1-1694-43e2-bf48-617a0ab9b43e.gif" width="70%"/>
</div>

_by PopaRares in https://github.com/mage-ai/mage-ai/pull/3813_

✨ Streaming: NATS JetStream Source

A big shoutout to community member mfreeman451 for adding the NATS JetStream message broker as a Streaming Source in Mage!

<div align="center">
<img src="https://user-images.githubusercontent.com/1821930/283172350-1e8852cc-1c1c-4ecf-b2be-6fcc6363733b.png" width="70%"/>
</div>

_by mfreeman451 in https://github.com/mage-ai/mage-ai/pull/3985_

🎏 Data integration: Kafka destination

Luishfs is back at it with another destination— this one for data integration. You can now write DI outputs to a Kafka topic. We can't wait to see what y'all cook up with this one!

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/05204fab-7fef-4468-b65b-91be7ce06745" width="70%"/>
</div>

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/3976_

🛟 Auto-save triggers in code

Last, but certainly not least, Mage is now able to auto-save triggers as code. That means (when enabled) you can update triggers and have them auto-save to your Mage project. WThis should help you keep track of your trigger changes across projects.

<div align="center">
<img src="https://user-images.githubusercontent.com/1066980/283650404-fcdc2f17-d7e3-4993-89b6-ec418b132eef.png" width="70%"/>
</div>

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4009_

🐛 Bug Fixes

* Add `pyarrow-hotfix` to `requirements.txt` by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3993
* Fix workspace user management fetching by dy46 in https://github.com/mage-ai/mage-ai/pull/3992
* Handle hidden block positions when split view by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4005
* Don’t show SSH tunnel option unless kernel is PySpark by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4006
* Fix Monoco Editor for base path by dy46 in https://github.com/mage-ai/mage-ai/pull/4012
* Update OAuth sign on and fix OAuth sign on with `REQUIRE_USER_PERMISSIONS` by dy46 in https://github.com/mage-ai/mage-ai/pull/4007
* Clarification when applying bookmark to all streams by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4036 and https://github.com/mage-ai/mage-ai/pull/4049
* Allow using block configuration when `run_pipeline_in_one_process` is true by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4046
* Only show Backfills in vertical nav for standard (python) pipelines by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4050
* Fix files not being selected in notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4051

💅 Enhancements & Polish

* Added application/gzip support to API source by Luishfs in https://github.com/mage-ai/mage-ai/pull/3990
* Re-enqueue the job if queue is empty by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3996
* Add better error message for API source by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4016
* Pass envFrom to job pod by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4031

New Contributors

* mfreeman451 made their first contribution in https://github.com/mage-ai/mage-ai/pull/3985

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/a20e6f84-9cae-46d1-b849-52bbdc845935" width="65%">
</div>

What's Changed

🎉 Exciting New Features

🔐 New SSO/OAuth providers

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/9e8f95c6-5d5b-4869-b5f3-8b2e30d8d645" width="30%"/>
</div>
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/46488e77-2e61-4375-befd-7a5bd74d270c" width="30%"/>
</div>
<br/>

With our latest release, Mage now supports SSO/OAuth from not one, but _two_ providers— Okta & Google. Our engineers also thought ahead, laying the groundwork for supporting more providers in the future, too! Check out the docs— [Google](https://docs.mage.ai/production/authentication/google) & [Okta](https://docs.mage.ai/production/authentication/okta).

_by dy46 in https://github.com/mage-ai/mage-ai/pull/3921_

🔥 Compute management for Apache Spark blocks

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/e77d1df7-8ec8-4a08-803a-aa806e10fb81" width="65%">
</div>

Tommy is back at it with another _massive_ PR, this one adding full support for compute management in Apache Spark blocks. For those of you who leverage Spark, this PR will allow you fine-grained control over your compute. Keep an eye out for completely revamped EMR functionality in the near future!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3883_

🤗 HuggingFace AI Client

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/c4d00071-d453-4e64-a50c-9824f1e2ff97" width="75%">
</div>

Mage now supports using different AI models for interfaces within the applicaiton, not just OpenAI! The first we've added is a HuggingFace client... You can now use HuggingFace with Mage's AI functionality! Read more about getting started [here](https://docs.mage.ai/guides/ai/ai-client).

_by matrixstone in https://github.com/mage-ai/mage-ai/pull/3850 and https://github.com/mage-ai/mage-ai/pull/3919_

🧱 Azure Databricks Delta Lake Destination

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/3452a517-7806-47d4-a506-da7ffe9bf245" width="75%">
</div>

🎉 Mage now supports Azure Databricks Delta Lake as a destination for data integration pipelines! That means you can write all of your favorite sources to the open, parquet-based storage system on Microsofts cloud infrastructure!

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/3932_

📊 Prometheus Metrics

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/4ee96ea1-b3d0-4b5a-8c19-4f2ecf425a96" width="75%">
</div>

Prometheus style metrics are a vendor neutral standard based on a pull model. Prometheus-enabled servers output time-series data on a route (usually /metrics), which can be scraped. Being an open standard, most monitoring tools know how to interface with Prometheus metrics (Open Telemetry support them too).

You can learn more about Prometheus here:

- https://prometheus.io/docs/concepts/data_model/
- https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md

This PR enables the basic built-in metrics, which cover the Tornado server (http metrics) and the Python runtime. More metrics can be added in the future! Check out our docs [here](https://docs.mage.ai/integrations/observability/prometheus#prometheus-metrics).

_by adelcast in https://github.com/mage-ai/mage-ai/pull/3910_

🐛 Bug Fixes

* Inject global and pipeline variables in the keyword arguments for extension blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3917
* Fixed Stripe `INCREMENTAL` run and updated tap by Luishfs in https://github.com/mage-ai/mage-ai/pull/3835
* Break `while` loop when not raising error on failure for pipeline triggered by code by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3933
* Make 2nd argument of lambda function optional by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3941
* Update git settings permissions by dy46 in https://github.com/mage-ai/mage-ai/pull/3935
* Fix saving tokens when creating triggers with code by dy46 in https://github.com/mage-ai/mage-ai/pull/3938
* Misc workspace changes by dy46 in https://github.com/mage-ai/mage-ai/pull/3931
* Fix MongoDB destination and add unit test by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3944
* Fix SQL destination reserved words by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3951
* Fix notebook block ordering for upstream blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3955
* Fix Dockerfile and API when using EMR by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3960
* Fix unit tests warnings and errors by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3961
* Clean column name when using batch load in Snowflake destination by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3968
* Serialize Snowflake dataframe `dict` column to `json` if column type is `string` by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3969
* Move dbt seed logic to downstream block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3953
* Fix project dashboard overview count formatting by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3980
* Fix pipeline scheduler for integration pipelines by dy46 in https://github.com/mage-ai/mage-ai/pull/3981
* Fix roles getting overwritten when updating profile by dy46 in https://github.com/mage-ai/mage-ai/pull/3982
* Fix tree for data integration pipeline by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3986
* Minor bug fix in Pinot config on `io_config.yaml` by shrutimantri in https://github.com/mage-ai/mage-ai/pull/3970

💅 Enhancements & Polish

* Support pipeline level EMR config by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3922
* Add keyboard shortcuts for inserting new scratchpad cell by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3926
* Add colour code for pipeline backfills by edmondwinston in https://github.com/mage-ai/mage-ai/pull/3904
* Show dependency graph zoom options by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3899
* Update how global data products are run by dy46 in https://github.com/mage-ai/mage-ai/pull/3872
* Consistent run status colors across tables by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3940
* Enable users to cancel in progress runs when disabling a pipeline trigger by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3905
* Allow getting instance type from environment variable by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3949
* Include Monaco Editor in build to avoid fetching from CDN by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3916
* Interpolate variables and upstream block output in dbt commands by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3945
* Update `duckdb` version by dy46 in https://github.com/mage-ai/mage-ai/pull/3959
* Update backfill variables by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3963
* Always show "Overwrite global variables" setting when editing a trigger by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3973
* Add pipeline run limit for a pipeline by dy46 in https://github.com/mage-ai/mage-ai/pull/3868
* Use personal access token if available by dy46 in https://github.com/mage-ai/mage-ai/pull/3974
* Add exception failure message in callbacks by dy46 in https://github.com/mage-ai/mage-ai/pull/3952
* Support override `assignPublicIp` and `enableExecuteCommand` in EcsConfig by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3966
* Add kafka `api_version` to the data loader and data exporter templates by shrutimantri in https://github.com/mage-ai/mage-ai/pull/3967

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.43...0.9.45

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/9191e2ba-fc9e-45e6-b10b-c1a15533051c" width="60%">
</div>

What's Changed
🎉 Exciting New Features

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/404e4e16-d656-4b9a-894b-f0f418f6347a" width="45%"/>
</div>

What's Changed

🎉 Exciting New Features

Workspace Lifecycle Management

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/6e0552c5-8fcc-4256-bd56-1f1464fabeec" width="65%"/>
</div>

🎉 Mage now provides support for managing the workspace lifecycles in Kubernetes! That means you can control _how_ Mage is deployed, start-to-finish, with the following options:

1. Auto-termination
2. Pre-start scripts
3. Post-start scripts

Read more about lifecycle management [here](https://docs.mage.ai/production/developing-in-the-cloud/workspaces/kubernetes#lifecycle-management) and give it a shot today!

_by dy46 in https://github.com/mage-ai/mage-ai/pull/3836_

Elasticsearch Data Integration Destination

Mage now supports writing data to Elasticsearch for all of your search & LLM needs! 🧙🏻‍♂️

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/ac7cc35d-5a59-4b1b-957a-281a2ad99a5c" width="65%"/>
</div>

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/3756_

Block Detach for Shared Pipelines

Ok, this one is huge— say you have a block in multiple pipelines, but you need to change the logic in a single block instance... That sounds tricky, right? 🤔

Now you can with _Block Detach_! Simply click _ Pipelines_ on the block, then _Detatch_ to create a clone of the block in your current pipeline!

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/02c3f0f7-6c6d-4996-9bb6-2f12b963dd4c" width="75%"/>
</div>

_by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3816_

🐛 Bug Fixes

* Fix disabled keyboard shortcuts due to Pipeline Runs table keyboard nav by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3833
* Allow admin users to read attributes on users list by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3837
* Fix `libodbc` conflicts in Dockerfile by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3840 and https://github.com/mage-ai/mage-ai/pull/3845
* Fix save block functionality after pipeline execution by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3839
* Fix GHE by dy46 in https://github.com/mage-ai/mage-ai/pull/3841
* Fix dynamic children not running and its downstream by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3847
* Only fetch spark jobs if compute is enabled by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3851
* Remove `pymssql` dependency by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3859
* Reposition file browser context menu by edmondwinston in https://github.com/mage-ai/mage-ai/pull/3819
* Send notification on block run intialization failure by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3861
* Update `authorize_query` check by dy46 in https://github.com/mage-ai/mage-ai/pull/3846

💅 Enhancements & Polish

* Make current time button transparent by default by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3829
* Expanded our vocabulary by MageKai in https://github.com/mage-ai/mage-ai/pull/3856
* Added magical nouns by MageKai in https://github.com/mage-ai/mage-ai/pull/3857
* Improve app header styling by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3849
* Add `ctrl/cmd + click` keyboard shortcut for selecting pipeline run rows by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3843
* Use read_namespaced_job instead of read_namespaced_job_status by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3863
* Support `text/csv` response type in API source by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3864
* Automatically clean up cached data integration files by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3869
* Bump snowflake-connector-python version by dy46 in https://github.com/mage-ai/mage-ai/pull/3871 and https://github.com/mage-ai/mage-ai/pull/3873
* Improve base DI destination and add unit tests by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3875
* Add `aws_session_token` support to `get_aws_boto3_client` by nyc-de in https://github.com/mage-ai/mage-ai/pull/3877
* Add clone action to version control by dy46 in https://github.com/mage-ai/mage-ai/pull/3878
* Added test connection and new index naming by Luishfs in https://github.com/mage-ai/mage-ai/pull/3848
* Make sure file browser context menu is always fully visible in the viewport when open by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3855

New Contributors

* nyc-de made their first contribution in https://github.com/mage-ai/mage-ai/pull/3877

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.38...0.9.41

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/87627328-da4d-40e9-9367-a6cf33fc898b" width="65%"/>
</div>

What's Changed

🎉 Exciting New Features

🧑‍💻 Side-by-side block view

Your Mage development workflow just got a _whole_ lot more efficient. Starting today, you can view blocks side-by-side for twice the editing power! Check this one out to improve your DevEx and make more data magic! Simply click the "side-by-side" icon in the center of the editor to get started!
</br>

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/c01dcd4c-9135-430d-8cee-acbbc1101241" width="70%"/>
</div>

</br>

_by tommydangerous in [https://github.com/mage-ai/mage-ai/pull/3804](https://github.com/mage-ai/mage-ai/pull/3804)_

🧱 Support for `dbt-dremio`

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/d4c781f0-433c-461f-931f-5ee51b5ecd0e" width="65%"/>
</div>

</br>

Dremio users, rejoice! You can now execute dbt models in you lakehouse thanks to support for the `dbt-dremio` package.

_by dy46 in [https://github.com/mage-ai/mage-ai/pull/3760](https://github.com/mage-ai/mage-ai/pull/3760)_

👨🏻‍💼 Support Github Enterprise authentication

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/d04a5fd0-e743-4135-90fb-cd2445807fdf" width="65%"/>
</div>

</br>

An often-requested feature, Mage now supports Github Enterprise authentication! 💥

_by dy46 in [https://github.com/mage-ai/mage-ai/pull/3817](https://github.com/mage-ai/mage-ai/pull/3817)_

🤖 Support auto termination in workspace

Mage now supports _auto-termination_ checks, which will run once every sixty seconds. This can be used to auto start/stop k8s workspaces. Configure it when creating your workspace to get started!

<img width="748" alt="Screenshot 2023-10-24 at 7 22 26 PM" src="https://github.com/mage-ai/mage-ai/assets/14357209/1f147dbf-eb42-48a5-a5bb-01abc0cb84e8">


_by dy46 in [https://github.com/mage-ai/mage-ai/pull/3721](https://github.com/mage-ai/mage-ai/pull/3721) and [https://github.com/mage-ai/mage-ai/pull/3751](https://github.com/mage-ai/mage-ai/pull/3751)_

⏰ Show server time in app

This PR adds a new current time display in the top right of the app header. By default, the time display shows in the UTC format, but if you click on the time display, a dropdown menu shows up:

</br>

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/a11ae6b2-80ab-4696-a237-a50fd9b9c22e" width="50%"/>
</div>

</br>

Nice! Frontend UX improvements coming in clutch!

_by anniexcheng in [https://github.com/mage-ai/mage-ai/pull/3785](https://github.com/mage-ai/mage-ai/pull/3785)_


🐛 Bug Fixes

* Fix aws secrets circular dependency by dy46 in [https://github.com/mage-ai/mage-ai/pull/3783](https://github.com/mage-ai/mage-ai/pull/3783)
* Fix adding dbt block with mage_secret_var by wangxiaoyou1993 in [https://github.com/mage-ai/mage-ai/pull/3786](https://github.com/mage-ai/mage-ai/pull/3786)
* Fix botocore library pickle error in pipeline scheduler by wangxiaoyou1993 in [https://github.com/mage-ai/mage-ai/pull/3788](https://github.com/mage-ai/mage-ai/pull/3788)
* Incorrect method call on PostgreSQL close connection by tommydangerous in [https://github.com/mage-ai/mage-ai/pull/3791](https://github.com/mage-ai/mage-ai/pull/3791)
* Update block policy by johnson-mage in [https://github.com/mage-ai/mage-ai/pull/3797](https://github.com/mage-ai/mage-ai/pull/3797)
* Fix `DISABLE_NOTEBOOK_EDIT_ACCESS` variable by dy46 in [https://github.com/mage-ai/mage-ai/pull/3796](https://github.com/mage-ai/mage-ai/pull/3796)
* Fix GCS `listdir` recursive bug by luizarvo in [https://github.com/mage-ai/mage-ai/pull/3807](https://github.com/mage-ai/mage-ai/pull/3807)
* Fix `IndexError` in upstream dbt block by wangxiaoyou1993 in [https://github.com/mage-ai/mage-ai/pull/3808](https://github.com/mage-ai/mage-ai/pull/3808)
* Redshift: Use `TIMESTAMPTZ` column type for datetimes by eofs in [https://github.com/mage-ai/mage-ai/pull/3814](https://github.com/mage-ai/mage-ai/pull/3814)
* Serialize `np.ndarray` before exporting to postgres by wangxiaoyou1993 in [https://github.com/mage-ai/mage-ai/pull/3821](https://github.com/mage-ai/mage-ai/pull/3821)
* Refactor workspace by dy46 in [https://github.com/mage-ai/mage-ai/pull/3811](https://github.com/mage-ai/mage-ai/pull/3811)
* Initialize `db_connection` session before interpolating repo config by wangxiaoyou1993 in [https://github.com/mage-ai/mage-ai/pull/3824](https://github.com/mage-ai/mage-ai/pull/3824)
* Fix code block header overflow menu by tommydangerous in [https://github.com/mage-ai/mage-ai/pull/3832](https://github.com/mage-ai/mage-ai/pull/3832)
* Fix `WorkspaceConfig` default by dy46 in [https://github.com/mage-ai/mage-ai/pull/3784](https://github.com/mage-ai/mage-ai/pull/3784)

💅 Enhancements & Polish

* Allow specifying credentials info in BigQuery source and dest by wangxiaoyou1993 in [https://github.com/mage-ai/mage-ai/pull/3790](https://github.com/mage-ai/mage-ai/pull/3790)
* Support Redshift Serverless by wangxiaoyou1993 in [https://github.com/mage-ai/mage-ai/pull/3823](https://github.com/mage-ai/mage-ai/pull/3823)
* Hide `mage-repo` remote by dy46 in [https://github.com/mage-ai/mage-ai/pull/3806](https://github.com/mage-ai/mage-ai/pull/3806)
* Added Custom Spark Cluster by KamarulAdha in [https://github.com/mage-ai/mage-ai/pull/3812](https://github.com/mage-ai/mage-ai/pull/3812)
* Improved Slack notification design by hanslemm in [https://github.com/mage-ai/mage-ai/pull/3778](https://github.com/mage-ai/mage-ai/pull/3778)
* Add total execution time to Pipeline Runs table by anniexcheng in [https://github.com/mage-ai/mage-ai/pull/3801](https://github.com/mage-ai/mage-ai/pull/3801)
* Case sensitive column names in DI SQL destinations by Luishfs in [https://github.com/mage-ai/mage-ai/pull/3685](https://github.com/mage-ai/mage-ai/pull/3685)

😎 New Contributors

* KamarulAdha made their first contribution in [https://github.com/mage-ai/mage-ai/pull/3812](https://github.com/mage-ai/mage-ai/pull/3812)
* eofs made their first contribution in [https://github.com/mage-ai/mage-ai/pull/3814](https://github.com/mage-ai/mage-ai/pull/3814)
* hanslemm made their first contribution in [https://github.com/mage-ai/mage-ai/pull/3778](https://github.com/mage-ai/mage-ai/pull/3778)
* edmondwinston made their first contribution in [https://github.com/mage-ai/mage-ai/pull/3820](https://github.com/mage-ai/mage-ai/pull/3820)

**Full Changelog**: [https://github.com/mage-ai/mage-ai/compare/0.9.35...0.9.38](https://github.com/mage-ai/mage-ai/compare/0.9.35...0.9.38)

<center>
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/a9bd848f-d0f7-4de3-859c-71c552a110f5" width="65%"/>
</div>
</center>

What's Changed

🎉 Exciting New Features

🙇‍♂️ The Great Pipeline Unification

Perhaps not as momentus as [The Second Great Unification](https://en.wikipedia.org/wiki/Unification_of_theories_in_physics), *this* unification is much more useful for data pipelines! Data Integration sources & destinations can now be added as blocks in batch pipelines! 🤯

**What does this mean?** Using Mage, you can now perform integration (extract), transformation, and loading in the SAME pipeline using Singer sources and your favorite tools (dbt, Python, SQL)! Read more in our docs [here](https://docs.mage.ai/data-integrations/batch-pipelines).

This is like having Fivetran/Airbyte, dbt, _and_ a jupyter notebook all-in-one WITH engineering best practices built-in!

<center>
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/bcb185a8-bbf7-4cb6-9161-641184203212" width="50%"/>
</div>
</center>

👩🏻‍💻 Interactions - a no-code UI for configuring data pipelines

Another _huge_ update, Mage now let's you build templates to unlock data _at scale_. You can configure data pipelines that are fully customizable for stakeholders and consumers. Read more about interactions [here](https://docs.mage.ai/interactions/overview) and get started building today!

This functionality will go a long way for democratizing data pipelines and easing data workloads!

<center>
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/e161704c-a175-4b71-8ba4-2aaa3d0b967c" width="50%"/>
</div>
</center>

🤓 Granular API Permissions

Mage now supports granular API permissions on ANY action. Each permission can grant read and write operations on specific resources (e.g. API endpoints). One or more roles can be assigned to 1 or multiple users.

**What does this mean?** You can create permissions for your team at the _most granular level possible_. Mage is now completely governable for ANY action. Read more [here](https://docs.mage.ai/authentication/permissions/overview).

<center>
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/96758160-2fd3-40f0-a705-e198cdda9ecf" width="50%"/>
</div>
</center>

🥳 NEW MongoDB CDC Streaming Source

<center>
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/2053ca17-57a3-40a3-985f-ed12d4d73297" width="50%"/>
</div>
</center>

Mage now supports MongoDB CDC Streaming Sources! A big thanks to emincanoguz11 for the contribution!

*by emincanoguz11 in https://github.com/mage-ai/mage-ai/pull/3716*

🐛 Bug Fixes

* Fixed imports and removed flake8 check for taps/targets by Luishfs in https://github.com/mage-ai/mage-ai/pull/3740
* Fix fetching roles new for User by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3743
* Fix extracting update statement from SQL by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3749
* Convert schema name to uppercase when `disable_double_quotes` by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3761
* Fix loading data integration sources by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3773

💅 Enhancements & Polish

* Project-wide pipeline run filters by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3659
* Cron expression syntax error display for triggers in code by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3696
* Flyout menu overflow by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3759
* Support upsert in MongoDB streaming sink by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3774
* Support setting token in the header for API trigger by dy46 in https://github.com/mage-ai/mage-ai/pull/3674
* Update Markdown block docs with image support by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3744
* Set attributes from PubSub message to handler dict by fajrifernanda in https://github.com/mage-ai/mage-ai/pull/3736
* Display elapsed time on `datetime` hover rather than UTC by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3739

New Contributors

* fajrifernanda made their first contribution in https://github.com/mage-ai/mage-ai/pull/3736
* emincanoguz11 made their first contribution in https://github.com/mage-ai/mage-ai/pull/3716

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.34...0.9.35