Mage-ai

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/c0299243-62f1-4378-947f-079dd264430b" width="50%" />
</div>

What's Changed
🎉 Exciting New Features

🦆 MotherDuck Support

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/283e30f5-5712-4cf6-9c4f-d2a120a46faf" width="60%"/>
</div>

</br>

This one's for all the ducklings out there! In addition to supporting DuckDB, Mage now supports [MotherDuck](https://motherduck.com) destinations!

By specifying a `MOTHERDUCK_TOKEN` and adding a `md:` prefix to your DuckDB database, you can read/write from/to MotherDuck locations! Check it out and get started [here](https://docs.mage.ai/integrations/databases/MotherDuck).

*by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4533*

🤖 Support `Thick` mode in OracleDB

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/e25a012e-2d3c-402a-a624-c3dbd2670040" width="60%"/>
</div>

</br>

We like our OracleDB connections like we like our pancakes, **THICK** 🥞. By default, Mage's Oracle client runs in a "Thin" mode which connects directly to Oracle Database— this mode does not need Oracle Client libraries. However, some additional functionality is available when they're used.

Now, you can use the "Thick" mode in Mage to connect to OracleDB using the Oracle Client libraries!

Check out our [docs](https://docs.mage.ai/design/data-loading#oracledb) to get started or read more about the differences between "Thin" and "Thick" modes [here](https://python-oracledb.readthedocs.io/en/latest/user_guide/appendix_b.html#connection-handling-differences-between-thin-and-thick-modes).

_by matrixstone in https://github.com/mage-ai/mage-ai/pull/4421_

🗄️ Show unused block files in file browser

This is one of our _most_ requested features and we're excited to announce that it's finally here! 🎉

You can now see which files are not being used in your pipeline! This is a great way to clean up your projects and remove any unnecessary files. Check out the gif below to see it in action!

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/78053898/3c9e41aa-ba18-4f97-8cec-730ffcc7cb8a" width="70%"/>
</div>

<br/>

Head over to our [docs](https://docs.mage.ai/developer-ux/file-browser#unused-block-files) to learn more!

*by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4449*

🤐 Import functionality for pipeline zips

Like to share? So do we! You can now import pipelines (via `.zip` files) in your Mage projects! We're optimistic that this simple improvement will make it easier to share your pipelines or even borrow pipelines from your friends!

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/33767023/1b3dc95d-fa00-4054-a9fe-150e57b59d0a" width="70%"/>
</div>

<br/>

*by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4453*

🐛 Bug Fixes
* Prevent add block menu from disappearing by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4502
* Fix some minor bugs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4505
* Remove table name helper by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4506
* Fix alter table column names cleaning (Postgresql integration exporter) by arnetkachev in https://github.com/mage-ai/mage-ai/pull/4493
* Fixed Oauth connection on Salesforce Source by Luishfs in https://github.com/mage-ai/mage-ai/pull/4402
* Fix unclickable vertical scrollbar and jumping before panel by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4512
* Catch unknown host error by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4518
* Upgrade snowflake library version and fix datetime column type by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4524
* Only show files with correct prefix by dy46 in https://github.com/mage-ai/mage-ai/pull/4509
* Update submodule sync for ssh auth by dy46 in https://github.com/mage-ai/mage-ai/pull/4522
* Update connection url by dy46 in https://github.com/mage-ai/mage-ai/pull/4523
* Update how the stale pipeline message modal is displayed by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4536
* Fix running dynamic blocks with k8s executor by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4543
* Widget policy update by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4545
* Catch exception of building cache key for block cache by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4548
* Disable stale pipeline modal by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4525

💅 Enhancements & Polish
* MSSQL data integration source - add support for DATETIMEOFFSET type by hugabora in https://github.com/mage-ai/mage-ai/pull/4499
* Fix: handle unprivileged user for postgres by jdbranham in https://github.com/mage-ai/mage-ai/pull/4357
* Optimize pipeline schedule queries by dy46 in https://github.com/mage-ai/mage-ai/pull/4188
* Remove pipeline's updated_at attribute by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4521
* Create and view workspaces in different namespaces by dy46 in https://github.com/mage-ai/mage-ai/pull/4513
* Select status for pipeline runs that time out by dy46 in https://github.com/mage-ai/mage-ai/pull/4519

New Contributors
* arnetkachev made their first contribution in https://github.com/mage-ai/mage-ai/pull/4493
* jdbranham made their first contribution in https://github.com/mage-ai/mage-ai/pull/4357
* tanjibpa made their first contribution in https://github.com/mage-ai/mage-ai/pull/4528

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.62...0.9.63

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/9ed42fcb-586f-427c-bdb8-55a17bef978b" width="60%"/>
</div>

What's Changed
🎉 Exciting New Features

🧵 [Mage IO] Weaviate Integration

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/cb530191-e304-4c6a-8449-01dffc99ea9c" width="60%"/>
</div>

🤔 Building AI apps with Mage? Perfect! Now you can use [Weaviate](https://weaviate.io/) as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs [here](https://docs.mage.ai/integrations/databases/Weaviate).

*by matrixstone in https://github.com/mage-ai/mage-ai/pull/4158*

🔍 [Mage IO] Alogia Integration

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/449e7e73-1c16-465b-a485-43144ec0ac63" width="60%"/>
</div>

Like search? Us too! That's why we've added support for [Alogia](https://www.alogia.com/) as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs [here](https://docs.mage.ai/integrations/databases/Algolia).

*by matrixstone in https://github.com/mage-ai/mage-ai/pull/4198*

💥 Dynamic SQL Blocks

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/1066980/913a6c29-dc74-4ca9-9388-13eb265c0a46" width="60%"/>
</div>

Another big improvement to dynamic blocks this week _SQL_ Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. 🎉

*by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4430*

🌊 [Kafka Streaming Sources] Offset & Partitions

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/8bf08bc0-2fdd-45e2-8824-a9b7d955d40d" width="60%"/>
</div>

For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: `Beginning`, `End`, `Int`, & `Timestamp`. This allows users to set specific positions inside a topic to consume data.

*by Luishfs in https://github.com/mage-ai/mage-ai/pull/4226*

⛴️ Configure Kubernetes Affinity & Tolerations

Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify `affinity` and `tolerations` in your Kubernetes settings. ⚓

Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.

Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.

These configurations should help our Kubernetes power users make the most of Mage! 🧙

*by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4407*


🐛 Bug Fixes
* Fix LDAP unpacking by dy46 in https://github.com/mage-ai/mage-ai/pull/4399
* Fix reduce output and triple layout saving by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4409
* Fix file browser bug and Git push bug by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4411
* Fix serializing list and dict when formatting output by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4412
* Prevent index out of bounds by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4425
* Remove test print statements by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4431
* Removing Draft7 validation from Clickhouse destination by Luishfs in https://github.com/mage-ai/mage-ai/pull/4424
* Fix `global_vars` context in pipeline executor by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4435
* Fix a few global data product bugs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4440
* Fix dynamic blocks OOM round 2 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4445
* Fix incremental sync in chargebee source by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4450
* Don’t count values if is None by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4454
* Fix policy issue updating settings by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4456
* Fix keyboard shortcuts when its empty array by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4458
* Run submodule sync by dy46 in https://github.com/mage-ai/mage-ai/pull/4457
* Fixing Snowflake `write_pandas` issue by Luishfs in https://github.com/mage-ai/mage-ai/pull/4395
* Fix writing to Snowflake with mixed int and str types by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4460
* Fix some bugs and improve the edit page by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4462
* Fix error logging in pipeline executor by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4468
* When clicking show file versions in arcane library, show right panel by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4472
* Catch BigQuery if it fails to fix table names by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4479
* Fix pipeline detail prop passed for fetching files by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4485
* Always show edit pipeline button by dy46 in https://github.com/mage-ai/mage-ai/pull/4484
* Revert change to PG IO by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4486
* Fix io redshift by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4487
* Fix multi project flag by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4490
* Fix Bigquery clean column name by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4500
* Convert datetime type for s3 data integration destination by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4501
* Fix dynamic child block outputs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4422
* Fix database missing and serializing QueryJob by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4428
* Fix incorrect spelling by ckfear in https://github.com/mage-ai/mage-ai/pull/4438
* Fix kafka type and doc by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4466

💅 Enhancements & Polish
* Workspace improvements by dy46 in https://github.com/mage-ai/mage-ai/pull/4469
* Scheduler improvements by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4467
* Prevent unnecessary initial pipeline run by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4291
* Speed up monitor stats and reduce calls on overview by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4408
* Bump up dependency versions to resolve vulnerabilities by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4433
* Limit dynamic block output in notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4436
* Improve bigquery name parsing to help fill in full name by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4447
* Add error logging by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4448
* Fix app slowness due to project platform check by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4483
* Add PVC retention policy by dy46 in https://github.com/mage-ai/mage-ai/pull/4491
* E2e test for `/pipelines` by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4306
* Add query decorator for data integration blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4465


New Contributors
* ckfear made their first contribution in https://github.com/mage-ai/mage-ai/pull/4438

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.60...0.9.62

<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/297928872-381a88e2-50a1-4545-9fae-c9a752427f7b.gif" width="60%"/>
</div>

What's Changed
🎉 Exciting New Features
🌊 [Streaming] Google Cloud Storage Destination

<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/297931125-1787abd8-2c1c-454c-8f2b-3892337d3120.png" width="80%"/>
</div>

🎉 Google Cloud users rejoice! Streaming pipelines just even got better— Mage now supports Google Cloud Storage as a streaming destination! Check out the docs [here](https://docs.mage.ai/streaming/destinations/google-cloud-storage) and get started today!

_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4340_

👷‍♂️ Overwrite SQL types

For anyone with a data warehouse, listen up! (We assume that's most of you 😅)

You can now specify custom column types when exporting to SQL destinations. This is useful when you want to export a dataframe with a column that has a type that is not supported by the default mapping. You can read more about overwriting types [here](https://docs.mage.ai/integrations/databases/PostgreSQL#overwrite-column-types).

Here's an example of an exporter that overwrites column types for a PostgreSQL destination:

python
data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' Specify the name of the schema to export data to
table_name = 'your_table_name' Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}

with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, Specifies whether to include index in exported table
if_exists='replace', Specify resolution policy if table name already exists
allow_reserved_words=True,
unique_conflict_method='UPDATE',
unique_constraints=['col'],
overwrite_types=overwrite_types,
)


This feature is currently supported for PostgreSQL, Redshift, Trino, MSSQL, MySQL, Clickhouse, and BigQuery.

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/3474_

👨‍💻 [Command Center] Version Control & Files

The Mage _Command Center_ can now be used for version control commands! You can both configure git _and_ run your favorite version control commands directly from the Mage UI. Additionally, you can manage your files via nav and Mage's new file explorer (🧙 Arcane Library)!

<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/298133980-a8998249-1626-4f04-8a19-be068d921bb0.png" width =49%>
<img src="https://user-images.githubusercontent.com/59450879/298134025-563be746-6674-48af-a9f6-b30b4ca3424b.png" width =49%>
</div>


As a reminder, to enable the command center, you can do so by going to `Settings` (the wizard icon) and click the `Command Center` toggle. It can be accessed via `⌘ + .` (Mac) or `Win + .` (Windows).

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4273_

👾 [Command Center] Terminal App

Mage LEGEND tommydangerous is back at it again— he's implemented a full terminal app into the command center. For those of you with this *beta* feature enabled, you'll now have a terminal at your fingertips _at all times_.

Check out the following video for sample usage:

https://github.com/mage-ai/mage-ai/assets/1066980/82db6eb4-ecca-49e4-aa77-f464e6a94620

As a reminder, to enable the command center, you can do so by going to `Settings` (the wizard icon) and click the `Command Center` toggle. It can be accessed via `⌘ + .` (Mac) or `Win + .` (Windows).

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4365_


JSON Logging

A huge shoutout to dy46 for adding JSON logging to Mage! This will make it easier to parse logs and integrate with other logging tools. Just specify `SERVER_LOGGING_FORMAT=json` to change the output to something like the following:

<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/297930508-bc4df2ee-186f-4eb0-a671-fbbe0dc73243.png" width="90%"/>
</div>

_by dy46 in https://github.com/mage-ai/mage-ai/pull/4336_


🐛 Bug Fixes
* Fix caching issues with block cache and shared pipelines by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4338
* Fix SQL blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4341
* Prevent error when searching for blocks by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4343
* Fix callbacks input data from dynamic child blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4342
* Enable command center when user auth not required by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4346
* Fix bug when searching for block files by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4347
* Fix dynamic child block getting input data by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4349
* Fix cron expression conversion when using local midnight time by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4359
* Fix block search by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4360
* Fix command center hiding by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4361
* Fix GDP and add terminal colors by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4363
* Minor tweaks to existing apps by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4367
* Added `custom_fields` to freshdesk source by Luishfs in https://github.com/mage-ai/mage-ai/pull/4354
* Fix creating widget by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4375
* Lowercase `auth_type` enum by dy46 in https://github.com/mage-ai/mage-ai/pull/4376
* Update cloud run workspace by dy46 in https://github.com/mage-ai/mage-ai/pull/4377
* Fix terminal by dy46 in https://github.com/mage-ai/mage-ai/pull/4389
* Fix pipeline run variable overwrite for sql block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4390
* Update dynamic block output and input data logic by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4388
* Fix dynamic block conditionals in runs and in notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4397
* Remove terminal colors by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4398
* Fix reduce output block tests by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4400
* Removing modified `google-ads` lib by Luishfs in https://github.com/mage-ai/mage-ai/pull/4330

💅 Enhancements & Polish
* Show multiple outputs and fix downstream dynamic child block inputs and outputs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4382
* Improve command center shortcut wording and example by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4348
* Add mapping for active directory roles by dy46 in https://github.com/mage-ai/mage-ai/pull/4345
* Make block type error more descriptive by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4353
* Upgrade app layout behavior by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4362
* Support overwriting column types in BigQuery by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4374
* Add spark jar files to `emr_config` if using EMR cluster by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4379

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.59...0.9.60

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.59 -->

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/50c5084f-86fe-4543-9f40-e5bfeaa55330" width="60%"/>
</div>

What's Changed
🎉 Exciting New Features

**Note:** many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/2af3f074-aeb5-4fda-b3cf-4cead6d945fd" width="60%"/>
</div>

🎮 Multi-project Platform [BETA]

We've reworked our support for mulitple projects with a new multi-project platform! tommydangerous is back at it again with this *huge* feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/80a88de4-615c-431a-9d8f-1d9e61af523a

If you'd like to try out the multi-project platform, you can check out [this repo](https://github.com/mage-ai/platform_template) for a sample structure. Head over to your Mage settings to enable

* Support nested projects and custom code paths by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4161
* Trigger and run pipelines across projects by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4186
* Split pipeline scheduler and schedule models for project platform by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4233
* Configure root project preferences and settings by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4234
* Fix pipeline schedule creation and repo name by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4247
* Don't try to use file source if not exist by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4196
* Fix the way we store pipelines by type in the cache by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4278

🚀 Command Center [BETA]

Another new & exciting feature this week— the Mage **Command Center**. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!

https://github.com/mage-ai/mage-ai/assets/59450879/a01a7f11-a0c8-43f1-9a44-253202659c75

Enable the Command Center in settings and give it a spin today!

* Command center by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4249
* Add command center models by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4254

🪣 Bitbucket Version Control

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/337f50eb-9728-472c-9781-64e00f5fdce2" width="55%"/>
</div>

Shout out to dy46 for continuing to crush the version control integrations!

You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more [here](https://docs.mage.ai/getting-started/setting-up-git).

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/efae5852-7c77-4d1f-8756-fae19621bca1" width="55%"/>
</div>

*by dy46 in https://github.com/mage-ai/mage-ai/pull/4176*

◔ Qdrant integration

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/3ee8f951-f2c4-44be-b790-9a7c85b06878" width="55%"/>
</div>

Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more [here](https://docs.mage.ai/integrations/databases/Qdrant).

*by matrixstone in https://github.com/mage-ai/mage-ai/pull/4081*

🧱 dbt DX v2 + dbt Upgrade [BETA]

This release contains a _huge_ dbt overhaul 🤯

Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!

Here's a quick demo:

https://github.com/mage-ai/mage-ai/assets/59450879/c6518d08-fd32-4bbb-b271-9aa28409b41d

* Upgrade dbt to 1.7 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4244
* dbt v2 browser UI by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4200
* Add block browser modal to notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4246
* Add dbt cache by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4193
* Add fields to dbt code block 2.0 for manual entry by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4331
* Use absolute paths in dbt block by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4307
* Add custom code block tags by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4250
* Collapse or expand folders by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4203

🐛 Bug Fixes

SQL blocks
* Remove double quotes for postgres by dy46 in https://github.com/mage-ai/mage-ai/pull/4170
* Escape BigQuery project name in SQL block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4294
* Pass in `query_vars` as a dict by dy46 in https://github.com/mage-ai/mage-ai/pull/4280

Data integration
* Fix syncing MySQL `TIME` type by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4275
* Bump Google Ads version from 14 to 15 by Luishfs in https://github.com/mage-ai/mage-ai/pull/4289
* Fix incremental sync bug with missing arg by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4315
* Fix executing data integration block with ecs executor by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4322
* Wrap MSSQL table name with double quotes in data integration pipeline by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4290

Trigger and scheduling
* Use UTC date for trigger start date by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4283
* Display correct default start datetime when editing trigger/backfill by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4292
* Try preventing creating duplicate pipeline runs in scheduler by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4311
* Wrap block run initialization logic with lock by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4296

dbt
* Fix dbt profiles interpolation by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4225
* Lazy import for dbt files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4251
* Re-work dbt project path for `yaml` files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4207
* Error handling for "project not found" in dbt by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4334
* Adjust error behavior when adding dbt files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4191

File browser
* Fix files page not opening files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4222
* File browser bug bash by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4237
* Fix bug when deleting block from file browser by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4243
* Fix `requestIdleCallback` not supported on Safari by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4284
* Fix file browser not refreshing by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4223

Dynamic blocks
* Fix dynamic block + dynamic child blocks spawning other blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4295
* Fix reduce output bug by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4326

Git
* Add actions to `GitBranchPolicy` by dy46 in https://github.com/mage-ai/mage-ai/pull/4213
* Fix Git bugs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4264
* Fix Git submodule sync by dy46 in https://github.com/mage-ai/mage-ai/pull/4316
* Fix missing Git module by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4206

Other
* Fix silent errors from global hooks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4173
* Fix block sorting bug by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4175
* Avoid saving error details in block run DB by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4179
* Reset page after applying pipeline filters by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4183
* Restrict opentelemetry package versions by dy46 in https://github.com/mage-ai/mage-ai/pull/4208
* Fix dataframe validation by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4255
* Fix add new button tooltips by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4263
* Save `statistics.json` in correct execution partition folder by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4271
* Fix dashboard resizing by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4282
* Only set the schema in the DB when the server is started by dy46 in https://github.com/mage-ai/mage-ai/pull/4293
* Fix several bugs on Pipelines dashboard by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4300
* Improve UI in several areas by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4309
* Fix dragging and dropping file in browser by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4312
* Fix interpolating mage secret in project metadata.yaml by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4319
* Remove undefined request by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4324
* Fix triple layout after width by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4325
* Fix cannot unpack non-iterable NoneType object by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4327
* Add small fixes by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4329
* Bug fixes and minor improvements on version control app by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4267
* Fix a ton of bugs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4214
* Fix escaping quotes in R block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4285


💅 Enhancements & Polish

File browser
* Improve folder performance by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4221
* View or hide hidden files in file browser by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4235
* Re-use file browser and editor in notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4231

Raw SQL enhancements
* Disable sample query when using raw SQL by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4252
* Add execute raw SQL without preprocessing to all loaders by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4303

Workspace management
* Add default values for workspace creation by dy46 in https://github.com/mage-ai/mage-ai/pull/4219
* Use current pod image as the default by dy46 in https://github.com/mage-ai/mage-ai/pull/4299
* Update url logic for workspace manager

Other
* Update Git settings when environment variables are set in https://github.com/mage-ai/mage-ai/pull/4154 in https://github.com/mage-ai/mage-ai/pull/4166
* Global hooks for platform by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4236
* Decrease image size by removing Qdrant libraries by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4277
* Update Git to set username and email when using version control by dy46 in https://github.com/mage-ai/mage-ai/pull/4262
* Dynamic dynamic child and replicas by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4308
* Speed up pipeline run count query by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4321
* Rewrite schedule queries by dy46 in https://github.com/mage-ai/mage-ai/pull/4185
* Update custom block template by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4177
* Support serialize Polars dataframe by dy46 in https://github.com/mage-ai/mage-ai/pull/4281
* Add `queue_subscription` support for NATS Jetstream streaming source by atjhoendz in https://github.com/mage-ai/mage-ai/pull/4169
* Added URL protocol option to RabbitMQ streaming by Luishfs in https://github.com/mage-ai/mage-ai/pull/4228
* AWS S3 Destination: Added support for IAM role-based authentication by eofs in https://github.com/mage-ai/mage-ai/pull/4320
* Added `role_arn` as an optional config for AWS DynamoDB source by robreinhold in https://github.com/mage-ai/mage-ai/pull/4256
* SQL block 1.5 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4302
* Add retry info in callbacks and retry function by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4332
* Add custom 404 page by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4149

New Contributors
* aliavni made their first contribution in https://github.com/mage-ai/mage-ai/pull/4181
* atjhoendz made their first contribution in https://github.com/mage-ai/mage-ai/pull/4169
* robreinhold made their first contribution in https://github.com/mage-ai/mage-ai/pull/4256

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.50...0.9.59

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/68865221-095c-4e6a-b5bd-01f680756a46" width="65%"/>
</div>

What's Changed
🎉 Exciting New Features
🌊 [Data Integration] Dremio Source

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/d4c781f0-433c-461f-931f-5ee51b5ecd0e" width="65%"/>
</div>

</br>

🥳 Dremio users, rejoice! Mage now supports Dremio as a [data integration source](https://docs.mage.ai/data-integrations/overview), meaning you can now build data integration pipelines pulling from data lakes and more!

_by Luishfs in https://github.com/mage-ai/mage-ai/pull/4072_

🏃‍♂️ Manually run pipeline once in same trigger

This update, courtesy of our frontend engineer, Johnson, is a big one!

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/78053898/7cd95dee-c0be-4a8e-9c19-4a62fa9bd075" width="65%"/>
</div>

</br>

First, he added a new button to the Trigger Detail page for running pipelines once in the same trigger! 👀

Next, some quality of life improvements:
- For `once` triggers, the trigger does not need to be `active` status before running the pipeline once using the `Runonce` button in the Trigger Detail page. _Many_ users have told us this is confusing... No more!
- The trigger will automatically be updated to `active` status. However, if the trigger is NOT an `once` trigger (e.g. a recurring interval or API trigger), the trigger must be set to `active` status before manually running the pipeline once from the Trigger Detail page.

Finally, Johnson renamed the "start/pause" trigger on Trigger Detail page to "enable/disable" to be a bit more clear and communicative. Again, we've heard this is a bit misleading, so we did something about it! 🗣️

_by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4133_

[Streaming] ActiveMQ Sink

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/bb49f0eb-2a20-4c5d-807d-c31e3fdad961" width="65%"/>
</div>

</br>

Shruti continues her _epic_ tear of contributing _magical_ ✨ pipelines. In this PR, she's added an ActiveMQ streaming sink to Mage. Apache ActiveMQ is an open source message broker written in Java... and now you can write data there via Mage! 💫

_by shrutimantri in https://github.com/mage-ai/mage-ai/pull/4141_

🐛 Bug Fixes
* Fix stale pipeline message not appearing by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4138
* Fixed Salesforce Source not running sync by Luishfs in https://github.com/mage-ai/mage-ai/pull/4048
* Fixed Salesforce Destination Upsert action by Luishfs in https://github.com/mage-ai/mage-ai/pull/4130
* Fix external cloud storage logs for k8s blocks by dy46 in https://github.com/mage-ai/mage-ai/pull/4128
* Update DownloadPolicy to allow downloading pipeline zip files by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4148
* Fix pagination by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4152
* Add tests and improve logs table by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4156
* Add tags to pipeline cache by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4162
* Disable error and UI limiting dynamic blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4164
💅 Enhancements & Polish
* Trigger global hooks on pipeline execution by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4147
* Close modal after saving by dy46 in https://github.com/mage-ai/mage-ai/pull/4131
* Speed up pipelines list API operation by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4132
* Add resource parent in the input data for hooks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4144
* Optimize pipeline schedules LIST api by dy46 in https://github.com/mage-ai/mage-ai/pull/4058
* Fix block runs page when there are a large number of block runs by dy46 in https://github.com/mage-ai/mage-ai/pull/4109
* Dynamic blocks 2.0 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4157
* Add more tests for pipeline execute global hook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4159
* Update git settings when environment variables are set by dy46 in https://github.com/mage-ai/mage-ai/pull/4154
* Update url logic for workspace manager by dy46 in https://github.com/mage-ai/mage-ai/pull/4166

New Contributors
* Abiodun made their first contribution in https://github.com/mage-ai/mage-ai/pull/4125

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.48...0.9.50

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/f932803b-c696-4231-84c3-d963d1b487fc">
</div>

What's Changed
🎉 Exciting New Features

Incremental data integration in batch pipelines

🥳 Data integrations in batch pipelines now support incremental replication! You can read more [here](https://docs.mage.ai/data-integrations/batch-pipelines) to get started!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4068_

[Streaming] RabbitMQ Destination

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/ffe32fe8-bf74-4e4f-8da3-7c9189e2613a" width="40%">
</div>
</br>

_Another_ community PR from shrutimantri adds support for RabbitMQ as a streaming data sink. 🔥

Check it out today with your favorite streaming sources! You can find the configuration reference [here](https://docs.mage.ai/streaming/destinations/rabbitmq).

_by shrutimantri in https://github.com/mage-ai/mage-ai/pull/4041_

Chroma integration

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/75615fb2-e569-4eb9-9541-019d23467f26" width="40%">
</div>
</br>

Mage now has a ChromaDB IO Class, meaning you can use data loaders and exporters in your batch pipelines to read/write from Chroma sources. You can read more about configuration [here](https://docs.mage.ai/integrations/databases/Chroma) or visit [Chroma's site](https://www.trychroma.com/) to learn more about their vector database.

_by matrixstone in https://github.com/mage-ai/mage-ai/pull/4017_

Bookmark overrides

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/7585e2ac-650d-4881-8b3c-2d599474a987" width="70%">
</div>
</br>

🎊 If you're creating a trigger on a data integration, you can now override bookmarks with your own custom values!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4073_

SQL Block environment variable interpolation

For our fans of SQL blocks, you can now interpolate environment variables directly in your queries!

sql
SELECT
'{{ env_var("ENV") }}' AS test
, '{{ variables("test") }}' AS test2
, {{ test }} AS test3


This should allow for _much_ greater flexibility in pipelines with SQL!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4076_

Additional upstream dependencies for dynamic children

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/59450879/ddb80a64-a094-4da4-a087-a34fa8d96a4e" width="50%">
</div>
</br>

Love dynamic blocks? 🤔 They dynamic children can now have additional upstream dependencies!

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4104_

Support caching block output in memory

Previously, pipelines with large Spark DataFrames faced out of heap space errors when persisting block outputs to disk. This PR allows the user to disable persisting output. The feature is only supported in standard batch pipeline (without dynamic blocks) for now.

yaml
cache_block_output_in_memory: true
run_pipeline_in_one_process: true


_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4127_

🐛 Bug Fixes
* Backend API for getting information about bookmarks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4070
* Support different operators when comparing bookmark properties by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4075
* Update backfill statuses by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3994
* Reduce block at any level UI by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4067
* Catch execption of empty integration streams in pipeline scheduler by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4083
* Backfill's date-picker date value mismatch by edmondwinston in https://github.com/mage-ai/mage-ai/pull/3972
* Gracefully access dictionaries in the Oauth Policy by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4086
* Pass tolerations to job pod by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4089
* Fix load sample data for integration pipelines by dy46 in https://github.com/mage-ai/mage-ai/pull/4034
* Default to using environment variables for git and workspace settings by dy46 in https://github.com/mage-ai/mage-ai/pull/4088
* Fixed Google Ads Source by Luishfs in https://github.com/mage-ai/mage-ai/pull/4099
* Fix chromadb dependency by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4107
* Fix chromadb in all package by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4108
* Update local timezone project setting from header by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4111
* Fix runtime variables not showing when creating new trigger by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4116
* Fix executing conditional blocks with pipeline executor by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4120
* Remove itertools groupby by dy46 in https://github.com/mage-ai/mage-ai/pull/4103
* Updates/nats add stream fixes by mfreeman451 in https://github.com/mage-ai/mage-ai/pull/4113
* Update `opentelemetry-exporter-prometheus` package version by dy46 in https://github.com/mage-ai/mage-ai/pull/4101
* Fix postgres streaming sink when there are no messages by shrutimantri in https://github.com/mage-ai/mage-ai/pull/4074

💅 Enhancements & Polish
* Add any runtime variables by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4071
* Include `message_events_json` in Postmark messages_outbound stream by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4085
* Hide "Unique" and "Key" columns for certain data integration destination blocks by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4096
* Add top padding to file code editor by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4098
* Display error in UI when variables directories configured incorrectly by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4091
* Improve the interface for Chroma class by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4110
* Add `column_header_format` option by dy46 in https://github.com/mage-ai/mage-ai/pull/4118
* Allow configuring Amplitude host by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4060

New Contributors
* andrewgetzdata made their first contribution in https://github.com/mage-ai/mage-ai/pull/4078
* suvhotta made their first contribution in https://github.com/mage-ai/mage-ai/pull/4097

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.46...0.9.48