<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/297928872-381a88e2-50a1-4545-9fae-c9a752427f7b.gif" width="60%"/>
</div>
What's Changed
🎉 Exciting New Features
🌊 [Streaming] Google Cloud Storage Destination
<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/297931125-1787abd8-2c1c-454c-8f2b-3892337d3120.png" width="80%"/>
</div>
🎉 Google Cloud users rejoice! Streaming pipelines just even got better— Mage now supports Google Cloud Storage as a streaming destination! Check out the docs [here](https://docs.mage.ai/streaming/destinations/google-cloud-storage) and get started today!
_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4340_
👷♂️ Overwrite SQL types
For anyone with a data warehouse, listen up! (We assume that's most of you 😅)
You can now specify custom column types when exporting to SQL destinations. This is useful when you want to export a dataframe with a column that has a type that is not supported by the default mapping. You can read more about overwriting types [here](https://docs.mage.ai/integrations/databases/PostgreSQL#overwrite-column-types).
Here's an example of an exporter that overwrites column types for a PostgreSQL destination:
python
data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' Specify the name of the schema to export data to
table_name = 'your_table_name' Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, Specifies whether to include index in exported table
if_exists='replace', Specify resolution policy if table name already exists
allow_reserved_words=True,
unique_conflict_method='UPDATE',
unique_constraints=['col'],
overwrite_types=overwrite_types,
)
This feature is currently supported for PostgreSQL, Redshift, Trino, MSSQL, MySQL, Clickhouse, and BigQuery.
_by Luishfs in https://github.com/mage-ai/mage-ai/pull/3474_
👨💻 [Command Center] Version Control & Files
The Mage _Command Center_ can now be used for version control commands! You can both configure git _and_ run your favorite version control commands directly from the Mage UI. Additionally, you can manage your files via nav and Mage's new file explorer (🧙 Arcane Library)!
<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/298133980-a8998249-1626-4f04-8a19-be068d921bb0.png" width =49%>
<img src="https://user-images.githubusercontent.com/59450879/298134025-563be746-6674-48af-a9f6-b30b4ca3424b.png" width =49%>
</div>
As a reminder, to enable the command center, you can do so by going to `Settings` (the wizard icon) and click the `Command Center` toggle. It can be accessed via `⌘ + .` (Mac) or `Win + .` (Windows).
_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4273_
👾 [Command Center] Terminal App
Mage LEGEND tommydangerous is back at it again— he's implemented a full terminal app into the command center. For those of you with this *beta* feature enabled, you'll now have a terminal at your fingertips _at all times_.
Check out the following video for sample usage:
https://github.com/mage-ai/mage-ai/assets/1066980/82db6eb4-ecca-49e4-aa77-f464e6a94620
As a reminder, to enable the command center, you can do so by going to `Settings` (the wizard icon) and click the `Command Center` toggle. It can be accessed via `⌘ + .` (Mac) or `Win + .` (Windows).
_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4365_
JSON Logging
A huge shoutout to dy46 for adding JSON logging to Mage! This will make it easier to parse logs and integrate with other logging tools. Just specify `SERVER_LOGGING_FORMAT=json` to change the output to something like the following:
<div align="center">
<img src="https://user-images.githubusercontent.com/59450879/297930508-bc4df2ee-186f-4eb0-a671-fbbe0dc73243.png" width="90%"/>
</div>
_by dy46 in https://github.com/mage-ai/mage-ai/pull/4336_
🐛 Bug Fixes
* Fix caching issues with block cache and shared pipelines by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4338
* Fix SQL blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4341
* Prevent error when searching for blocks by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4343
* Fix callbacks input data from dynamic child blocks by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4342
* Enable command center when user auth not required by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4346
* Fix bug when searching for block files by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4347
* Fix dynamic child block getting input data by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4349
* Fix cron expression conversion when using local midnight time by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4359
* Fix block search by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4360
* Fix command center hiding by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4361
* Fix GDP and add terminal colors by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4363
* Minor tweaks to existing apps by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4367
* Added `custom_fields` to freshdesk source by Luishfs in https://github.com/mage-ai/mage-ai/pull/4354
* Fix creating widget by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4375
* Lowercase `auth_type` enum by dy46 in https://github.com/mage-ai/mage-ai/pull/4376
* Update cloud run workspace by dy46 in https://github.com/mage-ai/mage-ai/pull/4377
* Fix terminal by dy46 in https://github.com/mage-ai/mage-ai/pull/4389
* Fix pipeline run variable overwrite for sql block by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4390
* Update dynamic block output and input data logic by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4388
* Fix dynamic block conditionals in runs and in notebook by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4397
* Remove terminal colors by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4398
* Fix reduce output block tests by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4400
* Removing modified `google-ads` lib by Luishfs in https://github.com/mage-ai/mage-ai/pull/4330
💅 Enhancements & Polish
* Show multiple outputs and fix downstream dynamic child block inputs and outputs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4382
* Improve command center shortcut wording and example by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4348
* Add mapping for active directory roles by dy46 in https://github.com/mage-ai/mage-ai/pull/4345
* Make block type error more descriptive by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4353
* Upgrade app layout behavior by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4362
* Support overwriting column types in BigQuery by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4374
* Add spark jar files to `emr_config` if using EMR cluster by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4379
**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.59...0.9.60