Mage-ai

Mage launched a new backfill framework to make backfills a lot easier. User can select a date range and date interval for backfill. Mage will automatically create the pipeline runs within the date range, and run them concurrently to backfill the data.

Docs

- Backfill framework overview: [https://docs.mage.ai/orchestration/backfills/overview](https://docs.mage.ai/orchestration/backfills/overview)
- Backfill guide: [https://docs.mage.ai/orchestration/backfills/guides](https://docs.mage.ai/orchestration/backfills/guides)



Data integration

New sources

- [Microsoft SQL Server](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/mssql/README.md)
- [Outreach](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/outreach/README.md)
- [LinkedIn Ads](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/linkedin_ads/README.md)
- [Paystack](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/paystack/README.md)
- [Postmark](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/postmark/README.md)

New destinations

- [Trino](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/trino/README.md) (all connectors)

Streaming pipeline

Add Kinesis as streaming source and destination(sink) to streaming pipeline.

- For Kinesis streaming source, configure the source stream name and batch size.
- For Kinesis streaming destination(sink), configure the destination stream name and partition key.
- To use Kinesis streaming source and destination, make sure the following environment variables exist:
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_REGION`


Kubernetes support

- Support running Mage in Kubernetes locally: [https://docs.mage.ai/getting-started/setup#using-kubernetes](https://docs.mage.ai/getting-started/setup#using-kubernetes)
- Support executing blocks in separate Kubernetes jobs: [https://docs.mage.ai/production/configuring-production-settings/compute-resource#kubernetes-executor](https://docs.mage.ai/production/configuring-production-settings/compute-resource#kubernetes-executor)

yaml
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
executor_type: k8s
...


- When managing dev environment in Kubernetes cluster, allow adding custom config for the mage container in Kubernetes.


DBT improvements

- Support running MySQL DBT models in Mage.
- When adding a DBT block to run all/multiple models, allow manual naming of the block.

Metaplane integration

Mage can run monitors in Metaplane via API integration. Check out the [guide](https://docs.mage.ai/integrations/metaplane) to learn about how to run monitors in Metaplane and poll statuses of the monitors.

Other bug fixes & polish

- SQL block: support SSH tunnel connection in Postgres SQL block
- Follow this [guide](https://docs.mage.ai/integrations/databases/PostgreSQL#ssh-tunneling) to configure Postgres SQL block to use SSH tunnel
- R block: support accessing runtime variables in R block
- Follow this [guide](https://docs.mage.ai/guides/r-blocks#runtime-variables) to use runtime variables in R blocks
- Added setting to skip current pipeline run if previous pipeline run hasn’t finished.


- Pass runtime variables to test functions. You can access runtime variables via `kwargs['key']` in test functions.

View full [Changelog](https://www.notion.so/mageai/What-s-new-7cc355e38e9c42839d23fdbef2dabd2c)

<!-- Release notes generated using configuration in .github/release.yml at 5f5ea5f831066dbb002c2c11ce3820906bd0cc1b -->

<div align="center">
<img src="https://media2.giphy.com/media/v1.Y2lkPTc5MGI3NjExMmx4bG92ajNtamVqOGhxZnVicms2c2s3bjhsaGFmb3V0ZDk3cTh6aiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/qgg7LoT5SmlqYFZQLB/giphy.gif" width="50%" />
</div>

What's Changed
🎉 Exciting New Features
* [xy] Add teradata source and destination to data integration pipeline by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5704
Airtable IO Class by TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5419
* [hw] Update dbt related modules to the latest 1.8.* versions by csharplus in https://github.com/mage-ai/mage-ai/pull/5490
* Feat/support doris by yugasun in https://github.com/mage-ai/mage-ai/pull/5732
* [xy] Support hiding and rotating API trigger token. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5625
* [xh] Add allow_local_infile to MySQL config to enable support for "LOAD DATA LOCAL INFILE" by matrixstone in https://github.com/mage-ai/mage-ai/pull/5679
🐛 Bug Fixes
* [hw] Resolve the input path issue with R transformer blocks by csharplus in https://github.com/mage-ai/mage-ai/pull/5606
* [xh]Fix disable query preprocessing by matrixstone in https://github.com/mage-ai/mage-ai/pull/5653
* [xy] Resolve dbt library conflicts. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5664
* [xy] Add safe_db_query_async decorator. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5665
* [xy] Pass execution_partition to interpolate content method. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5667
* [xy] Fetch git secret with repo query. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5677
* [xy] Fix table not defined error in io bigquery. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5678
* [xh] Support multiple statements processing in redshift by matrixstone in https://github.com/mage-ai/mage-ai/pull/5685
* [xy] Fix MySQL error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5687
* [xy] Fix bigquery write with replace policy. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5691
* [xy] Only allow api trigger endpoint for API trigger. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5705
* [xy] Fix API trigger request authentication. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5707
* [xy] Fix data integration pipeline run metrics. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5716
* [xy] Fix arguments in some data integration sources/destinations. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5737
💅 Enhancements & Polish
* [xy] Support key authentication in snowflake source and destination. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5741
* [hw] Update the version for the google-ads module by csharplus in https://github.com/mage-ai/mage-ai/pull/5601
* [xy] Upgrade package versions. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5742
* [xy] Support pool_size and max_overflow in mssql connect kwargs. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5655
* [xy] Auto cast timestamp columns for bigquery exporter. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5676
* [xy] Not save token to triggers.yaml if HIDE_API_TRIGGER_TOKEN is true. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5686
* [jk] User management pagination by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5735
* [xy] Update event logging to use sync method in scheduler by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5740
* enhancement: Upgrade DuckDB to the latest stable version by elefeint in https://github.com/mage-ai/mage-ai/pull/5646
* [hw] Update the version of the nkeys module by csharplus in https://github.com/mage-ai/mage-ai/pull/5604
* [xy] Move help improve_mage check outside of asyncio.run by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5624
* [xy] Support configuring redis default timeout by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5639
* [xy] Support passing chunksize and method to mssql export method. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5643
Other Changes
* Fix typo by oscarlofwenhamn in https://github.com/mage-ai/mage-ai/pull/5649
* [hw] Drop the support for Python 3.8 as it is EOL officially by csharplus in https://github.com/mage-ai/mage-ai/pull/5652
* [hw] Update a few out-of-date dbt doc links by csharplus in https://github.com/mage-ai/mage-ai/pull/5594
* Update prefect.mdx by McCalebTheSecond in https://github.com/mage-ai/mage-ai/pull/5728

New Contributors
* elefeint made their first contribution in https://github.com/mage-ai/mage-ai/pull/5646
* McCalebTheSecond made their first contribution in https://github.com/mage-ai/mage-ai/pull/5728
* yugasun made their first contribution in https://github.com/mage-ai/mage-ai/pull/5732

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.75...0.9.76

<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExeWR5NHRraGRiejl3dzZqZWxvMWJwdGxvajMxbGhwMG9xc3BjMzNrNSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/Da3Ubr6t6qtJeSJUU8/giphy-downsized.gif" width="50%" />
</div>

What's Changed
🎉 Exciting New Features
📋 Airtable Destination
Airtable as a now a destination sync in data integration pipelines. This means you can easily export your data directly to Airtable, keeping everything up to date without any hassle.

<div align="center">
<img src="https://github.com/mage-ai/assets/blob/main/pipelines/integration-pipelines/gcs_integration_test.png" width="600"/>
</div>
by TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5454

🐍 Support Mage in Python 3.11 and 3.12
Mage now supports Python versions 3.11 and 3.12, offering enhanced performance and access to the latest Python features for a more robust development experience.
by csharplus in https://github.com/mage-ai/mage-ai/pull/5393

<div align="center">
<img src="https://framerusercontent.com/images/cz0VDvmIp3LUgzmNLZRGiEEw1o.webp" width="600"/>
</div>

🪝 Support to multiples webhook endpoints for Microsoft Teams notification service
This enhancement allows users to send notifications to multiple Teams channels simultaneously, providing greater flexibility in notification management.
by messerzen in https://github.com/mage-ai/mage-ai/pull/5508

🔴 Updated Redshift connector to enable merge load and correct row count
We’ve added a new configuration option, use_merge_load, to the Redshift destination connector. By default, this setting is disabled. When enabled, it first ingests data into a staging table and then merges it into the target table using Redshift’s Merge process. This feature enhances data loading flexibility and ensures accurate data integration.

<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExd3B3MG1jMDFub3ZvNDMxcWVjb2hrcTR0c3ptbnd0Njh1cTZqZHZhaCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/o5BzNDDFQnepi/giphy.gif" width="50%"/>
</div>
by csharplus in https://github.com/mage-ai/mage-ai/pull/5522

🔍 ElasticSearch Enhancements
* Updated Elasticsearch + verify_certs connection option by syepes in https://github.com/mage-ai/mage-ai/pull/5462
* Added support of _op_type for bulk operations of destinations by syepes in https://github.com/mage-ai/mage-ai/pull/5482
* Elasticsearch support of _op_type for bulk operations by syepes in https://github.com/mage-ai/mage-ai/pull/5471
* Elasticsearch - Align the doc publishing with the Standard (batch) by syepes in https://github.com/mage-ai/mage-ai/pull/5510

<div align="center">
<img src="https://miro.medium.com/v2/resize:fit:1116/format:webp/1*AYP0Mg_MwJMm3Kbx8Xa8lQ.png" width="600"/>
</div>

🐘 Added Postgres client package
We've added a new PostgreSQL client to Mage. Now you can connect to your PostgreSQL databases more easily and reliably, making your data management smoother than ever.
by jx2lee in https://github.com/mage-ai/mage-ai/pull/5486

🐛 Bug Fixes
* [xh]Pass OpenAI API key to OpenAI Library by matrixstone in https://github.com/mage-ai/mage-ai/pull/5430
* [jk] Fix error detail log parsing by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5443
* [jk] Render json objects in block outputs as string instead of nested table by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5450
* [jk] Block output table cell overflow by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5451
* [Bug] Updated SalesForce source to handle multiple date formats by tolson17 in https://github.com/mage-ai/mage-ai/pull/5493
* [jk] Handle encoded page_block_layouts and block_outputs routes by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5544
* [jk] Revert name of test environment by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5478
* [hw] Correct the SSL settings in the nats configuration template by csharplus in https://github.com/mage-ai/mage-ai/pull/5579
* Fix encoding in GCS Integration by TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5447
* bugfix/5562 python data exporter failed insert 2d array to postgres by sugimiyanto in https://github.com/mage-ai/mage-ai/pull/5563
* Fix unable to pass extra parameters to psycopg2 in Postgres connector by kanenorman in https://github.com/mage-ai/mage-ai/pull/5449
* [hw] Avoid throwing exceptions when `block_type` is None by csharplus in https://github.com/mage-ai/mage-ai/pull/5439
* Fix load data in GCS integration by TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5467

💅 Enhancements & Polish
* [xh] Update python version to fix vulnerabilities by matrixstone in https://github.com/mage-ai/mage-ai/pull/5523
* [jk] Display ID in block runs table by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5457
* [Oracle DB] Modify inefficient code that converts dataframe to list of tuples by farmboy-dev in https://github.com/mage-ai/mage-ai/pull/5502
* [jk] Allow user to set number of lines displayed for block output sample preview by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5485
* [hw] Use `self.logger.debug` to replace `print` to make code clean by csharplus in https://github.com/mage-ai/mage-ai/pull/5494
* Updated Facebook Ads SDK to 20.0.2 by jonatansthlmstratlab in https://github.com/mage-ai/mage-ai/pull/5437
* [Enhancement] io/base.py: add Excel support by LucasGrugru in https://github.com/mage-ai/mage-ai/pull/5542
* [hw] Update deltalake to a recent version `0.20.2` by csharplus in https://github.com/mage-ai/mage-ai/pull/5541


✨ Other Changes
* Update documentation for Git in Mage in the Getting Started section by Lennardvb in https://github.com/mage-ai/mage-ai/pull/5459
* [hw] Use the built-in IntEnum instead of creating a new class by csharplus in https://github.com/mage-ai/mage-ai/pull/5433
* [hw] Add time delay to avoid getting the same file timestamp for multiple changes by csharplus in https://github.com/mage-ai/mage-ai/pull/5499
* Update text in documentation about enabling HTTPS in AWS by Lennardvb in https://github.com/mage-ai/mage-ai/pull/5517
* Fix typos in compute-resource.mdx by mvillaizan in https://github.com/mage-ai/mage-ai/pull/5519
* Update alerting-teams.mdx for multiples webhooks. by messerzen in https://github.com/mage-ai/mage-ai/pull/5511
* Update README.md by neubert-analytics in https://github.com/mage-ai/mage-ai/pull/5441
* Typo fix ai-client.mdx by MageKai in https://github.com/mage-ai/mage-ai/pull/5582
* Add information regarding feature dependency to docs by oscarlofwenhamn in https://github.com/mage-ai/mage-ai/pull/5584

👏 New Contributors
* jonatansthlmstratlab made their first contribution in https://github.com/mage-ai/mage-ai/pull/5437
* Lennardvb made their first contribution in https://github.com/mage-ai/mage-ai/pull/5459
* syepes made their first contribution in https://github.com/mage-ai/mage-ai/pull/5471
* jx2lee made their first contribution in https://github.com/mage-ai/mage-ai/pull/5486
* mvillaizan made their first contribution in https://github.com/mage-ai/mage-ai/pull/5519
* farmboy-dev made their first contribution in https://github.com/mage-ai/mage-ai/pull/5502
* tolson17 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5493
* neubert-analytics made their first contribution in https://github.com/mage-ai/mage-ai/pull/5441
* kanenorman made their first contribution in https://github.com/mage-ai/mage-ai/pull/5449
* LucasGrugru made their first contribution in https://github.com/mage-ai/mage-ai/pull/5542
* sugimiyanto made their first contribution in https://github.com/mage-ai/mage-ai/pull/5563
* oscarlofwenhamn made their first contribution in https://github.com/mage-ai/mage-ai/pull/5584

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.74...0.9.75

<!-- Release notes generated using configuration in .github/release.yml at master -->
<div align="center">
<img src="https://github.com/mage-ai/assets/blob/main/pipelines/integration-pipelines/kaos.jpg" width="600"/>
</div>

What's Changed

🎉 Exciting New Features

☁️ Google cloud storage source

You can now effortlessly configure your data integration source blocks to Google Cloud Storage using this snippet. It specifies the destination bucket, sets the file type to Parquet, and provides the path to your service account credentials needed for authentication.

<div align="center">
<img src="https://github.com/mage-ai/assets/blob/main/pipelines/integration-pipelines/gcs_integration_test.png" width="600"/>
</div>

by TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5334

🌬️ 🍽️ Airtable Integration

Effortlessly streamline your data data integration source blocks with Airtable using this tailored script. It establishes a connection to your Airtable base, designates the specific table for integration, and securely integrates your Airtable API key for access.

<div align="center">
<img src="https://github.com/mage-ai/assets/blob/main/pipelines/integration-pipelines/airtable_integration_test.png" width="600"/>
</div>

by TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5404

✂️ Add trim reformat action to transformer block
A new "trim reformat" action has been added to the Python transformer block, allowing for the removal of leading and trailing whitespace from specified text columns. This enhancement ensures cleaner and more consistent data formatting by automatically stripping unnecessary spaces around text entries.


by cristopheridlc in https://github.com/mage-ai/mage-ai/pull/5321

🐻‍❄️ Enable polars dataframe in GCS data exporter

This update enhances the export method of the GCS IO module, expanding its functionality to support exporting Polars DataFrames in addition to the previously supported Pandas DataFrames. This new capability allows users to seamlessly work with both data formats, offering greater flexibility in managing and exporting data from their workflows.

<div align="center">
<img src="https://media0.giphy.com/media/v1.Y2lkPTc5MGI3NjExbDM2amtld2toajMxMmR2cmt2eHc4MmdlZ2V6Y3IxcXdncmI1ZGN5YSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/aK4wh0UE3oddS/giphy.webp" width="50%" />
</div>

by TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5348

Add support to scheduler name on k8s executor

This update adds the ability to customize Kubernetes scheduler options when using the Kubernetes Executor. Users can now directly configure scheduler settings, providing increased flexibility and control over how pods are scheduled within Kubernetes environments.

by messerzen in https://github.com/mage-ai/mage-ai/pull/5412

🐛 Bug Fixes
* [jk] Update condition for cookie secure property by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5327
* [xy] Fix NoneType in trigger settings. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5329
* [xy] Fix dbt block delete request with nginx. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5332
* [xy] Fix API for file versions. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5338
* [xy] Fix schema creation when schema name is wrapped with double quotes. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5366
* [jk] Update test ENV value by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5372
* [xh] Add retry logic and throw if open AI API returns error by matrixstone in https://github.com/mage-ai/mage-ai/pull/5375
* [jk] Fix typo for custom frequency in trigger review by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5392
* [td] Fix Git syncs API error returning sync_on_executor_start by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5388
* 5417 Fix bug: Can't use git with Bitbucket by kennynguyeenx in https://github.com/mage-ai/mage-ai/pull/5418
* Dynamic block fixes
* [td] Fix index used in dynamic block by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5373
* [xy] Fix the condition of checking dynamic upstreams completed https://github.com/mage-ai/mage-ai/commit/2f2eeca44d1aa4555a1165bcc3c7f1bfa414c674
* [xy] Some updates and fixes for dynamic blocks https://github.com/mage-ai/mage-ai/commit/7eba140fba59d455da89b1eef54e723b76128693
* [hw] Fix a missing import issue with data types in a template for Data Exporters by csharplus in https://github.com/mage-ai/mage-ai/pull/5377
* SASL_SSL kafka Host Name Validation by sujiplr in https://github.com/mage-ai/mage-ai/pull/5380

💅 Enhancements & Polish
* [xy] Reduce CPU usage in scheduler. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5331
* [xy] Pass main pod labels to workspaces. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5344
* enhancement: bump duckdb to 1.0.0 by matsonj in https://github.com/mage-ai/mage-ai/pull/5358
* Pass all SSL settings through for SASL_SSL by csharplus in https://github.com/mage-ai/mage-ai/pull/5381
* Update unit tests to use the methods compatible with Python 3.12 by csharplus in https://github.com/mage-ai/mage-ai/pull/5398
* Introduce StrEnum and IntEnum classes to resolve unit test failures with Python 3.11+ by csharplus in https://github.com/mage-ai/mage-ai/pull/5397
* Add NATS credentials file authentication by ddecaro11 in https://github.com/mage-ai/mage-ai/pull/5410
* Update a few unit tests that failed on Python 3.12 by csharplus in https://github.com/mage-ai/mage-ai/pull/5420
* Fix kafka module related unit test failure issues in Python 3.12 by csharplus in https://github.com/mage-ai/mage-ai/pull/5424
* Add trigger name in kwargs https://github.com/mage-ai/mage-ai/commit/81e4f91a238d17ba152fe38246fb45b756c74882
* Expose column name related config to postgres sink config https://github.com/mage-ai/mage-ai/commit/c40af5f228491374a726b5a7b69b734b2730c81a
* [xh] Add ENV variables to turn on/off AI API by matrixstone in https://github.com/mage-ai/mage-ai/pull/5394

Other Changes
* Update amazon-s3.mdx by Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5337
* Fix spelling in compute-resource.mdx by samacciu23 in https://github.com/mage-ai/mage-ai/pull/5356
* Update logger.py by vffv2000 in https://github.com/mage-ai/mage-ai/pull/5383
* [hw] Update the user guide for Kafka streaming with latest changes based on testing by csharplus in https://github.com/mage-ai/mage-ai/pull/5378
* Update the development document to resolve one issue encountered by csharplus in https://github.com/mage-ai/mage-ai/pull/5396
* Fixing spelling error by benjaminhawn in https://github.com/mage-ai/mage-ai/pull/5407

New Contributors
* cristopheridlc made their first contribution in https://github.com/mage-ai/mage-ai/pull/5321
* TalaatHasanin made their first contribution in https://github.com/mage-ai/mage-ai/pull/5348
* samacciu23 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5356
* matsonj made their first contribution in https://github.com/mage-ai/mage-ai/pull/5358
* vffv2000 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5383
* benjaminhawn made their first contribution in https://github.com/mage-ai/mage-ai/pull/5407
* ddecaro11 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5410
* Jaykold made their first contribution in https://github.com/mage-ai/mage-ai/pull/5340
* kennynguyeenx made their first contribution in https://github.com/mage-ai/mage-ai/pull/5418
* messerzen made their first contribution in https://github.com/mage-ai/mage-ai/pull/5412

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.73...0.9.74

<!-- Release notes generated using configuration in .github/release.yml at 1de0a458821ae68deb4a81abfd13fd2defb1c1a2 -->

<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExdHBkd2c0eTlpdW9sbjFuMHFoZGl4ODUwYzdlY2phdnR0cTh4M2EzNCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/gdH0hXjRvh4VPbeM3a/giphy-downsized.gif" width="50%" />
</div>

What's Changed
🎉 Exciting New Features

🧠 Memory management upgrade

This feature enhances system performance and stability by optimizing allocation, deallocation, and overall utilization of RAM resources, resulting in improved efficiency and reduced memory-related errors.

Check out the [doc](https://www.notion.so/mageai/Memory-management-upgrades-e6c5e6e360ce410094091e00c46e3bb6?pvs=4) for details.

<div align="center">
<img src="https://file.notion.so/f/f/6a2090ba-8313-4c74-a4f8-33760f527b3d/58819022-ee38-44b6-9083-9c1aed86322c/Untitled.png?table=block&id=f74131b2-568d-483f-a4c4-29bd6672dd05&spaceId=6a2090ba-8313-4c74-a4f8-33760f527b3d&expirationTimestamp=1722700800000&signature=6SVCr26i7pi6d-VpmuXWhc_m9J-qMZDugj_BUe6gXIs&downloadName=Untitled.png" width="600"/>
</div>

<br/>

<!-- Release notes generated using configuration in .github/release.yml at master -->
<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExcTY2anNtcXB1YTQyY3BneGwyb2gyc3d2YnVuMmp2Y2V2MGJlODA3cyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/sWUhsr69ACcC2c0EYk/giphy-downsized.gif" width="50%" />
</div>

What's Changed

🎉 Exciting New Features

🎨 Canvas v2
Navigate large or complex layouts with a new canvas that allows developers to zoom, pan, drag, and drop visual features.

<div align="center">
<img
src=https://media4.giphy.com/media/v1.Y2lkPTc5MGI3NjExbWkxODRxdXFmNnNhdmIwcHFvYmUzMTZpMXhsdTE0aGtzOXozcW9kYyZlcD12MV9naWZzX3NlYXJjaCZjdD1n/Iaus5qUA99R4GvzxQ1/giphy.webp
/>
</div>

by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5193

🐍 Support custom python sink in Streaming pipeline

Support for custom Python sinks in streaming pipelines allows users to write their own streaming sink implementations, providing greater flexibility in how processed data is handled at the end of the pipeline.


by lgarcia11 in https://github.com/mage-ai/mage-ai/pull/5174

Allow configuring custom owner credentials

This feature allows users to configure custom owner credentials for their application by setting specific environment variables:

DEFAULT_OWNER_EMAIL
DEFAULT_OWNER_PASSWORD
DEFAULT_OWNER_USERNAME


by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5165


🐛 Bug Fixes
* [xy] Fix delete/update workspace. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5146
* [td] Fix Dockerfile to install faster fifo from source by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5152
* [xy] Fix import and add user to verify_project. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5151
* [xy] Fix auto_clean_name param. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5161
* [xy] Bump the version of dbt-sqlserver to 1.7.4 by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5162
* [xy] Fix api trigger token comparison. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5167
* [td] Update Snowflake timeout to network_timeout by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5173
* [xy] Fix executor type when not specified. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5175
* [td] Disable playright, never finishes no windows server 3.10 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5190
* [xy] Fix global data product in single project. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5205
* [td] Revert Next _app back before breaking change by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5207
* [td] Put back Monaco Editor files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5208
💅 Enhancements & Polish
* [xy] Update pypi github action by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5145
* [td] Magic Kernel using server-sent events to execute code from client by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5131
* [td] Standalone Monaco editor by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5160
* [td] Integrate LSP by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5163
* [xy] Allow configuring AZURE_HOST for authentication. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5166
* [xy] Upgrade polars package version by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5176
* [td] Grid layout v2 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5168
* [td] System browser by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5177
* [xy] Speed up Pipeline get and list. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5187
* [xy] Speed up autocomplete endpoint with cache. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5188
* [td] Browser items API by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5189
* [td] Fix styles and integrate real files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5184
* [td] Fix app manager and app loading by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5191
* [td] IDE CRUD operations by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5192
* [xy] Custom kubernetes job prefix by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5195
* [xy] Try using xlarge in circleci. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5206
✨ Other Changes
* Added repo_path to block kwargs and os.environ, improves multi-project feature by hugabora in https://github.com/mage-ai/mage-ai/pull/5144
* [td] DLS v2 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5147
* Updated Variable Name for Cohesion by Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5148
* Remove leading whitespace from Telegram notification message (Issue 4877) by Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5153
* docs: fix markdown formatting by ValentinOtt in https://github.com/mage-ai/mage-ai/pull/5059
* [tc] Update community-examples.mdx by thomaschung408 in https://github.com/mage-ai/mage-ai/pull/5172
* [td] System browser context menu by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5178
* [cf] Updates to schedules doc by colefreeman in https://github.com/mage-ai/mage-ai/pull/5212
* Revert "[xy] Upgrade polars package version" by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5213

🆕 New Contributors
* Ayumi08 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5148
* ValentinOtt made their first contribution in https://github.com/mage-ai/mage-ai/pull/5059
* colefreeman made their first contribution in https://github.com/mage-ai/mage-ai/pull/5171
* lgarcia11 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5174

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.71...0.9.72