Mage-ai

Mage launched a new backfill framework to make backfills a lot easier. User can select a date range and date interval for backfill. Mage will automatically create the pipeline runs within the date range, and run them concurrently to backfill the data.

Docs

- Backfill framework overview: [https://docs.mage.ai/orchestration/backfills/overview](https://docs.mage.ai/orchestration/backfills/overview)
- Backfill guide: [https://docs.mage.ai/orchestration/backfills/guides](https://docs.mage.ai/orchestration/backfills/guides)



Data integration

New sources

- [Microsoft SQL Server](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/mssql/README.md)
- [Outreach](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/outreach/README.md)
- [LinkedIn Ads](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/linkedin_ads/README.md)
- [Paystack](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/paystack/README.md)
- [Postmark](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/postmark/README.md)

New destinations

- [Trino](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/trino/README.md) (all connectors)

Streaming pipeline

Add Kinesis as streaming source and destination(sink) to streaming pipeline.

- For Kinesis streaming source, configure the source stream name and batch size.
- For Kinesis streaming destination(sink), configure the destination stream name and partition key.
- To use Kinesis streaming source and destination, make sure the following environment variables exist:
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_REGION`


Kubernetes support

- Support running Mage in Kubernetes locally: [https://docs.mage.ai/getting-started/setup#using-kubernetes](https://docs.mage.ai/getting-started/setup#using-kubernetes)
- Support executing blocks in separate Kubernetes jobs: [https://docs.mage.ai/production/configuring-production-settings/compute-resource#kubernetes-executor](https://docs.mage.ai/production/configuring-production-settings/compute-resource#kubernetes-executor)

yaml
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
executor_type: k8s
...


- When managing dev environment in Kubernetes cluster, allow adding custom config for the mage container in Kubernetes.


DBT improvements

- Support running MySQL DBT models in Mage.
- When adding a DBT block to run all/multiple models, allow manual naming of the block.

Metaplane integration

Mage can run monitors in Metaplane via API integration. Check out the [guide](https://docs.mage.ai/integrations/metaplane) to learn about how to run monitors in Metaplane and poll statuses of the monitors.

Other bug fixes & polish

- SQL block: support SSH tunnel connection in Postgres SQL block
- Follow this [guide](https://docs.mage.ai/integrations/databases/PostgreSQL#ssh-tunneling) to configure Postgres SQL block to use SSH tunnel
- R block: support accessing runtime variables in R block
- Follow this [guide](https://docs.mage.ai/guides/r-blocks#runtime-variables) to use runtime variables in R blocks
- Added setting to skip current pipeline run if previous pipeline run hasn’t finished.


- Pass runtime variables to test functions. You can access runtime variables via `kwargs['key']` in test functions.

View full [Changelog](https://www.notion.so/mageai/What-s-new-7cc355e38e9c42839d23fdbef2dabd2c)

<!-- Release notes generated using configuration in .github/release.yml at 1de0a458821ae68deb4a81abfd13fd2defb1c1a2 -->

<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExdHBkd2c0eTlpdW9sbjFuMHFoZGl4ODUwYzdlY2phdnR0cTh4M2EzNCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/gdH0hXjRvh4VPbeM3a/giphy-downsized.gif" width="50%" />
</div>

What's Changed
🎉 Exciting New Features

🧠 Memory management upgrade

This feature enhances system performance and stability by optimizing allocation, deallocation, and overall utilization of RAM resources, resulting in improved efficiency and reduced memory-related errors.

Check out the [doc](https://www.notion.so/mageai/Memory-management-upgrades-e6c5e6e360ce410094091e00c46e3bb6?pvs=4) for details.

<div align="center">
<img src="https://file.notion.so/f/f/6a2090ba-8313-4c74-a4f8-33760f527b3d/58819022-ee38-44b6-9083-9c1aed86322c/Untitled.png?table=block&id=f74131b2-568d-483f-a4c4-29bd6672dd05&spaceId=6a2090ba-8313-4c74-a4f8-33760f527b3d&expirationTimestamp=1722700800000&signature=6SVCr26i7pi6d-VpmuXWhc_m9J-qMZDugj_BUe6gXIs&downloadName=Untitled.png" width="600"/>
</div>

<br/>

<!-- Release notes generated using configuration in .github/release.yml at master -->
<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExcTY2anNtcXB1YTQyY3BneGwyb2gyc3d2YnVuMmp2Y2V2MGJlODA3cyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/sWUhsr69ACcC2c0EYk/giphy-downsized.gif" width="50%" />
</div>

What's Changed

🎉 Exciting New Features

🎨 Canvas v2
Navigate large or complex layouts with a new canvas that allows developers to zoom, pan, drag, and drop visual features.

<div align="center">
<img
src=https://media4.giphy.com/media/v1.Y2lkPTc5MGI3NjExbWkxODRxdXFmNnNhdmIwcHFvYmUzMTZpMXhsdTE0aGtzOXozcW9kYyZlcD12MV9naWZzX3NlYXJjaCZjdD1n/Iaus5qUA99R4GvzxQ1/giphy.webp
/>
</div>

by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5193

🐍 Support custom python sink in Streaming pipeline

Support for custom Python sinks in streaming pipelines allows users to write their own streaming sink implementations, providing greater flexibility in how processed data is handled at the end of the pipeline.


by lgarcia11 in https://github.com/mage-ai/mage-ai/pull/5174

Allow configuring custom owner credentials

This feature allows users to configure custom owner credentials for their application by setting specific environment variables:

DEFAULT_OWNER_EMAIL
DEFAULT_OWNER_PASSWORD
DEFAULT_OWNER_USERNAME


by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5165


🐛 Bug Fixes
* [xy] Fix delete/update workspace. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5146
* [td] Fix Dockerfile to install faster fifo from source by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5152
* [xy] Fix import and add user to verify_project. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5151
* [xy] Fix auto_clean_name param. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5161
* [xy] Bump the version of dbt-sqlserver to 1.7.4 by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5162
* [xy] Fix api trigger token comparison. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5167
* [td] Update Snowflake timeout to network_timeout by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5173
* [xy] Fix executor type when not specified. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5175
* [td] Disable playright, never finishes no windows server 3.10 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5190
* [xy] Fix global data product in single project. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5205
* [td] Revert Next _app back before breaking change by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5207
* [td] Put back Monaco Editor files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5208
💅 Enhancements & Polish
* [xy] Update pypi github action by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5145
* [td] Magic Kernel using server-sent events to execute code from client by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5131
* [td] Standalone Monaco editor by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5160
* [td] Integrate LSP by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5163
* [xy] Allow configuring AZURE_HOST for authentication. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5166
* [xy] Upgrade polars package version by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5176
* [td] Grid layout v2 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5168
* [td] System browser by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5177
* [xy] Speed up Pipeline get and list. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5187
* [xy] Speed up autocomplete endpoint with cache. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5188
* [td] Browser items API by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5189
* [td] Fix styles and integrate real files by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5184
* [td] Fix app manager and app loading by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5191
* [td] IDE CRUD operations by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5192
* [xy] Custom kubernetes job prefix by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5195
* [xy] Try using xlarge in circleci. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5206
✨ Other Changes
* Added repo_path to block kwargs and os.environ, improves multi-project feature by hugabora in https://github.com/mage-ai/mage-ai/pull/5144
* [td] DLS v2 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5147
* Updated Variable Name for Cohesion by Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5148
* Remove leading whitespace from Telegram notification message (Issue 4877) by Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5153
* docs: fix markdown formatting by ValentinOtt in https://github.com/mage-ai/mage-ai/pull/5059
* [tc] Update community-examples.mdx by thomaschung408 in https://github.com/mage-ai/mage-ai/pull/5172
* [td] System browser context menu by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5178
* [cf] Updates to schedules doc by colefreeman in https://github.com/mage-ai/mage-ai/pull/5212
* Revert "[xy] Upgrade polars package version" by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5213

🆕 New Contributors
* Ayumi08 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5148
* ValentinOtt made their first contribution in https://github.com/mage-ai/mage-ai/pull/5059
* colefreeman made their first contribution in https://github.com/mage-ai/mage-ai/pull/5171
* lgarcia11 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5174

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.71...0.9.72

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.71 -->
<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExMTBkMmZhdjVia21wN3AwMHkyOHBtdHh5aWZxNHJrN2ExbHppZWUwMyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/kXLxEHU3fajcIuZiCW/giphy.gif" width="50%" />
</div>

What's Changed

🎉 Exciting New Features

📡 Add Streaming Oracle Destination

This feature adds a new custom exporter option for OracleDB allowing developers to easily export their streaming data to Oracle DB.


by matrixstone in https://github.com/mage-ai/mage-ai/pull/4896

👤 Multi Project Platform Support

This feature adds support for each user to have their own active project within a multi project platform that keeps track of every project for every user.

by dy46 in https://github.com/mage-ai/mage-ai/pull/4865

⚙️ Support Job Parameters in K8s Executor
Adds support to help manage environment variables, and control job execution and cleanup policies within the Kubernetes cluster.


k8s_executor_config:
container_config:
image: mageai/mageai:0.9.7
env:
- name: USER_CODE_PATH
value: /home/src/k8s_project
job:
active_deadline_seconds: 120
backoff_limit: 3
ttl_seconds_after_finished: 86400

by artche in https://github.com/mage-ai/mage-ai/pull/5044

🔑 Support OIDC_ROLES_MAPPING and user_roles

This feature is used to map roles obtained from an OpenID Connect (OIDC) identity provider to roles used within your application allowing for a standardized way to manage user roles across different identity providers.

by mihaivint in https://github.com/mage-ai/mage-ai/pull/5053 and https://github.com/mage-ai/mage-ai/pull/4899


💅 Enhancements & Polish
* Optimize CPU and memory usage
* [xy] Disable autoreload in non-dev environments to reduce CPU usage. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5136
* [xy] Reduce CPU usage in scheduler. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5137
* [td] Memory upgrades by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5092
* [td] Track resource usage and run block execution function in a context by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5105
* Improve page performance
* [xy] Some improvements on multi project by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5063
* [xy] Cache project platform settings data in API calls. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5079
* [xy] Speed up pipeline update endpoint. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5125
* Improve dynamic blocks
* [td] Dynamic blocks use the new memory enhancements by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5106
* [td] Dynamic block stream mode by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5121
* [td] Dynamic block resilience by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5113
* [xy] Support configuring logs_dir_path in environment variable. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5018
* [dy] Add prune option for fetch by dy46 in https://github.com/mage-ai/mage-ai/pull/5014
* [xy] Include server and scheduler process id in status response. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5027
* [xy] Add pipeline_run_id to kwargs. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5030
* [xy] Allow configuring authentication mode in mssql config. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5051
* Reload libraries when running blocks from edit pipeline by hugabora in https://github.com/mage-ai/mage-ai/pull/4953
* [xy] Support configuring concurrency config via env vars. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5057
* [jk] Improve visibility of variable/secret names by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5072
* relax pytz version requirements by alex-hunsaker in https://github.com/mage-ai/mage-ai/pull/5006
* upgrade deltalake for tableMerger by oonyoontong in https://github.com/mage-ai/mage-ai/pull/5020
* [td] Turn off unoptimized images by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5082
* Add pipeline_schedule_description variable to notification sender by hjhdaniel in https://github.com/mage-ai/mage-ai/pull/5118

🐛 Bug Fixes
* [xy] Exclude dropped columns from the postgres schema discovery query. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5002
* [xy] Fix None Bigquery database error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5005
* [xy] Fix extracting table name from INSERT statement in SQL block. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5008
* [xy] Catch yaml interpolation error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5012
* [xy] Not run insert query for empty dataframe. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5017
* [dy] Fix git and seed issues by dy46 in https://github.com/mage-ai/mage-ai/pull/5025
* [xy] Fix custom template issue on server start. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5038
* [xy] Comment out flaky unit tests by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5046
* [jk] Allow files to be selected for pipeline zip import by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5042
* [dy] Update git email field by dy46 in https://github.com/mage-ai/mage-ai/pull/5045
* [dy] Fix circular dependency by dy46 in https://github.com/mage-ai/mage-ai/pull/5050
* [td] Update charts and make a ton of bug fixes by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5075
* [td] Fix IDE outputs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5088
* [td] Fix git issue by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5089
* Td fix git2 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5090
* [td] Upgrade deltalake because of Docker image build error by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5091
* [xy] Fix websocket authentication. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5094
* [td] Fix interpolating dictionaries when adding custom code to ide execution by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5095
* [xy] Fix switching active kernel by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5098
* [xy] Fix retry in streaming pipeline. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5099
* [xy] Fix performance issue of fetching variables. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5108
* [jk] Fix max concurrent runs value for backfill by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5115
* [xy] Fix circular dependency. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5122
* [td] Don’t raise error if can’t load by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5132
* [xy] Fix custom template creation. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5134
* [xy] Fix variables API and BaseEnum by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5135
* [td] Fix after on pipeline dashboard by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5142
* [td] Fix after panel showing up everywhere by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5143
* [td] Fix charts when using Spark by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5028
* [xy] Not show locals in CLI exception. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5009
* [dy] Remove error pop up by dy46 in https://github.com/mage-ai/mage-ai/pull/5015
* [td] Fix interactions, triggers, dynamic blocks, charts, and more by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5064
* [td] Fix infer variables by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5102
* Escaping Backslashes when interpolating code content by jimilp7 in https://github.com/mage-ai/mage-ai/pull/5112


✨ New Contributors
* alex-hunsaker made their first contribution in https://github.com/mage-ai/mage-ai/pull/5006
* mihaivint made their first contribution in https://github.com/mage-ai/mage-ai/pull/4899
* stefaan1o made their first contribution in https://github.com/mage-ai/mage-ai/pull/5034
* richardlhughes made their first contribution in https://github.com/mage-ai/mage-ai/pull/5033
* artche made their first contribution in https://github.com/mage-ai/mage-ai/pull/5044
* CLHdevOps made their first contribution in https://github.com/mage-ai/mage-ai/pull/5077
* jimilp7 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5112

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.70...0.9.71

<div align="center">
<img src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExdzFzYjFtbnRid2tjcWNrajV2ODI2bjNqdWYwa3RscWxub3N3YWl2aSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/sHuBShCwtm2J24VTHM/giphy.gif" width="50%" />
</div>

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.70 -->

What's Changed
🎉 Exciting New Features
💥 Add "Last run failed" filter to Pipeline Runs table
This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.

by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4926

🛠️ Add workspace update features
This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/c6c7bc4b-f6a0-49cd-baf6-60ff26ac3925" width="600"/>
</div>

<br/>

by dy46 in https://github.com/mage-ai/mage-ai/pull/4951

🔄 Add upsert to MySQL, BQ, and Snowflake
Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:

unique_conflict_method='UPDATE',
unique_constraints=['col'],


by dy46 in https://github.com/mage-ai/mage-ai/pull/4769

⚠️ Support alerts and limited retries in streaming pipeline
Streaming pipeline used to be retried automatically and infinitely.
With this feature, when `retry_config` is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config
yaml
retry_config:
Number of retry times
retries: 3
Initial delay (in seconds) before retry. If exponential_backoff is true,
the delay time is multiplied by 2 for the next retry
delay: 5
Maximum time between the first attempt and the last retry
max_delay: 60
Whether to use exponential backoff retry
exponential_backoff: true

by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4902

🔑 Add setting to update user roles on login
This feature adds the `UPDATE_ROLES_ON_LOGIN` setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.

by dy46 in https://github.com/mage-ai/mage-ai/pull/4905


🐛 Bug Fixes
* [dy] Catch JSON decode errors in mage targets by dy46 in https://github.com/mage-ai/mage-ai/pull/4838
* [dy] Fix permission seed by dy46 in https://github.com/mage-ai/mage-ai/pull/4846
* [jk] Replace Block reference with BlockFactory by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4850
* [jk] Fix block search add block error by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4855
* [xy] Fix spark dataframe read error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4856
* [jk] Fix filters from being reset when items fetched by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4859
* [xy] Support cancelling jobs when using multiple replicas. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4861
* [xy] Fix updating pipeline content with unicode. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4864
* [xy] Fix dangling process when reloading webserver. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4866
* [jk] Multi-project platform project settings updates by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4878
* [xy] Refresh block run statuses before fetching crashed block runs. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4885
* Fix: add template for oracle export by matrixstone in https://github.com/mage-ai/mage-ai/pull/4892
* [xy] Catch pipeline error in scheduler. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4911
* [dy] Create metadata.yaml file if it doesn't exist by dy46 in https://github.com/mage-ai/mage-ai/pull/4884
* [xy] Fix dynamic block with great expectation block. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4913
* [dy] Fix duckdb sql by dy46 in https://github.com/mage-ai/mage-ai/pull/4915
* [xy] Fix log based sync with uppercase columns. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4922
* [jk] Prevent renamed triggers in code from being duplicated by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4929
* [td] Lots of fixes and enhancements by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4937
* [xy] Skip checking memory in Windows. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4944
* [xy] Check type key in anyOf array objects. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4948
* [dy] Avoid creating duplicate admin user by dy46 in https://github.com/mage-ai/mage-ai/pull/4949
* [xy] Fix downloading xlsx file. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4957
* [jk] Bugfix - Dependency tree layout by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4961
* [xy] Fix logging tags for streaming pipeline run. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4963
* [xy] Catch notification send exceptions. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4964
* [dy] Remove git dependency by dy46 in https://github.com/mage-ai/mage-ai/pull/4959
* [jk] Cancel unsaved modified file dialogue by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4975
* [xy] Always set logger in different blocks separately. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4974
* [dy] Fix more git issues by dy46 in https://github.com/mage-ai/mage-ai/pull/4980
* [xy] Fix k8s affinity parsing. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4985
* [xy] Fix encoding for block action object. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4987
* [xy] Fix None job manager. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4995
* [dy] Remove unused submodule by dy46 in https://github.com/mage-ai/mage-ai/pull/4984
* [dy] Fix some git bugs by dy46 in https://github.com/mage-ai/mage-ai/pull/4973
* Update jupyter-server-proxy to fix CVE-2024-28179 by lqc in https://github.com/mage-ai/mage-ai/pull/4967
* Trino iceberg and delta lake array issues fix along with issue 4895 and other improvements by sumanshusamarora in https://github.com/mage-ai/mage-ai/pull/4952
* [edw] Fix typo in Kafka streaming source by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4918
* [dy] Cast ambiguous columns to strings by dy46 in https://github.com/mage-ai/mage-ai/pull/4857
* Bugfix/okta authorization url discovery by moseswynn in https://github.com/mage-ai/mage-ai/pull/4810
* Update ads_insights_age_and_gender.json by sumanshusamarora in https://github.com/mage-ai/mage-ai/pull/4891
* [dy] Use project uuid for workspace default permissions by dy46 in https://github.com/mage-ai/mage-ai/pull/4843
* [dy] Update oauth token column type by dy46 in https://github.com/mage-ai/mage-ai/pull/4933

💅 Enhancements & Polish
* [xy] Use executor type from pipeline level when block level executor is not set by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4968
* [jk] Replace blue font with cyan in terminal by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4848
* [xy] Kill all jobs on scheduler crash. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4851
* [td] Update GX and other libraries to support higher versions of Pydantic by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4860
* [xy] Update snowflake destination datetime to use TIMESTAMP type. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4875
* [jk] Block output view option by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4883
* [jk] Only make specific files request with ArcaneLibrary by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4888
* Caching pipeline_project outside of loop to avoid multple lookups by hugabora in https://github.com/mage-ai/mage-ai/pull/4887
* [jk] Remove pipeline_runs request appearing on different pages by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4890
* [xy] Add measurement to influxdb source. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4893
* [jk] Add visibility into project config loading errors by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4894
* [jk] Hide header actions on sign-in page by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4904
* [jk] Disallow numbers as first character of block names by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4914
* [jk] Triggers table UI fixes by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4934
* [xy] Support interpolating trigger name in k8s job name. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4947
* [xy] Only send failure notification when pipeline run status is failed. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4950
* [jk] Backfills table improvements by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4942
* [xy] Improve the performance of Postgres source metadata query. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4962
* [jk] Allow disabling pipeline autosave by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4988
* [xy] Support query pipelines with repo_path. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4993
* [Enhancement] Optionally Disable Wait Timeout in ECS Config by moseswynn in https://github.com/mage-ai/mage-ai/pull/4970
* Modifying parser to handle pandas._libs.missing.NAType by ashutosh-npsx in https://github.com/mage-ai/mage-ai/pull/4729
* [jk] Usage stats modal by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4863
* [edw] e2e: ensure example pipeline runs smoothly by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4771
* [jk] Autocomplete search results logging by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4858


✨ New Contributors
* Wesseldr made their first contribution in https://github.com/mage-ai/mage-ai/pull/4927
* ashutosh-npsx made their first contribution in https://github.com/mage-ai/mage-ai/pull/4729
* lqc made their first contribution in https://github.com/mage-ai/mage-ai/pull/4967

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.68...0.9.70

<div align="center">
<img src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExazJsM3BoNnN4NTNtcHplcm5peXNmeDdwNm9xeW82ajV4NjAweGdxZSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/51DjsXLhUfFozdos7F/giphy-downsized.gif" width="300" />
</div>

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.68 -->

What's Changed
🎉 Exciting New Features
🐍 Custom python source in Streaming pipeline
This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:
python
from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable

if 'streaming_source' not in globals():
from mage_ai.data_preparation.decorators import streaming_source


streaming_source
class CustomSource(BasePythonSource):
def init_client(self):
"""
Implement the logic of initializing the client.
"""

def batch_read(self, handler: Callable):
"""
Batch read the messages from the source and use handler to process the messages.
"""
while True:
records = []
Implement the logic of fetching the records
if len(records) > 0:
handler(records)

by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4763

📦 OracleDB exporter
Add OracleDB exporter template to standard batch pipeline.

by matrixstone in https://github.com/mage-ai/mage-ai/pull/4785

🔧 Server logging format template overriding
This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems.
Here is an example setting:
bash
export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'

by afal007 in https://github.com/mage-ai/mage-ai/pull/4772

🔄 Update pipeline name before pipeline creation
Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.

by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4790


🐛 Bug Fixes
* [jk] Add check to avoid rendering of invalid JSX element by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4755
* [dy] Fix GitLab auth issues by dy46 in https://github.com/mage-ai/mage-ai/pull/4760
* [jk] Increase visibility of code block context menu by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4768
* [dy] Check status response for REQUIRE_USER_AUTHENTICATION by dy46 in https://github.com/mage-ai/mage-ai/pull/4782
* [xy] Fix exporting dataframe to bigquery. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4783
* [xy] Catch job manager init error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4784
* [dy] Add all parent block variables to the add on block by dy46 in https://github.com/mage-ai/mage-ai/pull/4747
* [dy] Fix clickhouse destination by dy46 in https://github.com/mage-ai/mage-ai/pull/4788
* [jk] Bugfixes - table render / rename destination block by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4797
* [dy] Update API trigger url by dy46 in https://github.com/mage-ai/mage-ai/pull/4800
* [dy] Load pipeline tags from config by dy46 in https://github.com/mage-ai/mage-ai/pull/4806
* [xy] Fix insert command for trino deltalake array type. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4819
* [jk] Fix unclickable dbt sql file in dbt v1 model select by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4822
* [xy] Raise ImportError when spark is not supported. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4825
* [xy] Fix mongodb source typo. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4833
* [jk] Configure dbt v2 yaml block and other dbt code block improvements by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4834
* [xy] Fix writing polars dataframe to gcs storage. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4839
* [xy] Create snowflake temp table like original table. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4802
* [jk] Bugfix pipeline tag cache by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4841
* [xh] Oracle fix issue: incremental datetime bookmark fetch fails by matrixstone in https://github.com/mage-ai/mage-ai/pull/4816
* [jk] Clear block_uuid query param when selecting new block by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4844
* [dy] Add sign in check to index.tsx by dy46 in https://github.com/mage-ai/mage-ai/pull/4781
* [dy] Fix SSO sign in for frontend build by dy46 in https://github.com/mage-ai/mage-ai/pull/4794
* [dy] Put cache in try/except by dy46 in https://github.com/mage-ai/mage-ai/pull/4796
* [dy] Fix slack notifications with stacktrace by dy46 in https://github.com/mage-ai/mage-ai/pull/4826
* [dy] Update config_overwrite by dy46 in https://github.com/mage-ai/mage-ai/pull/4811

💅 Enhancements & Polish
* [xh] OracleDB thick mode support in standard pipeline by matrixstone in https://github.com/mage-ai/mage-ai/pull/4701
* [jk] Limit number of block output files read and included in pipeline response by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4758
* [xy] Only show sql logs in debug environment. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4789
* [xy] Interpolate global vars in k8s executor namespace by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4792
* [enhancement] mongodb stream full_document config option by moseswynn in https://github.com/mage-ai/mage-ai/pull/4793
* [xy] Improve the performance of loading dbt blocks. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4807
* [jk] Avoid N dbt config option requests for N blocks on Pipeline Editor by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4812
* [xy] Speed up backend requests. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4815
* [xy] Test kafka connection when initializing the consumer. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4832
* [xy] Make path_to_credential optional in GCS logger. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4837
* [dy] Expire certain pipeline runs by dy46 in https://github.com/mage-ai/mage-ai/pull/4727
* [dy] Add pagination to workspace roles by dy46 in https://github.com/mage-ai/mage-ai/pull/4799
* [dy] Improve elasticsearch speed by dy46 in https://github.com/mage-ai/mage-ai/pull/4824


✨ New Contributors
* afal007 made their first contribution in https://github.com/mage-ai/mage-ai/pull/4772
* moseswynn made their first contribution in https://github.com/mage-ai/mage-ai/pull/4793

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.66...0.9.68