Mage-ai

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.71 -->
<div align="center">
<img src="https://i.giphy.com/media/v1.Y2lkPTc5MGI3NjExMTBkMmZhdjVia21wN3AwMHkyOHBtdHh5aWZxNHJrN2ExbHppZWUwMyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/kXLxEHU3fajcIuZiCW/giphy.gif" width="50%" />
</div>

What's Changed

🎉 Exciting New Features

📡 Add Streaming Oracle Destination

This feature adds a new custom exporter option for OracleDB allowing developers to easily export their streaming data to Oracle DB.


by matrixstone in https://github.com/mage-ai/mage-ai/pull/4896

👤 Multi Project Platform Support

This feature adds support for each user to have their own active project within a multi project platform that keeps track of every project for every user.

by dy46 in https://github.com/mage-ai/mage-ai/pull/4865

⚙️ Support Job Parameters in K8s Executor
Adds support to help manage environment variables, and control job execution and cleanup policies within the Kubernetes cluster.


k8s_executor_config:
container_config:
image: mageai/mageai:0.9.7
env:
- name: USER_CODE_PATH
value: /home/src/k8s_project
job:
active_deadline_seconds: 120
backoff_limit: 3
ttl_seconds_after_finished: 86400

by artche in https://github.com/mage-ai/mage-ai/pull/5044

🔑 Support OIDC_ROLES_MAPPING and user_roles

This feature is used to map roles obtained from an OpenID Connect (OIDC) identity provider to roles used within your application allowing for a standardized way to manage user roles across different identity providers.

by mihaivint in https://github.com/mage-ai/mage-ai/pull/5053 and https://github.com/mage-ai/mage-ai/pull/4899


💅 Enhancements & Polish
* Optimize CPU and memory usage
* [xy] Disable autoreload in non-dev environments to reduce CPU usage. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5136
* [xy] Reduce CPU usage in scheduler. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5137
* [td] Memory upgrades by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5092
* [td] Track resource usage and run block execution function in a context by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5105
* Improve page performance
* [xy] Some improvements on multi project by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5063
* [xy] Cache project platform settings data in API calls. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5079
* [xy] Speed up pipeline update endpoint. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5125
* Improve dynamic blocks
* [td] Dynamic blocks use the new memory enhancements by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5106
* [td] Dynamic block stream mode by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5121
* [td] Dynamic block resilience by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5113
* [xy] Support configuring logs_dir_path in environment variable. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5018
* [dy] Add prune option for fetch by dy46 in https://github.com/mage-ai/mage-ai/pull/5014
* [xy] Include server and scheduler process id in status response. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5027
* [xy] Add pipeline_run_id to kwargs. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5030
* [xy] Allow configuring authentication mode in mssql config. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5051
* Reload libraries when running blocks from edit pipeline by hugabora in https://github.com/mage-ai/mage-ai/pull/4953
* [xy] Support configuring concurrency config via env vars. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5057
* [jk] Improve visibility of variable/secret names by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5072
* relax pytz version requirements by alex-hunsaker in https://github.com/mage-ai/mage-ai/pull/5006
* upgrade deltalake for tableMerger by oonyoontong in https://github.com/mage-ai/mage-ai/pull/5020
* [td] Turn off unoptimized images by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5082
* Add pipeline_schedule_description variable to notification sender by hjhdaniel in https://github.com/mage-ai/mage-ai/pull/5118

🐛 Bug Fixes
* [xy] Exclude dropped columns from the postgres schema discovery query. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5002
* [xy] Fix None Bigquery database error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5005
* [xy] Fix extracting table name from INSERT statement in SQL block. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5008
* [xy] Catch yaml interpolation error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5012
* [xy] Not run insert query for empty dataframe. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5017
* [dy] Fix git and seed issues by dy46 in https://github.com/mage-ai/mage-ai/pull/5025
* [xy] Fix custom template issue on server start. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5038
* [xy] Comment out flaky unit tests by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5046
* [jk] Allow files to be selected for pipeline zip import by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5042
* [dy] Update git email field by dy46 in https://github.com/mage-ai/mage-ai/pull/5045
* [dy] Fix circular dependency by dy46 in https://github.com/mage-ai/mage-ai/pull/5050
* [td] Update charts and make a ton of bug fixes by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5075
* [td] Fix IDE outputs by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5088
* [td] Fix git issue by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5089
* Td fix git2 by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5090
* [td] Upgrade deltalake because of Docker image build error by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5091
* [xy] Fix websocket authentication. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5094
* [td] Fix interpolating dictionaries when adding custom code to ide execution by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5095
* [xy] Fix switching active kernel by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5098
* [xy] Fix retry in streaming pipeline. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5099
* [xy] Fix performance issue of fetching variables. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5108
* [jk] Fix max concurrent runs value for backfill by johnson-mage in https://github.com/mage-ai/mage-ai/pull/5115
* [xy] Fix circular dependency. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5122
* [td] Don’t raise error if can’t load by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5132
* [xy] Fix custom template creation. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5134
* [xy] Fix variables API and BaseEnum by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5135
* [td] Fix after on pipeline dashboard by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5142
* [td] Fix after panel showing up everywhere by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5143
* [td] Fix charts when using Spark by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5028
* [xy] Not show locals in CLI exception. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5009
* [dy] Remove error pop up by dy46 in https://github.com/mage-ai/mage-ai/pull/5015
* [td] Fix interactions, triggers, dynamic blocks, charts, and more by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5064
* [td] Fix infer variables by tommydangerous in https://github.com/mage-ai/mage-ai/pull/5102
* Escaping Backslashes when interpolating code content by jimilp7 in https://github.com/mage-ai/mage-ai/pull/5112


✨ New Contributors
* alex-hunsaker made their first contribution in https://github.com/mage-ai/mage-ai/pull/5006
* mihaivint made their first contribution in https://github.com/mage-ai/mage-ai/pull/4899
* stefaan1o made their first contribution in https://github.com/mage-ai/mage-ai/pull/5034
* richardlhughes made their first contribution in https://github.com/mage-ai/mage-ai/pull/5033
* artche made their first contribution in https://github.com/mage-ai/mage-ai/pull/5044
* CLHdevOps made their first contribution in https://github.com/mage-ai/mage-ai/pull/5077
* jimilp7 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5112

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.70...0.9.71

<div align="center">
<img src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExdzFzYjFtbnRid2tjcWNrajV2ODI2bjNqdWYwa3RscWxub3N3YWl2aSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/sHuBShCwtm2J24VTHM/giphy.gif" width="50%" />
</div>

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.70 -->

What's Changed
🎉 Exciting New Features
💥 Add "Last run failed" filter to Pipeline Runs table
This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.

by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4926

🛠️ Add workspace update features
This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/c6c7bc4b-f6a0-49cd-baf6-60ff26ac3925" width="600"/>
</div>

<br/>

by dy46 in https://github.com/mage-ai/mage-ai/pull/4951

🔄 Add upsert to MySQL, BQ, and Snowflake
Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:

unique_conflict_method='UPDATE',
unique_constraints=['col'],


by dy46 in https://github.com/mage-ai/mage-ai/pull/4769

⚠️ Support alerts and limited retries in streaming pipeline
Streaming pipeline used to be retried automatically and infinitely.
With this feature, when `retry_config` is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config
yaml
retry_config:
Number of retry times
retries: 3
Initial delay (in seconds) before retry. If exponential_backoff is true,
the delay time is multiplied by 2 for the next retry
delay: 5
Maximum time between the first attempt and the last retry
max_delay: 60
Whether to use exponential backoff retry
exponential_backoff: true

by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4902

🔑 Add setting to update user roles on login
This feature adds the `UPDATE_ROLES_ON_LOGIN` setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.

by dy46 in https://github.com/mage-ai/mage-ai/pull/4905


🐛 Bug Fixes
* [dy] Catch JSON decode errors in mage targets by dy46 in https://github.com/mage-ai/mage-ai/pull/4838
* [dy] Fix permission seed by dy46 in https://github.com/mage-ai/mage-ai/pull/4846
* [jk] Replace Block reference with BlockFactory by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4850
* [jk] Fix block search add block error by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4855
* [xy] Fix spark dataframe read error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4856
* [jk] Fix filters from being reset when items fetched by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4859
* [xy] Support cancelling jobs when using multiple replicas. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4861
* [xy] Fix updating pipeline content with unicode. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4864
* [xy] Fix dangling process when reloading webserver. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4866
* [jk] Multi-project platform project settings updates by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4878
* [xy] Refresh block run statuses before fetching crashed block runs. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4885
* Fix: add template for oracle export by matrixstone in https://github.com/mage-ai/mage-ai/pull/4892
* [xy] Catch pipeline error in scheduler. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4911
* [dy] Create metadata.yaml file if it doesn't exist by dy46 in https://github.com/mage-ai/mage-ai/pull/4884
* [xy] Fix dynamic block with great expectation block. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4913
* [dy] Fix duckdb sql by dy46 in https://github.com/mage-ai/mage-ai/pull/4915
* [xy] Fix log based sync with uppercase columns. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4922
* [jk] Prevent renamed triggers in code from being duplicated by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4929
* [td] Lots of fixes and enhancements by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4937
* [xy] Skip checking memory in Windows. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4944
* [xy] Check type key in anyOf array objects. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4948
* [dy] Avoid creating duplicate admin user by dy46 in https://github.com/mage-ai/mage-ai/pull/4949
* [xy] Fix downloading xlsx file. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4957
* [jk] Bugfix - Dependency tree layout by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4961
* [xy] Fix logging tags for streaming pipeline run. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4963
* [xy] Catch notification send exceptions. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4964
* [dy] Remove git dependency by dy46 in https://github.com/mage-ai/mage-ai/pull/4959
* [jk] Cancel unsaved modified file dialogue by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4975
* [xy] Always set logger in different blocks separately. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4974
* [dy] Fix more git issues by dy46 in https://github.com/mage-ai/mage-ai/pull/4980
* [xy] Fix k8s affinity parsing. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4985
* [xy] Fix encoding for block action object. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4987
* [xy] Fix None job manager. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4995
* [dy] Remove unused submodule by dy46 in https://github.com/mage-ai/mage-ai/pull/4984
* [dy] Fix some git bugs by dy46 in https://github.com/mage-ai/mage-ai/pull/4973
* Update jupyter-server-proxy to fix CVE-2024-28179 by lqc in https://github.com/mage-ai/mage-ai/pull/4967
* Trino iceberg and delta lake array issues fix along with issue 4895 and other improvements by sumanshusamarora in https://github.com/mage-ai/mage-ai/pull/4952
* [edw] Fix typo in Kafka streaming source by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4918
* [dy] Cast ambiguous columns to strings by dy46 in https://github.com/mage-ai/mage-ai/pull/4857
* Bugfix/okta authorization url discovery by moseswynn in https://github.com/mage-ai/mage-ai/pull/4810
* Update ads_insights_age_and_gender.json by sumanshusamarora in https://github.com/mage-ai/mage-ai/pull/4891
* [dy] Use project uuid for workspace default permissions by dy46 in https://github.com/mage-ai/mage-ai/pull/4843
* [dy] Update oauth token column type by dy46 in https://github.com/mage-ai/mage-ai/pull/4933

💅 Enhancements & Polish
* [xy] Use executor type from pipeline level when block level executor is not set by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4968
* [jk] Replace blue font with cyan in terminal by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4848
* [xy] Kill all jobs on scheduler crash. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4851
* [td] Update GX and other libraries to support higher versions of Pydantic by tommydangerous in https://github.com/mage-ai/mage-ai/pull/4860
* [xy] Update snowflake destination datetime to use TIMESTAMP type. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4875
* [jk] Block output view option by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4883
* [jk] Only make specific files request with ArcaneLibrary by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4888
* Caching pipeline_project outside of loop to avoid multple lookups by hugabora in https://github.com/mage-ai/mage-ai/pull/4887
* [jk] Remove pipeline_runs request appearing on different pages by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4890
* [xy] Add measurement to influxdb source. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4893
* [jk] Add visibility into project config loading errors by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4894
* [jk] Hide header actions on sign-in page by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4904
* [jk] Disallow numbers as first character of block names by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4914
* [jk] Triggers table UI fixes by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4934
* [xy] Support interpolating trigger name in k8s job name. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4947
* [xy] Only send failure notification when pipeline run status is failed. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4950
* [jk] Backfills table improvements by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4942
* [xy] Improve the performance of Postgres source metadata query. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4962
* [jk] Allow disabling pipeline autosave by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4988
* [xy] Support query pipelines with repo_path. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4993
* [Enhancement] Optionally Disable Wait Timeout in ECS Config by moseswynn in https://github.com/mage-ai/mage-ai/pull/4970
* Modifying parser to handle pandas._libs.missing.NAType by ashutosh-npsx in https://github.com/mage-ai/mage-ai/pull/4729
* [jk] Usage stats modal by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4863
* [edw] e2e: ensure example pipeline runs smoothly by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4771
* [jk] Autocomplete search results logging by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4858


✨ New Contributors
* Wesseldr made their first contribution in https://github.com/mage-ai/mage-ai/pull/4927
* ashutosh-npsx made their first contribution in https://github.com/mage-ai/mage-ai/pull/4729
* lqc made their first contribution in https://github.com/mage-ai/mage-ai/pull/4967

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.68...0.9.70

<div align="center">
<img src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExazJsM3BoNnN4NTNtcHplcm5peXNmeDdwNm9xeW82ajV4NjAweGdxZSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/51DjsXLhUfFozdos7F/giphy-downsized.gif" width="300" />
</div>

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.68 -->

What's Changed
🎉 Exciting New Features
🐍 Custom python source in Streaming pipeline
This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:
python
from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable

if 'streaming_source' not in globals():
from mage_ai.data_preparation.decorators import streaming_source


streaming_source
class CustomSource(BasePythonSource):
def init_client(self):
"""
Implement the logic of initializing the client.
"""

def batch_read(self, handler: Callable):
"""
Batch read the messages from the source and use handler to process the messages.
"""
while True:
records = []
Implement the logic of fetching the records
if len(records) > 0:
handler(records)

by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4763

📦 OracleDB exporter
Add OracleDB exporter template to standard batch pipeline.

by matrixstone in https://github.com/mage-ai/mage-ai/pull/4785

🔧 Server logging format template overriding
This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems.
Here is an example setting:
bash
export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'

by afal007 in https://github.com/mage-ai/mage-ai/pull/4772

🔄 Update pipeline name before pipeline creation
Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.

by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4790


🐛 Bug Fixes
* [jk] Add check to avoid rendering of invalid JSX element by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4755
* [dy] Fix GitLab auth issues by dy46 in https://github.com/mage-ai/mage-ai/pull/4760
* [jk] Increase visibility of code block context menu by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4768
* [dy] Check status response for REQUIRE_USER_AUTHENTICATION by dy46 in https://github.com/mage-ai/mage-ai/pull/4782
* [xy] Fix exporting dataframe to bigquery. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4783
* [xy] Catch job manager init error. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4784
* [dy] Add all parent block variables to the add on block by dy46 in https://github.com/mage-ai/mage-ai/pull/4747
* [dy] Fix clickhouse destination by dy46 in https://github.com/mage-ai/mage-ai/pull/4788
* [jk] Bugfixes - table render / rename destination block by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4797
* [dy] Update API trigger url by dy46 in https://github.com/mage-ai/mage-ai/pull/4800
* [dy] Load pipeline tags from config by dy46 in https://github.com/mage-ai/mage-ai/pull/4806
* [xy] Fix insert command for trino deltalake array type. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4819
* [jk] Fix unclickable dbt sql file in dbt v1 model select by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4822
* [xy] Raise ImportError when spark is not supported. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4825
* [xy] Fix mongodb source typo. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4833
* [jk] Configure dbt v2 yaml block and other dbt code block improvements by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4834
* [xy] Fix writing polars dataframe to gcs storage. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4839
* [xy] Create snowflake temp table like original table. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4802
* [jk] Bugfix pipeline tag cache by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4841
* [xh] Oracle fix issue: incremental datetime bookmark fetch fails by matrixstone in https://github.com/mage-ai/mage-ai/pull/4816
* [jk] Clear block_uuid query param when selecting new block by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4844
* [dy] Add sign in check to index.tsx by dy46 in https://github.com/mage-ai/mage-ai/pull/4781
* [dy] Fix SSO sign in for frontend build by dy46 in https://github.com/mage-ai/mage-ai/pull/4794
* [dy] Put cache in try/except by dy46 in https://github.com/mage-ai/mage-ai/pull/4796
* [dy] Fix slack notifications with stacktrace by dy46 in https://github.com/mage-ai/mage-ai/pull/4826
* [dy] Update config_overwrite by dy46 in https://github.com/mage-ai/mage-ai/pull/4811

💅 Enhancements & Polish
* [xh] OracleDB thick mode support in standard pipeline by matrixstone in https://github.com/mage-ai/mage-ai/pull/4701
* [jk] Limit number of block output files read and included in pipeline response by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4758
* [xy] Only show sql logs in debug environment. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4789
* [xy] Interpolate global vars in k8s executor namespace by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4792
* [enhancement] mongodb stream full_document config option by moseswynn in https://github.com/mage-ai/mage-ai/pull/4793
* [xy] Improve the performance of loading dbt blocks. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4807
* [jk] Avoid N dbt config option requests for N blocks on Pipeline Editor by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4812
* [xy] Speed up backend requests. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4815
* [xy] Test kafka connection when initializing the consumer. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4832
* [xy] Make path_to_credential optional in GCS logger. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4837
* [dy] Expire certain pipeline runs by dy46 in https://github.com/mage-ai/mage-ai/pull/4727
* [dy] Add pagination to workspace roles by dy46 in https://github.com/mage-ai/mage-ai/pull/4799
* [dy] Improve elasticsearch speed by dy46 in https://github.com/mage-ai/mage-ai/pull/4824


✨ New Contributors
* afal007 made their first contribution in https://github.com/mage-ai/mage-ai/pull/4772
* moseswynn made their first contribution in https://github.com/mage-ai/mage-ai/pull/4793

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.66...0.9.68

<div align="center">
<img src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExM3NzODRwdGlodHhhNWRqNzNvaTh4aG1qOHhrdjZoODFwZDk2NjU1bSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/5hdJF3HJGhVeiYFzht/giphy.gif" width="50%" />
</div>

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.66 -->

What's Changed
🎉 Exciting New Features
🗑️ Delete backfills
* Add delete button to backfills table that allows user to delete backfills individually.
* Prevent user from setting a backfill interval unit value less than 1, which could cause issues loading the backfills.

by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4726

🪪 Support AzureDevOps using Entra ID/AD authentication
If you have an Azure DevOps organization connected to your Active Directory/Microsoft Entra ID tenant, you can enable DevOps in the version control app.
* First, you will need to enable the [Active Directory](https://docs.mage.ai/production/authentication/microsoft) integration in order for Mage to authenticate your user.
* Once the Active Directory integration is enabled, you simply need to add your DevOps organization as an environment variable, and you should be able to authenticate with DevOps in the version control app
* `export AZURE_DEVOPS_ORGANIZATION=<organization_name>`

by dy46 in https://github.com/mage-ai/mage-ai/pull/4721

📅  Scheduler status endpoint
Add `/api/status` endpoint to scheduler when running scheduler in standalone mode.
This endpoint can be used to check the liveness of the scheduler.
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/b97549df-8e65-497d-a1b6-e9fc7c0b42e2" width="600"/>
</div>

<br/>

by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4736

🔐 Read authentication settings from AWS secrets manager
Add support for reading settings from AWS secrets manager. The existing authentication settings have been moved to using the new Settings class.

In order to set the settings backend, the user will need to add a section to the project metadata. Here is an example to set the setting backend to AWS secrets manager:
yaml
settings_backend:
backend_type: aws_secrets_manager
prefix: default_repo/settings/

Read the [doc](https://docs.mage.ai/production/configuring-production-settings/settings-backend) for details.

by dy46 in https://github.com/mage-ai/mage-ai/pull/4674

💅 Enhancements & Polish
* [jk] Limit block output for dynamic blocks in UI by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4693
* [xy] Add auto clean name flag to io classes. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4698
* [dy] Add interval variables for cron based schedules by dy46 in https://github.com/mage-ai/mage-ai/pull/4699
* [dy] Add error stack trace to the message for pipeline run failure by dy46 in https://github.com/mage-ai/mage-ai/pull/4697
* [dy] Add unique_conflict_method for mssql by dy46 in https://github.com/mage-ai/mage-ai/pull/4712
* [jk] Scrollbar polish by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4732
* [xy] Remove Redshift stl insert query. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4735
* [edw] e2e test: load pages no errors by edmondwinston in https://github.com/mage-ai/mage-ai/pull/4429
* [dy] Share main project logging config with workspaces by dy46 in https://github.com/mage-ai/mage-ai/pull/4656

🐛 Bug Fixes
* [jk] Fix "filter is not a function" type error in schema props by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4686
* [xy] Fix list index out of range for dynamic blocks. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4688
* [xy] Set default domain for salesforce source by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4689
* [dy] Handle JSON reading errors for elasticsearch by dy46 in https://github.com/mage-ai/mage-ai/pull/4696
* [dy] Add try/except by dy46 in https://github.com/mage-ai/mage-ai/pull/4683
* [jk] Update backfill policy to allow update on settings by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4706
* [dy] Fix logging for elasticsearch destination by dy46 in https://github.com/mage-ai/mage-ai/pull/4709
* [xy] Add upstream_dbt_blocks method back to DBTBlockSQL. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4711
* [jk] Bugfix for pipeline interactions by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4710
* [xy] Catch sync schedules exception. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4717
* [xy] Fix pipeline metadata encoding issue. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4718
* [jk] Fix buggy pipeline dashboard by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4714
* Bugfix: Fix migration crash when using MSSQL as mage database by tuokor in https://github.com/mage-ai/mage-ai/pull/4695
* [xy] Set repo_path when syncing trigger configs. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4719
* [jk] Allow file to be opened in Pipeline Editor without Command Center enabled by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4728
* [jk] Bugfix dock overlay by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4733
* [dy] Display error on oauth by dy46 in https://github.com/mage-ai/mage-ai/pull/4734
* [jk] Avoid client error when making request before interactions are done loading by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4744
* Trino temp table location fix - Fixes issue https://github.com/mage-ai/mage-ai/issues/4738 by sumanshusamarora in https://github.com/mage-ai/mage-ai/pull/4739
* [xy] Fix alter Snowflake table with lower_case=false by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4750
* [jk] Fix block run sorting by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4751
* [dy] Fix sign in issue by dy46 in https://github.com/mage-ai/mage-ai/pull/4753
* [dy] Fix elasticsearch destination by dy46 in https://github.com/mage-ai/mage-ai/pull/4687
* fix the incorrect spelling by ckfear in https://github.com/mage-ai/mage-ai/pull/4680
* [Bug] Resolve "parsing error" in the trino.mdx screen by devharipragaz007 in https://github.com/mage-ai/mage-ai/pull/4702
* Pyarrow version conflicts from Snowflake connector by Jawahars in https://github.com/mage-ai/mage-ai/pull/4707
* [dy] Update access token user queries and add tests by dy46 in https://github.com/mage-ai/mage-ai/pull/4713
* bug fix with stripe sync by sumanshusamarora in https://github.com/mage-ai/mage-ai/pull/4748

✨ New Contributors
* devharipragaz007 made their first contribution in https://github.com/mage-ai/mage-ai/pull/4702
* Jawahars made their first contribution in https://github.com/mage-ai/mage-ai/pull/4707
* tuokor made their first contribution in https://github.com/mage-ai/mage-ai/pull/4695

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.65...0.9.66

<div align="center">
<img src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExNGs2enJlbXV1MmJkamlkcTluY3RuZ3g5cmFiMDkyd2E0emRpa3gwZiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/QUKqSLmE7vmZP2PkZk/giphy.gif" width="50%" />
</div>

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.65 -->

What's Changed
🎉 Exciting New Features
🔔 Telegram notification
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/ca7b240f-1e07-4c0a-8c61-0b3809fefc33" width="600"/>
</div>

<br/>

Support for setting up Telegram notification channel for alerts is now available.
Check out the [doc](https://docs.mage.ai/integrations/observability/alerting-telegram) for details

_by luthfir-rahman in https://github.com/mage-ai/mage-ai/pull/4632_

🔄 Runtime variables and concurrency config in Backfills
Backfills support configuring global variables override and concurrency limit is now available.
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/c359d942-3dbe-4cd4-ba07-3ef636f5d9cc" width="600"/>
</div>

<br/>

_by dy46 in https://github.com/mage-ai/mage-ai/pull/4618_

☁️ Export raw files to S3 and GCS
Using S3 and GCS data exporter templates, you can now directly upload raw files in addition to dataframes.

python
bucket_name = 'test-bucket'
object_key = 'test_file_copy.csv'

S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
'/home/src/test_file.csv',
bucket_name,
object_key,
)
GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
'/home/src/test_file.csv',
bucket_name,
object_key,
)

_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4607_

🐛 Bug Fixes
* Luthfir/discord notification by luthfir-rahman in https://github.com/mage-ai/mage-ai/pull/4590
* [xy] Support exporting to mysql with selected columns. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4605
* [xy] Update the port check to use specified host. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4606
* [xy] Set default host when checking port. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4610
* [dy] Refactor duplicate git code by dy46 in https://github.com/mage-ai/mage-ai/pull/4611
* [jk] Fix unclickable minimized windows by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4613
* [dy] Attempt to resolve network issues by dy46 in https://github.com/mage-ai/mage-ai/pull/4620
* [jk] Fix block caching when renaming/detaching blocks by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4626
* [jk] Update tag and block cache when cloning pipeline by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4627
* [jk] Various bugfixes for pipeline search and filter by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4631
* [xy] Use milliseconds in parquet by default. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4634
* Fixed MongoDB reading input by Luishfs in https://github.com/mage-ai/mage-ai/pull/4639
* Fixed random input inside singer destinations by Luishfs in https://github.com/mage-ai/mage-ai/pull/4642
* [jk] Fix double scrollbars and draggable height by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4643
* [jk] UI bugfixes on Version Control page by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4644
* [dy] Fix unit test logging by dy46 in https://github.com/mage-ai/mage-ai/pull/4630
* [dy] Add DB locking by dy46 in https://github.com/mage-ai/mage-ai/pull/4629
* [jk] Bugfixes for Overview Dashboard by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4659
* [jk] Bugfix for save block shortcut by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4667
* Fixed postgreSQL reserved words by Luishfs in https://github.com/mage-ai/mage-ai/pull/4640
* [dy] Minor workspace fixes by dy46 in https://github.com/mage-ai/mage-ai/pull/4658
* [jk] Sync block contents for replica block when minimizing by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4672
* Incorrect spelling for the word level in documentation by Sa1tyBiscuit in https://github.com/mage-ai/mage-ai/pull/4635

💅 Enhancements & Polish
* [xy] Support filtering google sheets by sheet names by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4604
* [xy] Combine schedulers. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4614
* [xy] Separate scheduler logs and pipeline logs. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4621
* [xy] Update initial load of log_based replication to use incremental sync by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4623
* [jk] Redirect to login page immediately by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4662
* [jk] Clean up chart block widths in Pipeline Editor by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4665
* [xy] Support extra connection arguments in mysql source and destination. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4668
* [xy] Fix image build by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4671
* Add DEFAULT_EXECUTOR_TYPE by danieldiamond in https://github.com/mage-ai/mage-ai/pull/4597
* [dy] Filter out sensitive config by dy46 in https://github.com/mage-ai/mage-ai/pull/4594
* mage_ai/io MSSQL connection: add support for DATETIMEOFFSET by hugabora in https://github.com/mage-ai/mage-ai/pull/4619
* [xy] Bump up more versions to resolve vulnerabilities. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4444


New Contributors
* danieldiamond made their first contribution in https://github.com/mage-ai/mage-ai/pull/4597
* carillpower made their first contribution in https://github.com/mage-ai/mage-ai/pull/4625
* Sa1tyBiscuit made their first contribution in https://github.com/mage-ai/mage-ai/pull/4635

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.64...0.9.65

<div align="center">
<img src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExZnJ6dnd3ZGwzbjN3NHdkOXRwejJxaG5reThwdWs0Mnl1NGw1OWR2MiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/tyfRqk4pw3W8Qpcuhp/giphy-downsized.gif" width="50%" />
</div>

<!-- Release notes generated using configuration in .github/release.yml at master -->

What's Changed
🎉 Exciting New Features
🔭 OpenTelemetry integration
- **Added OpenTelemetry Libraries**: Integrated OpenTelemetry Python libraries to instrument the application for telemetry data collection.
- `opentelemetry-api`: Provides the API for tracing and metrics.
- `opentelemetry-sdk`: The core SDK for OpenTelemetry in Python.
- `opentelemetry-instrumentation`: General instrumentation library.
- `opentelemetry-instrumentation-sqlalchemy`: Specific for instrumenting SQLAlchemy operations.
- `opentelemetry-instrumentation-tornado`: For monitoring Tornado-based HTTP server activities.
- **Environment Variables**: Added new environment variables (`OTEL_EXPORTER_OTLP_HTTP_ENDPOINT` and `OTEL_EXPORTER_OTLP_ENDPOINT`) for configuring the OpenTelemetry Collector endpoints.
- **Database and HTTP Server Instrumentation**: Configured instrumentation for both SQLAlchemy database operations and Tornado HTTP server activities to gather traces and metrics.
- Check out this [doc](https://docs.mage.ai/integrations/observability/opentelemetry) for more details
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/4f57a23b-9047-4cc6-91dc-b8763e596fd0" width="600"/>
</div>

<br/>

by mfreeman451 in https://github.com/mage-ai/mage-ai/pull/4476

🦊 Add GitLab authentication to version control

<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/d8b00ab6-5b3c-41bf-bd7e-3e33a6a0d962" alt="Gitlab" width="600"/>
</div>

<br/>

Support authenticating with GitLab in version control app.
The feature can be enabled by configuring the following environment variables:
* `GITLAB_HOST`
* `GITLAB_CLIENT_ID`
* `GITLAB_CLIENT_SECRET`

_by dy46 in https://github.com/mage-ai/mage-ai/pull/4534_

🔔 Discord notification integration
<div align="center">
<img src="https://github.com/mage-ai/mage-ai/assets/80284865/ff1bb5a7-3110-4b57-a09c-d3a9a4df8331" alt="Discord" width="600"/>
</div>

<br/>

Support setting up Discord notification channel for alerts.
Check out the [doc](https://docs.mage.ai/integrations/observability/alerting-discord) for details

_by luthfir-rahman in https://github.com/mage-ai/mage-ai/pull/4478_

🔑 Add OIDC generic provider
This feature allows users to hook up their OIDC server (e.g. Keycloak ) and use it to authenticate with Mage.
It can be enabled by configuring the following environment variables:
* `OIDC_CLIENT_ID`
* `OIDC_CLIENT_SECRET`
* `OIDC_DISCOVERY_URL`

_by dy46 in https://github.com/mage-ai/mage-ai/pull/4563_

🐛 Bug Fixes
* [xy] Use utc time as the default time for monitor_stats. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4554
* Fix Google Ads Catalog to Data Integration Block by Luishfs in https://github.com/mage-ai/mage-ai/pull/4565
* [jk] Load custom design logo by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4574
* [dy] Adjust alembic logger initialization by dy46 in https://github.com/mage-ai/mage-ai/pull/4552
* [dy] Add support for AWS codecommit by dy46 in https://github.com/mage-ai/mage-ai/pull/4568
* [dy] Fix frontend crashing issues by dy46 in https://github.com/mage-ai/mage-ai/pull/4573
* [xy] Set max width for yaml file to inf. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4576
* [dy] Fix DBT logging by dy46 in https://github.com/mage-ai/mage-ai/pull/4577
* [dy] Show the load balancer ingress host if there is no host in the ingress rules by dy46 in https://github.com/mage-ai/mage-ai/pull/4575
* [xy] Match postgres logical replication records with stream name. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4588
💅 Enhancements & Polish
* Added reserved words settings to PostgreSQL by Luishfs in https://github.com/mage-ai/mage-ai/pull/4142
* [xy] Improve google sheets source. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4561
* [jk] Reuse callback and conditional blocks by johnson-mage in https://github.com/mage-ai/mage-ai/pull/4562
* [xy] Show pandas datetime in iso format in notebook. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4564
* [xy] Add case_sensitive param to the sql export method. by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4569
* [xy] Create TEMP Snowflake table in data integration destination by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/4578
* [dy] Misc workspace/git improvements by dy46 in https://github.com/mage-ai/mage-ai/pull/4583
* [dy] Update user authentication for subprojects by dy46 in https://github.com/mage-ai/mage-ai/pull/4586

New Contributors
* luthfir-rahman made their first contribution in https://github.com/mage-ai/mage-ai/pull/4478

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.63...0.9.64