Mage-ai

<div align="center">
<img width="50%" src="https://github.com/mage-ai/mage-ai/assets/59450879/63d9a6c4-5db1-44fe-b46e-ede319ceada7">
</div>

What's Changed

🎉 Exciting New Features

🧠 Add support for `dbt-synapse`

This is going to "dbt" amazing! Mage now supports the `dbt-synapse` library, allowing dbt to be executed against the Azure Synapse Data Warehouse. You can read more on the package [here](https://github.com/mage-ai/mage-ai/pull/3657). Excited to see our Azure users make use of this one!

_dy46 in https://github.com/mage-ai/mage-ai/pull/3657_

☁️ Added Google Cloud Pubsub as a sink for streaming pipeline

A big shout out to pammusankolli, who recently added Google Cloud Pubsub as a sink for data streaming pipelines! If you're not familiar with PubSub, you can read more [here](https://cloud.google.com/pubsub?hl=en). Be sure to check out the [docs](https://docs.mage.ai/streaming/destinations/google-cloud-pubsub) in Mage to build your next pipeline!

_pammusankolli in https://github.com/mage-ai/mage-ai/pull/3689_

🤓 Support HTML tags in Markdown blocks

All of our Markdown enthusiasts will appreciate this one! Previously, Mage's markdown blocks only supported images via this format:




Now, Mage supports `<img>` elements with custom sizes in Markdown blocks by providing `width` and `height` attributes like so:


<img src="https://images.photowall.com/products/57215/golden-retriever-puppy.jpg?h=699&q=85" alt="drawing" width="200"/>


Here are some examples:

<div align="center">
<img width="60%" src="https://user-images.githubusercontent.com/8130751/273016163-91d46a77-3fdb-4d20-a9eb-fda72c440667.png">
</div>
<div align="center">
<img width="60%" src="https://user-images.githubusercontent.com/8130751/273017945-08f607cb-204d-44b2-b08b-1ac33ec4276f.png">
</div>

Nice!

_anniexcheng in https://github.com/mage-ai/mage-ai/pull/3692_

🔍 Added Opensearch Destination

<div align="center">
<img width="70%" src="https://github.com/mage-ai/mage-ai/assets/59450879/5a0a4569-72db-4723-a735-b6710ba3a791">
</div>

Mage now supports OpenSearch as a destination in data integration pipelines! 🥳

_Luishfs in https://github.com/mage-ai/mage-ai/pull/3719_

🐛 Bug Fixes

* Hide empty images by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3660
* YAML block language display by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3678
* Add None for block check by dy46 in https://github.com/mage-ai/mage-ai/pull/3677
* Raise exception if there are serialization errors by dy46 in https://github.com/mage-ai/mage-ai/pull/3630
* Fixed Twitter ads imports by Luishfs in https://github.com/mage-ai/mage-ai/pull/3690
* Fixed target-salesforce imports by Luishfs in https://github.com/mage-ai/mage-ai/pull/3688
* Fix git config preserve when `.preferences.yaml` file doesn't exist by dy46 in https://github.com/mage-ai/mage-ai/pull/3693
* Fix MySQL source int comparisonby wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3709
* Suppress secret logs for aws secrets by dy46 in https://github.com/mage-ai/mage-ai/pull/3704
* Update Backfill policy by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3723
* Fix block tests not getting output by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3724
* Remove the requirement for schedule type by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3726
* Check for empty column headers by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3729
* Fixed typo and simplified text for empty dashboard by MageKai in https://github.com/mage-ai/mage-ai/pull/3668

💅 Enhancements & Polish

* Propagate tags of ECS task definition to task executors by gabrieleiacono in https://github.com/mage-ai/mage-ai/pull/3670
* Also search pipeline run event_variables by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3672
* Read AWS secrets from Mage secrets when no ENVs are set by jamesking-github in https://github.com/mage-ai/mage-ai/pull/3607
* Speed up IO MSSQL export method by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3697
* Support `UPDATE` command in raw SQL command by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3705
* Allow customizing batch fetch limit for data integration pipelineby wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3713
* Support using custom Spark sessionby wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3725
* Add activity to status endpoint when requested by dy46 in https://github.com/mage-ai/mage-ai/pull/3708
* Replace runtime variables card list with table by anniexcheng in https://github.com/mage-ai/mage-ai/pull/3703
* Storybook cleanup by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3658

New Contributors

* gabrieleiacono made their first contribution in https://github.com/mage-ai/mage-ai/pull/3670
* anniexcheng made their first contribution in https://github.com/mage-ai/mage-ai/pull/3692

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.30...0.9.34

<div align="center">
<img width="50%" src="https://github.com/mage-ai/mage-ai/assets/59450879/bd58e276-2671-430f-be66-4dedde8064df">
</div>


What's Changed
🎉 Exciting New Features

🌊 Streaming: Base Class Overhaul + 8 New Destinations

This. is. huge. With a complete base class re-write, _every_ IO destination is now a streaming destination.

That means you can stream to:
* [BigQuery](https://docs.mage.ai/streaming/destinations/bigquery)
* [ClickHouse](https://docs.mage.ai/streaming/destinations/clickhouse)
* [DuckDB](https://docs.mage.ai/streaming/destinations/duckdb)
* [MSSQL](https://docs.mage.ai/streaming/destinations/mssql)
* [MySQL](https://docs.mage.ai/streaming/destinations/mysql)
* [Redshift](https://docs.mage.ai/streaming/destinations/redshift)
* [Snowflake](https://docs.mage.ai/streaming/destinations/snowflake)
* [Trino](https://docs.mage.ai/streaming/destinations/trino)

And _any_ future destinations added as an IO base. Huge shutout to wangxiaoyou1993 on the herculean effort!

_by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3623_

👀 Recently viewed pipelines

Some frontend polish now allows you to see your _Recently Viewed_ pipelines from the overview page— a nice touch!

<div align="center"> <img width="50%" src="https://user-images.githubusercontent.com/1066980/271766515-bcca0bb8-1cae-49ee-a0eb-fe6bce6d5113.png">
</div>

_by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3633_

👨‍👩‍👧‍👦 Community: GCS Sensors

pammusankolli deliver's his first contribution by adding a Google Cloud Storage sensor to check if a file exists in a given bucket! Thanks for the add— this will be super useful to our Google platform users!

_by pammusankolli in https://github.com/mage-ai/mage-ai/pull/3651_

🧩 Support syncing Git submodules

A solid improvement to our [Git Sync](https://docs.mage.ai/getting-started/setting-up-git) functionality, you can now sync submodules, too! Just check the Git Sync settings to enable the feature.

<div align="center"> <img width="50%" src="https://user-images.githubusercontent.com/14357209/270447859-9c2220d4-5a67-4e4f-8148-459f539d6954.png">
</div>

_by dy46 in https://github.com/mage-ai/mage-ai/pull/3593_

🔂 Add always on schedule interval

On user request, we've added an `always_on` interval for scheduled triggers. Always on schedules will trigger the _new_ pipeline run as soon as the _latest_ pipeline run is completed.

Once a pipeline run ends, regardless of whether or not it failed or succeeded, it will start a new run. Let us know if you find that valuable!

_by dy46 in https://github.com/mage-ai/mage-ai/pull/3611_


🐛 Bug Fixes
* Lazy import dbt block and library by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3608
* Fix invalid keyword argument error when requesting a list from Stripe by jdvermeire in https://github.com/mage-ai/mage-ai/pull/3479
* Fix snowflake output column names by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3612
* Fix charts when trying to change name by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3614
* Fix export_batch_data not accepting tags as a kwarg by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3619
* Handle secret key and uuid whitespace by dy46 in https://github.com/mage-ai/mage-ai/pull/3621
* Fix materializing upstream outputs for dbt by christopherscholz in https://github.com/mage-ai/mage-ai/pull/3617
* Fix memory leak in data integration pipelines by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3641
* Initialize db connection session by dy46 in https://github.com/mage-ai/mage-ai/pull/3609
* Fix auto generating mage_sources.yml for dbt upstream blocks by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3647
* Pipelines list fixes by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3648

💅 Enhancements & Polish
* Add sortable block run columns by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3613
* Clarify components based on UTC time in Overview page by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3615
* Support interpolating pipeline variables in streaming configs by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3616
* Scheduler trigger interval as environment variable setting by PopaRares in https://github.com/mage-ai/mage-ai/pull/3591
* Pipeline runs table (for individual pipelines) updates by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3629
* Reduce dynamic child block at any level by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3634
* Apache Kafka - SASL_PLAINTEXT by Senpumaru in https://github.com/mage-ai/mage-ai/pull/3643
* Support Kafka SASL_SSL SCRAM-SHA-512 authentication by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3649
* Update search_path command by dy46 in https://github.com/mage-ai/mage-ai/pull/3625

New Contributors
* jdvermeire made their first contribution in https://github.com/mage-ai/mage-ai/pull/3479
* PopaRares made their first contribution in https://github.com/mage-ai/mage-ai/pull/3591
* Senpumaru made their first contribution in https://github.com/mage-ai/mage-ai/pull/3643
* pammusankolli made their first contribution in https://github.com/mage-ai/mage-ai/pull/3651

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.28...0.9.30

<!-- Release notes generated using configuration in .github/release.yml at release/0.9.28 -->

<div align="center">
<img width="50%" src="https://github.com/mage-ai/mage-ai/assets/59450879/dec4df43-3b1a-4f18-87ba-597bfd8a6ff8">
</div>

What's Changed

🎉 Exciting New Features

🫨 Brand new dbt blocks!

One of our _top_ contributors christopherscholz just delivered a _huge_ feature! A completely streamlined dbt Block!

<div align="center">
<img width="50%" src="https://github.com/mage-ai/mage-ai/assets/59450879/c781a298-eebe-4615-8059-9ee2c91f0180">
</div>

Here are some of the highlights:

- Directly integrated into `dbt-core`, instead of calling it via a subprocess, which allows to use all of dbts functionalities
- Use dbt to seed output dataframes from upstream blocks
- Use dbt to generate correct relations e.g. default schema names, which differ between databases
- Use dbt to preview models, by backporting the `dbt seed` command to `dbt-core==1.4.7`
- No use of any mage based database connections to handle the block
- Allows to install any dbt adapter, which supports the dbt-core version
- Moved all code into a single interface called `DBTBlock`
- Doubles as a factory for child blocks `DBTBlockSQL` and `DBTBlockYAML`
- Child blocks make it easier to understand which block does what

There's lots to unpack in this one, so be sure to read more in the PR below and check out our updated [docs](https://docs.mage.ai/dbt/overview).

_by christopherscholz in https://github.com/mage-ai/mage-ai/pull/3497_

➕ Add GCS storage to store block output variables

Google Cloud users rejoice! Mage _already_ supports storing block output variables in S3, but thanks to contributor luizarvo, you can now do the same in GCS!

Check out the PR for more details and read-up on implementation [here](https://docs.mage.ai/design/data-pipeline-management#variable-storage).

_by luizarvo in https://github.com/mage-ai/mage-ai/pull/3597_

✨ Tableau Data Integration Source

<div align="center">
<img width="65%" src="https://dev3lop.com/wp-content/uploads/2017/04/tableau-software-logo.png">
</div>

Another community-led integration! Thank you mohamad-balouza for adding a Tableau source for data integration pipelines!

<div align="center">
<img width="65%" src="https://user-images.githubusercontent.com/125284049/270114084-b034876e-9ea5-4790-8dbe-a5039a281ff5.png">
</div>

_by mohamad-balouza in https://github.com/mage-ai/mage-ai/pull/3581_

🦆 Add DuckDB loader and exporter templates

Last week, we rolled out _a ton_ of new DuckDB functionality, this week, we're adding DuckDB loader and exporter templates! Be sure to check them out when building your _new_ DuckDB pipelines! 😄

<p align="center">
<img width="35%" alt="image" src="https://user-images.githubusercontent.com/5386254/269154636-c2b93d8d-bfde-4323-a498-e16341991ce2.png">
<img width=30% alt="image" src="https://user-images.githubusercontent.com/5386254/269154582-475aa465-6df7-40b5-a8e0-4c1579665b11.png">
</p>

_by matrixstone in https://github.com/mage-ai/mage-ai/pull/3553_

🧱 Bulk retry incomplete block runs

Exciting frontend improvements are coming your way! You can now retry all of a pipeline's incomplete block runs from the UI. This includes all block runs that do not have `completed` status.

<div align="center">
<img width="65%" src="https://github.com/mage-ai/mage-ai/assets/59450879/08c31548-9847-41e1-9dfd-1a8a41d90d77">
</div>




🐛 Bug Fixes

* Fix using `S3Storage` to store block output variables by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3559 and https://github.com/mage-ai/mage-ai/pull/3588
* Support local timezone for cron expressions by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3561
* Make API middlware set status codes if there's an API key or OAUTH token error by splatcollision in https://github.com/mage-ai/mage-ai/pull/3560
* Fix Postgres connection url parsing by dy46 in https://github.com/mage-ai/mage-ai/pull/3570
* Fix passing in logger for alternative block execution methods by dy46 in https://github.com/mage-ai/mage-ai/pull/3571
* Fix variables interpolation in dbt target by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3578
* Fixed `dbt seed` requiring variables by tommydangerous in https://github.com/mage-ai/mage-ai/pull/3579
* Fix updating of `pipelineRowsSorted` when clearing search query by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3596
* Check yaml serialization before writing variable by dy46 in https://github.com/mage-ai/mage-ai/pull/3598
* Fix `condition_failed` check for dynamic blocks by dy46 in https://github.com/mage-ai/mage-ai/pull/3595

💅 Enhancements & Polish

* Display warning on demo site to prevent users from entering private credentials by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3550
* Update wording for empty pipeline template state by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3557
* Reorder upstream blocks by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3541
* Prioritize using `remote_variables_dir` for variable manager by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3562
* Customize ecs config by dy46 in https://github.com/mage-ai/mage-ai/pull/3558
* Bookmark values minor improvements by johnson-mage in https://github.com/mage-ai/mage-ai/pull/3576
* Speed up bigquery destination in data integration pipeline by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3590
* Allow declared cookies to propagate to resources by hugabora in https://github.com/mage-ai/mage-ai/pull/3555
* Refactor: use generic function to call LLM avoid code duplication by matrixstone in https://github.com/mage-ai/mage-ai/pull/3358
* Allow configuring EMR cluster spark properties by wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/3592
* Support `TIMESTAMP` in redshift convert by RobinFrcd https://github.com/mage-ai/mage-ai/pull/3567

New Contributors

* luizarvo made their first contribution in https://github.com/mage-ai/mage-ai/pull/3597
* RobinFrcd made their first contribution in https://github.com/mage-ai/mage-ai/pull/3567

**Full Changelog**: https://github.com/mage-ai/mage-ai/compare/0.9.26...0.9.28

<div align="center">
<img width="45%" src="https://media.tenor.com/k4CM51R0zFoAAAAC/welcome-home-lee-christmas.gif">
</div>

What's Changed

🎉 Exciting New Features

🐣 DuckDB IO Class and SQL Block

Folks, we've got some ducking magic going on in this release. You can now use DuckDB files inside Mage's SQL Blocks. 🥳 🦆 🪄

You can use data loaders to `CREATE` and `SELECT` from DuckDB tables as well as write new data to DuckDB.

<div align="center">
<img width="85%" src="https://github.com/mage-ai/mage-ai/assets/59450879/416c2b5a-9f03-42f7-9504-a45d01b64851">
</div>

Check out our [docs](https://docs.mage.ai/integrations/databases/DuckDB) to get started today!

_by matrixstone in https://github.com/mage-ai/mage-ai/pull/3463_

<p align="center">
<img width="65%" src= "https://github.com/mage-ai/mage-ai/assets/59450879/aba79128-64fa-4daa-8272-c8dfd9f1ef56"/>
</p>

What's Changed

🎉 Exciting New Features

✨ Add & update block variables through UI

📰 Hot off the press, you can now add and update block variables via the Mage UI!

<p align="center">
<img width="40%" alt="image" src="https://user-images.githubusercontent.com/78053898/266462210-865e738c-ab81-4040-96a0-d631916db65f.gif">
<img width=51% alt="image" src="https://user-images.githubusercontent.com/78053898/266471463-cdefeb1f-6db5-4123-b57c-23895ecf1ba5.png">
</p>

Check out our [docs](https://docs.mage.ai/development/variables/overview#block-variables) to learn more about block variables.

_by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3451>_

✨ PowerBI source [Data Integration]

<p align="center">
<img width="65%" src= "https://github.com/mage-ai/mage-ai/assets/59450879/1323e29a-7676-4f91-848c-d8d972692652"/>
</p>

You can now pull PowerBI data in to all your Mage projects!

A big shoutout to mohamad-balouza for this awesome contribution. 🎉

Read more about the connection [here](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/powerbi).

_by mohamad-balouza in <https://github.com/mage-ai/mage-ai/pull/3433>_

✨ Knowi source [Data Integration]

<p align="center">
<img width="65%" src= "https://github.com/mage-ai/mage-ai/assets/59450879/56d64008-7433-4db1-81ae-12bf39f2dd89"/>
</p>

mohamad-balouza was hard at work! You can _also_ integrate Knowi data in your Mage projects! 🤯

Read more about the connection [here](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/knowi#readme).

_by mohamad-balouza in <https://github.com/mage-ai/mage-ai/pull/3446>_

✨ InfluxDB Destination [Streaming]

<p align="center">
<img width="65%" src= "https://github.com/mage-ai/mage-ai/assets/59450879/050341d7-78af-4cd8-abb7-b8df861e8edb"/>
</p>

A big shoutout to mazieb! You can now stream data to InfluxDB via Mage. Thanks for your hard work!

Read more in our docs [here](http://docs.mage.ai/streaming/destinations/influxdb).

_by mazieb in <https://github.com/mage-ai/mage-ai/pull/3378>_

🐛 Bug Fixes

* Fix command on GCP cloud run by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3437>
* Fix Git sync issues by dy46 in <https://github.com/mage-ai/mage-ai/pull/3436>
* Bump GitPython and update Git Sync logic by dy46 in <https://github.com/mage-ai/mage-ai/pull/3458>
* Remove unnecessary `status` query from block runs request by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3444>
* Fix AWS secrets manager dependency issue by dy46 in <https://github.com/mage-ai/mage-ai/pull/3448>
* Fix comparing bookmark value for timestamp column in BigQuery by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3452>
* Compare `start_time` and `execution_date` in `should_schedule` by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3466>
* Fix initial ordering of pipeline rows by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3472>

💅 Enhancements & Polish

* Update Triggers list page with filters and clean up subheader by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3424>
* Fix and optimize pre-commit hook check-yaml and add check-json by christopherscholz in <https://github.com/mage-ai/mage-ai/pull/3412>
* Full Code Quality Check in Github Action Workflow "Build and Test" by christopherscholz in <https://github.com/mage-ai/mage-ai/pull/3413>
* Docker: Lint Image and some best practices by christopherscholz in <https://github.com/mage-ai/mage-ai/pull/3410>
* Support for `local_python_force` executor type and configuring ECS executor launch type wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3447>
* Limit Github Action Workflow "Build and Test" by christopherscholz in <https://github.com/mage-ai/mage-ai/pull/3411>
* Hide OpenAI API key by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3455>
* Add additional health check message and support unique constraints in Postgres sink by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3457>
* Support autoscaling EMR clusters by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3468>
* Support GitHub Auth for different workspaces by dy46 in <https://github.com/mage-ai/mage-ai/pull/3456>
* Add timestamp precision in Trino by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3465>
* Handle transformer renaming columns not breaking destination data types in integration pipelines by splatcollision in <https://github.com/mage-ai/mage-ai/pull/3462>

New Contributors

* mazieb made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3378>

**Full Changelog**: <https://github.com/mage-ai/mage-ai/compare/0.9.21...0.9.23>

<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/59450879/f3e31865-cfaf-41bf-a82d-738607a13dac"/>
</p>

What's Changed

🎉 Exciting New Features

✨ Single-task ECS pipeline executor 🤖

Mage now supports running the whole pipeline process in one AWS ECS task instead of running pipeline blocks in separate ECS tasks! This allows you to speed up pipeline execution in ECS tasks by saving ECS task startup time.

Here's an example pipeline `metadata.yaml`:

yaml
blocks:
- ...
- ...
executor_type: ecs
run_pipeline_in_one_process: true
name: example_pipeline
...


The ECS `executor_config` can also be configured at the pipeline level.

_by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3418>_

✨ Postgres streaming destination 🐘

Postgres enthusiasts rejoice! You can now stream data _directly_ to Postgres via streaming pipelines! 😳

Check out [the docs](https://docs.mage.ai/streaming/destinations/postgres) for more information on this handy new destination.

_by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3423>_

✨ Added sorting to the _Block Runs_ table

You can now sort the _Block Runs_ table by clicking on the column headers! Those of us who are passionate about having our ducks in a row are happy about this one! 🦆

<p align="center">
<img width="75%" src="https://user-images.githubusercontent.com/78053898/263441796-c11d4689-74de-40cf-8258-fe436034c345.gif"/>
</p>

_by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3356>_

✨ Enable deleting individual runs from pipeline runs table

Bothered by that one run you'd rather forget? Individual runs can be dropped from the pipeline runs table, so you don't have to worry about them anymore!

<p align="center">
<img width="75%" src= "https://user-images.githubusercontent.com/78053898/263891384-65e00954-6751-44b2-9f36-da8883f4e5ab.gif"/>
</p>

_by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3370>_

✨ Add timeout for block runs and pipeline runs

Much like Buzz Lightyear, we're headed "to infinity and beyond," but we get that your pipelines shouldn't be. This feature allows you to configure timeouts for both blocks and pipelines— if a run exceeds the timeout, it will be marked as failed.

<p align="center">
<img width="40%" alt="image" src="https://user-images.githubusercontent.com/14357209/264756458-dca57502-66b3-4b31-9895-578533ab1a14.png">
<img width="40%" alt="image" src="https://github.com/mage-ai/mage-ai/assets/59450879/8dfcec8f-fdcb-44bd-9947-05b85af3cdcb">
</p>

_by dy46 in <https://github.com/mage-ai/mage-ai/pull/3399>_

🐛 Bug Fixes

* Fix `nextjs` local build type error by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3389>
* Fixed `NULL` headers breaking API Source by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3386>
* Fixes on MongoDB destination (Data integration) by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3388>
* Fix Git commit from version control by dy46 in <https://github.com/mage-ai/mage-ai/pull/3397>
* Change the default search path to the schema given in the connection URL for PostgreSQL by csharplus in <https://github.com/mage-ai/mage-ai/pull/3406>
* Raise exception if a pipeline has duplicate block uuids by dy46 in <https://github.com/mage-ai/mage-ai/pull/3385>
* Fix creating subfolder when renaming block by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3419>
* Docker: Installation Method of Node using `setup_17.x` is no longer supported by christopherscholz in <https://github.com/mage-ai/mage-ai/pull/3405>
* [dy] Add support for key properties for postgresql destination by dy46 in https://github.com/mage-ai/mage-ai/pull/3422

💅 Enhancements & Polish

* Added disable schema creation config by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3417>
* Read DB credentials from AWS secret manager by dy46 in <https://github.com/mage-ai/mage-ai/pull/3354>
* Add healthcheck to sqs streaming sourceby wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3393>
* Enhancement: token expiry time env by juancaven1988 in <https://github.com/mage-ai/mage-ai/pull/3396>
* Auto-update triggers in code when making changes in UI by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3395>
* Add fetch and reset to version control app by dy46 in <https://github.com/mage-ai/mage-ai/pull/3409>
* Set correct file permissions when using shutil by christopherscholz in <https://github.com/mage-ai/mage-ai/pull/3262>
* Create `lsn` and `_mage_deleted_at` in initial log_based syncby wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3394>
* Feat: Added Missing destination block warning by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3407>

New Contributors

* WesAkers made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3382>

**Full Changelog**: <https://github.com/mage-ai/mage-ai/compare/0.9.19...0.9.21>