<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/59450879/7c5a63d7-2703-49b6-83bf-3595b31d6b2f" width="75%"/>
</p>
What's Changed
🎉 Exciting New Features
✨ New AI Functionality
As a part of this release, we have some exciting new AI functionality: you can now generate pipelines and add inline comments using AI. 🤯
<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/1066980/1bdc7b70-0a54-4a2b-8d35-3340adb6eaec" width="75%"/>
</p>
<center><i>Generate entire pipelines using AI</i></center>
<br>
<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/59450879/4fb2856e-af03-4ccd-a63b-5464e75c2fa3" width="75%"/>
</p>
<center><i>Add inline comments using AI</i></center>
<br/>
See the following links for documentation of the new functionality.
* Inline comment guide [here](https://docs.mage.ai/guides/ai/overview#add-comments-in-a-block)
* Generative pipeline guide [here](https://docs.mage.ai/guides/ai/overview#generate-pipeline-using-ai)
_by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3365> and <https://github.com/mage-ai/mage-ai/pull/3359>_
✨ Elasticsearch as sink for streaming
Elasiticsearch is now available as a streaming sink. 🥳🎉
<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/59450879/b4233740-88f8-4677-9b63-23221b4549b1" width="75%"/>
</p>
A _big_ thanks to sujiplr for their contribution!
_by sujiplr in <https://github.com/mage-ai/mage-ai/pull/3335>_
✨ Add GitHub source
The GitHub API is now available as a data integrations— you can pull in commits, changes, and more from the [GitHub API](https://docs.github.com/en/rest)!
_by mattppal in <https://github.com/mage-ai/mage-ai/pull/3252>_
✨ Interpolate variables in dbt profiles
This is a big one for our dbt users out there! You can now use `{{ variables('...') }}` in dbt `profiles.yml`.
yaml
jaffle_shop:
outputs:
dev:
dbname: postgres
host: host.docker.internal
port: 5432
schema: {{ variables('dbt_schema') }}
target: dev
That means pulling in custom Mage variables, directly!
_by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3337>_
✨ Enable sorting on pipelines dashboard
Some great frontend improvements are going down! You can now sort pipelines on the dashboards, both with and without groups/filters enabled!
<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/78053898/b4717277-1006-4a91-b6b9-a63e0901a4ec" width="75%"/>
</p>
_by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3327>_
✨ Display data integration source docs inline
Another _awesome_ community contribution— this one also on the frontend. Thanks to splatcollision, we now have inline documentation for our data integration sources!
<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/106864/3d1f6251-5781-49e3-b0e5-26d1ceb6521b" width="75%"/>
</p>
Now, you can see _exactly_ what you need, directly from the UI!
_by splatcollision in <https://github.com/mage-ai/mage-ai/pull/3349>_
🐛 Bug Fixes
* Fix MongoDB source by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3342>
* Overwrite existing project_uuid if a project_uuid is set when starting the server by dy46 in <https://github.com/mage-ai/mage-ai/pull/3338>
* Add started_at to pipeline run policy by dy46 in <https://github.com/mage-ai/mage-ai/pull/3345>
* Fix passing variables to run_blocks by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3346>
* Not pass liveness_probe to cloud run jobs by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3355>
* Fix data integration pipeline column renaming in transformer block by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3360>
* Fix FileContentResource file path check by dy46 in <https://github.com/mage-ai/mage-ai/pull/3363>
* Update api/status endpoint by dy46 in <https://github.com/mage-ai/mage-ai/pull/3352>
* Removed unique_constraints from valid_replication_keys definition by Luishfs in <https://github.com/mage-ai/mage-ai/pull/3369>
* Fix dbt command failure due to empty content args by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3376>
💅 Enhancements & Polish
✨ New Docs Structure
You might notice our docs have a new look! We've changed how we think about side-navs and tabs.
<p align="center">
<img src= "https://github.com/mage-ai/mage-ai/assets/59450879/2e2adcc4-3ca0-4761-8c82-71d0d7e5e6c4" width="75%"/>
</p>
Our goal is to help you find what you need, faster. We hope you like it!
_by mattppal in <https://github.com/mage-ai/mage-ai/pull/3324> and <https://github.com/mage-ai/mage-ai/pull/3367>
✨ Other Enhancements
* Explain cron syntax in a human readable way by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3348>
* Add run_pipeline_in_one_process in pipeline settings by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3347>
* Preview next pipeline run time for triggers by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3341>
* Optimize Docker build by christopherscholz in <https://github.com/mage-ai/mage-ai/pull/3329>
* Integration pipeline column explanations by johnson-mage in <https://github.com/mage-ai/mage-ai/pull/3353>
* Allow using k8s executor config for streaming pipeline by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3371>
* Add role to Snowflake data integration pipeline by wangxiaoyou1993 in <https://github.com/mage-ai/mage-ai/pull/3374>
* Disable automatic dbt tests by tommydangerous in <https://github.com/mage-ai/mage-ai/pull/3357>
* Update terminal server user token handling by dy46 in <https://github.com/mage-ai/mage-ai/pull/3350>
* Restrict file creation via API by dy46 in <https://github.com/mage-ai/mage-ai/pull/3351>
New Contributors
* sujiplr made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3335>
* splatcollision made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3349>
* pilosoposerio made their first contribution in <https://github.com/mage-ai/mage-ai/pull/3377>
**Full Changelog**: <https://github.com/mage-ai/mage-ai/compare/0.9.16...0.9.19>