Released on 09-22-2022
PipeRider version 0.9.0 is out now! We added volume & freshness metrics, and you can now customized the tables and row count you want to profile!
:chart_with_upwards_trend: Added volume & freshness metrics
See volume size of your table, when created, and when last update:
- Snowflake: Volume size, Created time, Last altered time
- BigQuery: Volume size, Created time, Last altered time
- Redshift: Volume size
[See full overview of supported metrics](https://github.com/InfuseAI/piperider/blob/main/docs/metrics.md#table-metrics)
:file_folder: Include or exclude tables in config
Now, you can specify the tables to include or exclude in a project
1. Edit the configuration file under `.piperider/config.yml`
2. Add the following config
yaml
config.yml
includes: ["table1", "table2", "table3"]
or tables to exclude
excludes: ["table4"]
:zap: Do a speedrun by limiting the amount of rows
You can now set a limit* on the rows to profile, so you can do a quick run on large datasets.
1. Edit the configuration file under `.piperider/config.yml`
2. Add the following config
yaml
config.yml
profiler:
table:
the maximum row count to profile. (Default unlimited)
limit: 1000000
**Keep in mind: setting the row limit will pick the first n rows of your table. The behavior of the limit is the same as the keyword limit in SQL*
New Contributors
* jenswilms made their first contribution in https://github.com/InfuseAI/piperider/pull/427
Get in touch :heart:
We'd love your feedback! Send us a message via Intercom on [piperider.io](https://piperider.io), join our [Discord](https://discord.gg/xKxsdPx4d5), report an issue on [GitHub](https://github.com/InfuseAI/piperider/issues), or send us a good old-fashioned [email](mailto:productinfuseai.io)
----
<details>
<summary>All changes</summary>
* [Release] Bump version to 0.9.0 by kentwelcome in https://github.com/InfuseAI/piperider/pull/423
* [Feature] simple breadcrumb nav 2 by jonycfu in https://github.com/InfuseAI/piperider/pull/424
* [Feature] sc-28513 Show progress when loading large CSV or parquet file by kentwelcome in https://github.com/InfuseAI/piperider/pull/426
* [Feature] - sc 28379/UI add table metadata info by jonycfu in https://github.com/InfuseAI/piperider/pull/428
* [Feature] Add table profiling rows limit by wcchang1115 in https://github.com/InfuseAI/piperider/pull/429
* [Feature] Apply includes and excludes config to profiler by wcchang1115 in https://github.com/InfuseAI/piperider/pull/430
* [Feature] upload report command by qrtt1 in https://github.com/InfuseAI/piperider/pull/408
* [Document] Update README.md by jenswilms in https://github.com/InfuseAI/piperider/pull/427
* [Document] Update readme data profiling by jenswilms in https://github.com/InfuseAI/piperider/pull/435
* [Chore] assertion labels by neighborhood999 in https://github.com/InfuseAI/piperider/pull/420
* [Chore] Feature/sc 28380 - Prepare UI Library Components by jonycfu in https://github.com/InfuseAI/piperider/pull/410
* [Chore] Enhance piperider init flow by wcchang1115 in https://github.com/InfuseAI/piperider/pull/409
* [Bug] table schema overflow by neighborhood999 in https://github.com/InfuseAI/piperider/pull/425
* [Bug] Fix the overflow issue for the `sum` metric by popcornylu in https://github.com/InfuseAI/piperider/pull/433
* [Bug] Use built-in stddev in all data sources except sqlite by wcchang1115 in https://github.com/InfuseAI/piperider/pull/431
* [Bug] In piperider init, merge the credential only in non-dbt project by popcornylu in https://github.com/InfuseAI/piperider/pull/434
**Full Changelog**: https://github.com/InfuseAI/piperider/compare/v0.8.0...v0.9.0
</details>