Great-expectations

What’s Changed

* Allow basic operations in evaluation parameters, with or without evaluation parameters.
* When unexpected exceptions occur (e.g., during data docs rendering), the user will see detailed error messages, providing information about the specific issue as well as the stack trace.
* Remove the "project new" option from the command line (since it is not implemented; users can only run "init" to create a new project).
* Update type detection for bigquery based on driver changes in pybigquery driver 0.4.14. Added a warning for users who are running an older pybigquery driver
* added execution tests to the NotebookRenderer to mitigate codegen risks
* Add option "persist", true by default, for SparkDFDataset to persist the DataFrame it is passed. This addresses 1133
in a deeper way (thanks tejsvirai for the robust debugging support and reproduction on spark).
- Disabling this option should *only* be done if the user has *already* externally persisted the DataFrame, or if the
dataset is too large to persist but *computations are guaranteed to be stable across jobs*.
* Enable passing dataset kwargs through datasource via dataset_options batch_kwarg.
* Fix AttributeError when validating expectations from a JSON file
* Data Docs: fix bug that was causing erratic scrolling behavior when table of contents contains many columns
* Make expectation suite json deterministically human-parseable (1247) mzjp2
* Change the verb of the cli command project in docs (1237) Bussmeyer
* Fix AttributeError when validating expectations from a JSON file (1238) eorbe
* Fixed link to typical_workflow (1227) jvanbuel
* Remove Python 2 tags from setup.py (1229) mzjp2

Changes

* Update marshmallow dependency to >3. NOTE: as of this release, you MUST use marshamllow >3.0, which REQUIRES python 3. 1187 jcampbell
- Schema checking is now stricter for expectation suites, and data_asset_name must not be present as a top-level
key in expectation suite json. It is safe to remove.
- Similarly, datasource configuration must now adhere strictly to the required schema, including having any
required credentials stored in the "credentials" dictionary.
* New beta CLI command: `tap new` that generates an executable python file to expedite deployments. 1193 Aylr
* Added feature maturity in README 1203 kyleaton

Fixes

* Fix failing test that should skip if postgresql not running 1199 cicdw
* bugfix in TableBatchKwargsGenerator docs

* validate result dict when instantiating an ExpectationValidationResult (1133)
* DataDocs: Expectation Suite name on Validation Result pages now link to Expectation Suite page
* `great_expectations init`: cli now asks user if csv has header when adding a Spark Datasource with csv file
* Improve support for using GCP Storage Bucket as a Data Docs Site backend (thanks hammadzz)
* fix notebook renderer handling for expectations with no column kwarg and table not in their name (1194 )

What’s Changed

* Notebook empty suite fixes, suite edit bugfixes (1155) Aylr
* Update evaluation_parameters.rst (1167) spbail
* Update expectation_glossary.rst (1164) spbail
* release checklist (1146) Aylr

* Update CLI `init` flow to support snowflake transient tables
* Use filename for default expectation suite name in CLI `init`
* Tables created by SqlAlchemyDataset use a shorter name with 8 hex characters of randomness instead of a full uuid
* Better error message when config substitution variable is missing
* removed an unused directory in the GE folder
* removed obsolete config error handling
* Docs typo fixes
* Jupyter notebook improvements
* `great_expectations init` improvements
* Simpler messaging in valiation notebooks
* replaced hacky loop with suite list call in notebooks
* CLI suite new now supports `--empty` flag that generates an empty suite and opens a notebook
* add error handling to `init` flow for cases where user tries using a broken file

* Add support for transient table creation in snowflake (1012)
* Improve path support in TupleStoreBackend for better cross-platform compatibility
* New features on `ExpecatationSuite`
- `.add_citation()`
- `get_citations()`
* `SampleExpectationsDatasetProfiler` now leaves a citation containing the original batch kwargs
* `great_expectations suite edit` now uses batch_kwargs from citations if they exist
* Bugfix :: suite edit notebooks no longer blow away the existing suite while loading a batch of data
* More robust and tested logic in `suite edit`
* DataDocs: bugfixes and improvements for smaller viewports
* Bugfix :: fix for bug that crashes SampleExpectationsDatasetProfiler if unexpected_percent is of type decimal.Decimal (`1109 <https://github.com/great-expectations/great_expectations/issues/1109>`_)