Great-expectations

Changelog:
* [DOCS] correct errors and reference complete example for custom expectations (thanks jdimatteo)
* [DOCS] How to connect to : in-memory Pandas Dataframe
* [DOCS] How to connect to in memory dataframe with spark
* [DOCS] How to connect to : S3 data using Pandas
* [DOCS] How to connect to : Sqlite database
* [DOCS] no longer show util import to users
* [DOCS] How to connect to data on a filesystem using Spark guide
* [DOCS] GDOC-102/GDOC-127 Port in References and Tutorials
* [DOCS] How to connect to a MySQL database
* [DOCS] improved clarity in how to write guide templates and docs
* [DOCS] Add documentation for Rule Based Profilers
* [BUGFIX] Update mssql image version for Azure
* [MAINTENANCE] Update test-sqlalchemy-latest.yml
* [MAINTENANCE] Clean Up Design for Configuration and Flow of Rules, Domain Builders, and Parameter Builders
* [MAINTENANCE] Update Profiler docstring args
* [MAINTENANCE] Remove date format parameter builder
* [MAINTENANCE] Move metrics computations to top-level ParameterBuilder
* [MAINTENANCE] use tmp dot UUID for discardable expectation suite name
* [MAINTENANCE] Refactor ExpectationSuite to include profiler_config in citations
* [FEATURE] Add citations to Profiler.profile()
* [FEATURE] Bootstrapped Range Parameter Builder

untagged-8eb05a585638d336999a
What’s Changed

* [MAINTENANCE] changelog and version number update (2943) OrtalG
* [DOCS] Update TOC to match Design Doc (2915) Shinnnyshinshin
* [ENHANCEMENT] Make expect_column_values_to_be_between faster (2633) janjagusch
* [BUGFIX] Minor fixes to v3 API CLI (2742) spbail
* [FEATURE]/GDOC/GDOC-107/GDOC-124/Adding In Intro Files and Splitting jsx (2934) petermoyer
* [BUGFIX] Docs integration Pipeline now references new requirements files (2941) Shinnnyshinshin
* [MAINTENANCE] Pin json-schema version (2938) anthonyburdi
* Update requirements for docs_rtd (2940) jcampbell
* [MAINTENANCE] remove reference to old requirements file in CI (2939) Aylr
* [DOCS] add Local Python Install to docs (2920) petermoyer
* [BUGFIX] Temporary table objects persist across sessions in Oracle versions 18c, 19c, and 21c (2862) NathanFarmer
* [BUGFIX] Using cache in order not to fetch already known secrets (2882) Cedric-Magnan
* feat: remove accidental new line in .ge_store_backend_id (2896) dnz-bdeboer
* Bump urllib3 from 1.25.10 to 1.26.5 in /docs_rtd (2854) dependabot
* [MAINTENANCE] Remove numpy pin (2930) jcampbell
* [DOCS] How to write a how-to-guide (Docusaurus) (2924) Shinnnyshinshin
* [DOCS] Fix version name in V3 Quick start tutorial (2897) ioni-garfunkel
* [MAINTENANCE] upgrade docusaurus (2932) Aylr
* [MAINTENANCE] Refactor variable and rules configs in profiler into a single dict (2919) cdkini
* [BUGFIX] Instantiate PandasExecutionEngine with custom parameters (2904) xaniasd
* [DOCS] Fixed the expectation suite name kwarg (2875) vikramaditya91
* [BUGFIX] Render 0 properly in datadocs (2923) shpolina
* [FEATURE] Eliminate the use of Validator in creating Profiler, DomainBuilder, and ParameterBuilder classes of the Rule Based Profiler (2925) alexsherstinsky
* [MAINTENANCE] Allow tests to properly connect to local sqlite db on Windows (2891) shpolina
* [BUGFIX] Temp table creation fix (2859) shpolina
* [DOCS] How to connect to your data in BigQuery (2902) Shinnnyshinshin
* [DOCS] Update SLACK_GUIDELINES.md For Tools and Services (2921) kyleaton
* [BUGFIX] Fix inconsistent column names in NYC taxi datasets (2917) cdkini
* [BUGFIX] Make JSON Serialization Utility More Robust (currently fails for Python-3.6 and Pandas-0.23 (2913) alexsherstinsky
* [MAINTENANCE] Add NYC taxi data to testing datasets (2911) cdkini
* [BUGFIX] Fix rendering error (2909) shpolina
* [FEATURE] Initial/Incremental Multi-Batch Profiler Code Contribution (2869) alexsherstinsky
* [DOCS] Small fix to the tutorial (2901) ioni-garfunkel
* Small fix for batch_filter (the tests already cover it). (2899) alexsherstinsky
* [DOCS] Update pr template and remove enhancement feature type (2894) OrtalG
* [MAINTENANCE] DOCS integration tests have moved to a new pipeline (2895) Shinnnyshinshin
* [WIP][FEATURE] Mer-45: GE Cloud Store Backend (2822) roblim
* [DOCS] Bugfix for Database how-to-docs (2883) Shinnnyshinshin
* [BUGFIX] Docs integration tests now only run when `--docs-tests` option is specified (2892) Shinnnyshinshin
* [FEATURE] Ability to Launch Azure Checks Using a Prefix in the Name of a Branch (2890) alexsherstinsky
* Alexsherstinsky/bugfix/duplicated aws region instead of secret 06 08 2021 63 (2888) alexsherstinsky
* [DOCS] Update docstring for SlackNotificationAction (2889) anthonyburdi
* [DOCS] How to connect to your data on a S3 using spark guide (2877) Aylr
* [DOCS] How to Connect to your data on Redshift guide (2879) Shinnnyshinshin
* [ENHANCEMENT] Improve filter_properties_dict handling of falsy values (along with new unit tests for this) (2878) alexsherstinsky
* [BUGFIX] Fixing batch_filter string processing of slice query parameters (and adding corresponding unit tests) (2876) alexsherstinsky
* [MAINTENANCE] Fix broken links after merge of knoxpod branch into develop (2874) Shinnnyshinshin
* [DOCS][MAINTENANCE] remove interactive prototype (2873) Aylr
* [DOCS][MAINTENANCE] docs test CI pipeline trigger is more precise (2867) Aylr
* [ENHANCEMENT] Added SerializableDotDict (as a JSON-serializable extension of DotDict) (2871) alexsherstinsky
* [MAINTENANCE] Fix non-functional Validator test. Make small type hints and style adjustment. (2866) alexsherstinsky
* [KNOXPOD] Snowflake-how-to-guide follow up changes (2851) Shinnnyshinshin
* [MAINTENANCE] Make test utils more general. Also, add an assertion for an expected log warning. (2857) alexsherstinsky
* [DOCS] First portion of alpha docs site (2852) Aylr
* [FEATURE] Rule Based Profiler: Initial Commit (Work-In-Progress) (2788) anthonyburdi
* [DOCS] Fix typo in readme (2838) dafeda
* [DOCS] Update testing.rst (2817) rschmidtner
* [DOCS] Word missing in slack guidelines doc (2835) discdiver
* [BUGFIX] Update util.convert_to_json_serializable() to handle UUID type (2805) YFGu0618
* [BUGFIX] expect_column_kl_divergence_to_be_less_than to not double count on lowest bin boundary (2756) peterdhansen
* [DOCS] Fix typo in Core Concepts documentation (2797) cdkini
* [DOCS] remove broken link (2769) gsajko
* [MAINTENANCE] CLI tests now support click 8.0 and 7.x (2791) Aylr
* Add integration test for multi-BatchRequest (2776) talagluck
* [MAINTENANCE] Improve robustness of integration test_runner (2772) talagluck
* [FEATURE] - new expectation: expect_column_values_to_change_between (2618) maikelpenz
* Enable instantiation of a validator with multiple batches (2765) talagluck
* [MAINTENANCE] Add nightly CI run to test the Latest SQLAlchemy version with GE/postgres (2760) anthonyburdi
* [MAINTENANCE] Add type hints to BatchFilter (2749) alexsherstinsky
* [BUGFIX] Allow decimals without leading zeros in evaluation parameters (2725) makotonium
* [BUGFIX] Add a parameter to `SparkDFDatasource` to reuse the Spark Context (2733) j-f-r
* [DOCS] Getting started tutorial for the v3 API (2724) spbail
* [ENHANCEMENT] Ge 5/add taxi multibatch fixture (2716) talagluck

This release fixes a packaging error that prevented the CLI commands nested under `suite` from working in the V3 API.

* [ENHANCEMENT] Improve support for quantiles calculation in Athena
* [ENHANCEMENT] V3 API CLI docs commands have better error messages and more consistent short flags
* [ENHANCEMENT] Update all Data Connectors to allow for `batch_spec_passthrough` in config
* [ENHANCEMENT] Update `DataConnector.build_batch_spec` to use `batch_spec_passthrough` in config
* [ENHANCEMENT] Update `ConfiguredAssetSqlDataConnector.build_batch_spec` and `ConfiguredAssetFilePathDataConnector.build_batch_spec` to properly process `Asset.batch_spec_passthrough`
* [ENHANCEMENT] Update `SqlAlchemyExecutionEngine.get_batch_data_and_markers` to handle `create_temp_table` in `RuntimeQueryBatchSpec`
* [ENHANCEMENT] Usage stats messages for the v3 API CLI are now sent before and after the command runs 2661
* [ENHANCEMENT} Update the datasource new notebook for improved data asset inference
* [ENHANCEMENT] Update the `datasource new` notebook for improved data asset inference
* [ENHANCEMENT] Made stylistic improvements to the `checkpoint new` notebook
* [ENHANCEMENT] Add mode prompt to suite new and suite edit 2706
* [ENHANCEMENT] Update build_gallery.py script to better-handle user-submitted Expectations failing 2705
* [ENHANCEMENT] Docs + Tests for passing in reader_options to Spark 2670
* [ENHANCEMENT] Adding progressbar to validator loop 2620 (Thanks peterdhansen!)
* [ENHANCEMENT] Great Expectations Compatibility with SqlAlchemy 1.4 2641
* [ENHANCEMENT] Athena expect column quantile values to be between 2544 (Thanks RicardoPedrotti!)
* [BUGFIX] Rename assets in SqlDataConnectors to be consistent with other DataConnectors 2665
* [BUGFIX] V3 API CLI docs build now opens all built sites rather than only the last one
* [BUGFIX] Handle limit for oracle with rownum 2691 (Thanks NathanFarmer!)
* [BUGFIX] add create table logic for athena 2668 (Thanks kj-9!)
* [BUGFIX] Add note for user-submitted Expectation that is not compatible with SqlAlchemy 1.4 (uszipcode) 2677
* [BUGFIX] Usage stats cli payload schema 2680
* [BUGFIX] Rename assets in SqlDataConnectors 2665
* [DOCS] Update how_to_create_a_new_checkpoint.rst with description of new CLI functionality
* [DOCS] Update Configuring Datasources documentation for V3 API CLI
* [DOCS] Update Configuring Data Docs documentation for V3 API CLI
* [DOCS] Update Configuring metadata stores documentation for V3 API CLI
* [DOCS] Update How to configure a Pandas/S3 Datasource for V3 API CLI
* [DOCS] Fix typos in "How to load a database table, view, or query result as a batch" guide and update with `create_temp_table` info
* [DOCS] Update "How to add a Validation Operator" guide to make it clear it is only for V2 API
* [DOCS] Update Version Migration Guide to recommend using V3 without caveats
* [DOCS] Formatting fixes for datasource docs 2686
* [DOCS] Add note about v3 API to How to use the Great Expectations command line interface (CLI) 2675
* [DOCS] CLI SUITE Documentation for V3 2687
* [DOCS] how to share data docs on azure 2589 (Thanks benoitLebreton-perso!)
* [DOCS] Fix typo in Core concepts/Key Ideas section 2660 (Thanks svenhofstede!)
* [DOCS] typo in datasource documentation 2654 (Thanks Gfeuillen!)
* [DOCS] fix grammar 2579 (Thanks carlsonp!)
* [DOCS] Typo fix in Core Concepts/ Key Ideas section 2644 (Thanks TremaMiguel!)
* [DOCS] Corrects wrong pypi package in Contrib Packages README 2653 (Thanks mielvds!)
* [DOCS] Update dividing_data_assets_into_batches.rst 2651 (Thanks lhayhurst!)
* [MAINTENANCE] Temporarily pin sqlalchemy (1.4.9) and add new CI stage 2708
* [MAINTENANCE] Run CLI tests as a separate stage in Azure pipelines 2672
* [MAINTENANCE] Updates to usage stats messages & tests for new CLI 2689
* [MAINTENANCE] Making user configurable profile test more robust; minor cleanup 2685
* [MAINTENANCE] remove cli.project.upgrade event 2682
* [MAINTENANCE] column reflection fallback should introspect one table (not all tables) 2657 (Thank you peterdhansen!)
* [MAINTENANCE] Refactor Tests to Use Common Libraries 2663

- [BREAKING-EXPERIMENTAL] The `batch_data` attribute of `BatchRequest` has been removed. To pass in in-memory dataframes at runtime, the new `RuntimeDataConnector` should be used
- [BREAKING-EXPERIMENTAL] `RuntimeDataConnector` must now be passed Batch Requests of type `RuntimeBatchRequest`
- [BREAKING-EXPERIMENTAL] The `PartitionDefinitionSubset` class has been removed - the parent class `IDDict` is used in its place
- [BREAKING-EXPERIMENTAL] `partition_request` was renamed `data_connector_query`. The related `PartitionRequest` class has been removed - the parent class `IDDict` is used in its place
- [BREAKING-EXPERIMENTAL] `partition_definition` was renamed `batch_identifiers`. The related `PartitionDefinition` class has been removed - the parent class `IDDict` is used in its place
- [BREAKING-EXPERIMENTAL] The `PartitionQuery` class has been renamed to `BatchFilter`
- [BREAKING-EXPERIMENTAL] The `batch_identifiers` key on `DataConnectorQuery` (formerly `PartitionRequest`) has been changed to `batch_filter_parameters`
- [ENHANCEMENT] Added a new `RuntimeBatchRequest` class, which can be used alongside `RuntimeDataConnector` to specify batches at runtime with either an in-memory dataframe, path (filesystem or s3), or sql query
- [ENHANCEMENT] Added a new `RuntimeQueryBatchSpec` class
- [BUGFIX] Fixed issue where Sorters were not being applied correctly when `data_connector_query` contained limit or index 2617
- [DOCS] Updated docs to reflect above class name changes
- [DOCS] Added the following docs: "How to configure sorting in Data Connectors", "How to configure a Runtime Data Connector", "How to create a Batch Request using an Active Data Connector", "How to load a database table, view, or query result as a Batch"
- [DOCS] Updated the V3 API section of the following docs: "How to load a Pandas DataFrame as a Batch", "How to load a Spark DataFrame as a Batch"

* [ENHANCEMENT] CLI `docs list` command implemented for v3 api 2612
* [MAINTENANCE] Add testing for overwrite_existing in sanitize_yaml_and_save_datasource 2613
* [ENHANCEMENT] CLI `docs build` command implemented for v3 api 2614
* [ENHANCEMENT] CLI `docs clean` command implemented for v3 api 2615
* [ENHANCEMENT] DataContext.clean_data_docs now raises helpful errors 2621
* [ENHANCEMENT] CLI `init` command implemented for v3 api 2626
* [ENHANCEMENT] CLI `store list` command implemented for v3 api 2627

* [FEATURE] Added support for references to secrets stores for AWS Secrets Manager, GCP Secret Manager and Azure Key Vault in `great_expectations.yml` project config file (Thanks Cedric-Magnan!)
* [ENHANCEMENT] Datasource CLI functionality for v3 api and global --assume-yes flag 2590
* [ENHANCEMENT] Update UserConfigurableProfiler to increase tolerance for mostly parameter of nullity expectations
* [ENHANCEMENT] Adding tqdm to Profiler (Thanks peterdhansen). New library in requirements.txt
* [ENHANCEMENT][MAINTENANCE] Use Metrics to Protect Against Wrong Column Names
* [BUGFIX] Remove parentheses call at os.curdir in data_context.py 2566 (thanks henriquejsfj)
* [BUGFIX] Sorter Configuration Added to DataConnectorConfig and DataConnectorConfigSchema 2572
* [BUGFIX] Remove autosave of Checkpoints in test_yaml_config and store SimpleCheckpoint as Checkpoint 2549
* [BUGFIX] Populate (data) asset name in data docs for SimpleSqlalchemy datasource (Thanks xaniasd)
* [BUGFIX] pandas partial read_ functions not being unwrapped (Thanks luke321321)
* [BUGFIX] Don't stop SparkContext when running in Databricks (2587) (Thanks jarandaf)