Mlrun

Features / Enhancements
* **Docs**: How to build a docker image using a dockerfile, 4692, jillnogold
* **Docs**: Update grafana model monitoring applications dashboard, 4643, jillnogold
* **Feature Store**: Improve `onehotencoder` warning, 4698, gtopper
* **Runtimes**: Use http connection pool to speed up function invocations, 4693, gtopper
* **Model Monitoring**: Use the last request for the "last updated" time, 4690, jond01
* **Datastore**: Support data store profiles by spark, 4687, alxtkr77
* **Requirments**: Bump storey to v1.6.10, 4686, assaf758
* **Data types**: Avoid using deprecated numpy data type, 4684, gtopper
* **Datastore**: Avoid saving `update_last_written` function in parquettarget attributes, 4682, gtopper
* **Serving**: Add logs to termination callback, 4675, gtopper
* **Docs**: Add google-site-verification to page head, 4679, jillnogold
* **Datastore**: Write parquet files compatible with pyspark 3.2.3, 4678, gtopper
* **Model Monitoring**: Add model monitoring controller api, 4671, Eyal-Danieli
* **FeatureStore**: Change apis to be featureset/featurevector class methods, 4664, alxtkr77
* **Dask**: Retry reconnection before bailing upon connection error, 4674, liranbg
* **UI**: [Features & enhancement](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc11#features-and-enhancements)

Bug fixes
* **Tests**: Fix spark test, 4699, alxtkr77
* **Tests**: Fix `testfeaturestore` system tests, 4697, gtopper
* **Feature Store**: Fix timestamp precision in pyspark/pandas2 compatibilty workaround, 4691, gtopper
* **Tests**: Fix system tests failures, 4683, gtopper
* **FeatureStore**: Fixing featureset.preview(), 4676, alxtkr77
* **UI**: [Bug fixes](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc11#bug-fixes)


Pull requests:
420a5419 [Docs] How to build a docker image using a dockerfile (4692)
e4982528 [Docs] Update Grafana Model Monitoring Applications dashboard (4643)
460a1cfa [Tests] Fix spark test (4699)
b3a35c94 [Feature Store] Improve `OneHotEncoder` warning (4698)
54037498 [Tests] Fix `TestFeatureStore` system tests (4697)
40621e56 [Runtimes] Use HTTP connection pool to speed up function invocations (4693)
66460cc4 [Feature Store] Fix timestamp precision in pyspark/pandas2 compatibilty workaround (4691)
371fb541 [Model Monitoring] Use the last request for the "last updated" time (4690)
85e685c0 [Datastore] Support data store profiles by spark (4687)
96dc32aa [Requirments] Bump storey to v1.6.10 (4686)
3132f29f [Data types] Avoid using deprecated numpy data type (4684)
ee866865 Revert "[Data types] Avoid using deprecated data type"
fcb997f7 [Data types] Avoid using deprecated data type
80d61454 [Tests] Fix system tests failures (4683)
2c8e1395 [Datastore] Avoid saving `update_last_written` function in ParquetTarget attributes (4682)
e19b1182 [Serving] Add logs to termination callback (4675)
266248ba [Docs] Add google-site-verification to page head (4679)
e58b9950 [Datastore] Write parquet files compatible with pyspark 3.2.3 (4678)
24904797 [FeatureStore] Fixing FeatureSet.preview() (4676)
95a93892 [Model Monitoring] Add model monitoring controller API (4671)
0ca89639 [FeatureStore] Change APIs to be FeatureSet/FeatureVector class methods (4664)
c9fc966a [Dask] Retry reconnection before bailing upon connection error (4674)

Failed parsing:
ee866865 {Gal Topper} Revert "[Data types] Avoid using deprecated data type"
fcb997f7 {Gal Topper} [Data types] Avoid using deprecated data type

Features / Enhancements
* **Requirments**: Bump storey to v1.6.8, 4669, assaf758
* **API**: Do not enrich image if build required, 4667, alonmr
* **Jupyter**: Data persistent, 4536, eliyahu77
* **Requirments**: Upgrade pyarrow to 14.0.1, 4653, assaf758
* **Config**: Enable explicit ack on nuclio 1.12.7 or newer only, 4666, gtopper
* **API**: Move automated run abortion text to `status.error`, 4663, alonmr
* **Model Monitoring**: Start from the last analyzed time in the controller, 4647, jond01
* **API**: Search run if missing in monitoring, 4660, alonmr
* **Log Collector**: Add logs to adding/removing items from state, 4654, TomerShor
* **API**: Reduce memory footprint of runs monitoring, 4656, alonmr
* **Datastore**: Add support for gcsfs datastore-profiles, 4638, tomerm-iguazio
* **API**: Normalize project parameters when storing a project, 4640, Yacouby
* **Tests**: Avoid running helper methods as tests, 4651, gtopper
* **Model Monitoring**: Calculate `current_stats` for each application and forward to writer, 4624, Eyal-Danieli
* **Datastore**: Avoid setting number of replicas in `kafkasource`, 4649, gtopper
* **UI**: [Features & enhancement](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc10#features-and-enhancements)

Bug fixes
* **Jupyter**: Fix dockerfile script copy, 4673, alonmr
* **Unknown**: Fix `testfeaturestore::test_relation_join_multi_entities[dask-true]`, 4665, gtopper
* **Feature Store**: Fix issues with default `start_time` and `end_time`, 4652, gtopper
* **Tests**: Fix `testfeaturestore` dask tests, 4655, gtopper
* **Docs**: Fix some docstring formatting issues, 4650, alonmr
* **UI**: [Bug fixes](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc10#bug-fixes)


Pull requests:
9c4002bb [Jupyter] Fix dockerfile script copy (4673)
4d35774a [Requirments] Bump storey to v1.6.8 (4669)
10a952b0 [API] Do not enrich image if build required (4667)
0f52131e [Jupyter] Data persistent (4536)
d626c275 [Requirments] Upgrade pyarrow to 14.0.1 (4653)
fb812591 [Config] Enable explicit ack on nuclio 1.12.7 or newer only (4666)
dcfe8fa0 Fix `TestFeatureStore::test_relation_join_multi_entities[dask-True]` (4665)
3c97bb3b [API] Move automated run abortion text to `status.error` (4663)
4aaf3e6d [Feature Store] Fix issues with default `start_time` and `end_time` (4652)
f84b0493 [Model Monitoring] Start from the last analyzed time in the controller (4647)
f300671f [API] Search run if missing in monitoring (4660)
7d323e2d [Log Collector] Add logs to adding/removing items from state (4654)
5f24fbf1 [API] Reduce memory footprint of runs monitoring (4656)
db64a2fd [Datastore] add support for GcsFS datastore-profiles (4638)
959fa8ba [API] Normalize project parameters when storing a project (4640)
61c07ef6 [Tests] Fix `TestFeatureStore` dask tests (4655)
7a4ff1b3 [Tests] Avoid running helper methods as tests (4651)
d9988863 [Docs] Fix some docstring formatting issues (4650)
0bf63100 [Model Monitoring] Calculate `current_stats` for each application and forward to writer (4624)
530e2527 [Datastore] Avoid setting number of replicas in `KafkaSource` (4649)

Features / Enhancements
* **Auth**: Add missing permission check when scheduling workflows, 4637, theSaarco
* **Tutorials**: Adding align_tutorials execute permissions, 4641, aviaIguazio
* **Docs**: Add s3 and kafka datastore profiles, 4594, jillnogold
* **Builder**: Add mlrun requirement when installing pyps on mlrun base image, 4623, alonmr
* **Docs**: V1.5.2 change log, 4639, jillnogold
* **UI**: [Features & enhancement](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc9#features-and-enhancements)

Bug fixes
* **Builder**: Fix validation requiring image for remote spark runtime, 4648, liranbg
* **Datastore**: Fix new kubernetes version bug in databricks runtime, 4645, tomerm-iguazio
* **Tests**: Fix system test expected string, 4646, alonmr
* **UI**: [Bug fixes](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc9#bug-fixes)


Pull requests:
4ec173d8 [Builder] Fix validation requiring image for remote spark runtime (4648)
0ee03ac6 [Datastore] Fix new Kubernetes version bug in Databricks Runtime (4645)
21b1f3ba [Auth] Add missing permission check when scheduling workflows (4637)
68b4f0bb [Tutorials] adding align_tutorials execute permissions (4641)
a28240fb [Tests] Fix system test expected string (4646)
fa570a68 [Docs] Add S3 and Kafka datastore profiles (4594)
91b458e2 [Builder] Add mlrun requirement when installing PyPs on mlrun base image (4623)
470c994d [Docs] V1.5.2 change log (4639)

Features / Enhancements
* **Docs**: Added documentation page for mlrun project bootstrapping with project_setup.py, 4581, nschenone
* **Runtimes**: Remove requirement to provide `image` in `set_function` when using `pull_at_runtime`, 4628, theSaarco
* **Pipelines**: Indicate workflow final state in kfp/remote engines and cosmetics, 4633, alonmr
* **Nuclio**: Introduce nuclio async client for mlrun api, 4610, rokatyy
* **Project**: Log when project directory has existing git remote, 4631, liranbg
* **Runtimes**: Modify logger level when handler is not provided, 4629, Yacouby
* **Model Monitoring**: Refactor the controller code, 4621, jond01
* **Tests**: Improve the model monitoring framework system test, 4632, jond01
* **Docs**: Add "state_threshold" to monitoring, 4626, jillnogold
* **Tutorials**: Update serving function image from python:3.9 to mlrun/mlrun, 4627, aviaIguazio
* **Docs**: Updates to v1.5.1 change log, 4609, jillnogold
* **Pipelines**: Upgrade kfp sdk to the latest patch release available under 1.8, 4619, laurybueno
* **Docs**: Ignore colab tutorial in search, 4618, alonmr
* **Dask**: Do not support state thresholds, 4616, alonmr
* **Feature Store**: Set unit and timezone of default start and end times, 4612, gtopper
* **UI**: [Features & enhancement](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc8#features-and-enhancements)

Bug fixes
* **Datastore**: Fix dbfs store url, 4636, tomerm-iguazio
* **Datastore**: Fix support for datastore profile for cvs/parquet sources, 4613, alxtkr77
* **Datastore**: Fix feature inference in side target, 4620, gtopper
* **Feature Store**: Fix workaround for pyspark->pandas2 timestamp conversion issue, 4635, gtopper
* **Package**: Corrections and bug fixes, 4625, guy1992l
* **UI**: [Bug fixes](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc8#bug-fixes)


Pull requests:
e82173e4 [Docs] Added documentation page for MLRun project bootstrapping with project_setup.py (4581)
c4ddb69e [Runtimes] Remove requirement to provide `image` in `set_function` when using `pull_at_runtime` (4628)
476e3bdd [Datastore] Fix DBFS store URL (4636)
e9527cec [Datastore] Fix support for datastore profile for CVS/Parquet sources (4613)
4a070d5a [Pipelines] Indicate workflow final state in kfp/remote engines and cosmetics (4633)
7e782e3c [Nuclio] Introduce nuclio async client for mlrun api (4610)
d6e2495d [Project] Log when project directory has existing git remote (4631)
34dc3019 [Runtimes] Modify logger level when handler is not provided (4629)
7e87ffc5 [Datastore] Fix feature inference in side target (4620)
8539bed7 [Model Monitoring] Refactor the controller code (4621)
02c13740 [Feature Store] Fix workaround for pyspark->pandas2 timestamp conversion issue (4635)
fa547110 [Tests] Improve the model monitoring framework system test (4632)
7fbdcf71 [Docs] Add "state_threshold" to monitoring (4626)
140a9be6 [Tutorials] update serving function image from python:3.9 to mlrun/mlrun (4627)
7d1cca24 [Package] Corrections and Bug Fixes (4625)
d174bcb8 [Docs] Updates to V1.5.1 change log (4609)
c92ca752 [Pipelines] Upgrade KFP SDK to the latest patch release available under 1.8 (4619)
10232c51 [Docs] Ignore colab tutorial in search (4618)
4c6c711a [Dask] Do not support state thresholds (4616)
587af26d [Feature Store] Set unit and timezone of default start and end times (4612)

Features / Enhancements
* **Doc**: Add datastore profile module, 4607, alxtkr77
* **Datastore**: Partially revert 4582, 4605, gtopper
* **Docs**: Remote pipeline notifications updates, 4585, quaark
* **Nuclio**: Correct the check for the existence of the http trigger, 4598, rokatyy
* **Model Monitoring**: Sync the controller range with the stream pod flush interval, 4590, jond01
* **Requirements**: Temporarily blacklist aiohttp 3.8.4, 4602, alonmr
* **Docs**: Add using single step and create context, 4597, jillnogold
* **Docs**: Bump sphinx and myst-nb, 4593, alonmr
* **CI**: Disable github action color for unit tests job, 4595, alonmr
* **Images**: Use libmamba solver for mlrun image, 4592, alonmr
* **Runtimes**: Add set categories method to runtimes, 4589, Yacouby
* **Databricks**: Accept `state_threshold` param, 4591, alonmr
* **Docs**: Update change log with open issues and limitations, 4577, jillnogold
* **Tests**: Move databricks env config to env template, 4583, alonmr
* **Requirments**: Bump storey to v1.6.6, 4584, assaf758
* **Datastore**: Support dbfs datastore profiles, 4579, tomerm-iguazio
* **Nuclio**: Allow not creating an http trigger for nuclio functions by default, 4552, rokatyy
* **API**: Change minimum state threshold to 1 minute, 4578, alonmr
* **Notifications**: Remote pipeline notifications, 4567, quaark
* **UI**: [Features & enhancement](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc7#features-and-enhancements)

Bug fixes
* **Storey**: Fix test for aggregations returning nan, 4608, alxtkr77
* **Datastore**: Fixing datastore profile target path resolution in csv and parquet targets, 4601, alxtkr77
* **CLI**: Fix default image overrides runtime, 4604, alonmr
* **Workflows**: Fix workflow request notifications read, 4606, alonmr
* **Runtimes**: Fixed missing `force_build` in `deploy()` for spark and serving runtimes, 4599, theSaarco
* **Model Monitoring**: Fix labels when calling `list_model_monitoring_functions`, 4600, Eyal-Danieli
* **Unknown**: Revert "[datastore] fixing datastore profile target path resolution in csv and parquet targets (4586)", 4596, assaf758
* **Datastore**: Fixing datastore profile target path resolution in csv and parquet targets, 4586, alxtkr77
* **Secrets**: Fix v3io access-key that remained in cleartext when creating function through ui, 4587, theSaarco
* **Requirements**: Fix apple silicon conda dependencies, 4580, jond01
* **Feature Store**: Fix `csvtarget` in ingestion service, fix feature order in target with `after_step`, 4582, gtopper
* **Builder**: Fix resolving secret name for using mlrun default registry, 4575, liranbg
* **Dask**: Fix dask cli command for old mlrun clients, 4576, alonmr
* **UI**: [Bug fixes](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc7#bug-fixes)


Pull requests:
93faebe6 [Storey] Fix test for aggregations returning NaN (4608)
7e39b41b [Doc] Add datastore profile module (4607)
550bc6cc [Datastore] Fixing datastore profile target path resolution in CSV and Parquet targets (4601)
b8cb854b [CLI] Fix default image overrides runtime (4604)
fc7dd6cd [Workflows] Fix workflow request notifications read (4606)
e8df5bd2 [Datastore] Partially revert 4582 (4605)
d909ade0 [Docs] Remote Pipeline Notifications Updates (4585)
014d0521 [Nuclio] Correct the check for the existence of the HTTP trigger (4598)
7d2c198f [Model Monitoring] Sync the controller range with the stream pod flush interval (4590)
ec850049 [Requirements] Temporarily blacklist aiohttp 3.8.4 (4602)
53874c36 [Runtimes] Fixed missing `force_build` in `deploy()` for spark and serving runtimes (4599)
b56b3d8f [Docs] Add using single step and create context (4597)
cf824ca2 [Docs] Bump sphinx and myst-nb (4593)
ae330ccf [CI] Disable github action color for unit tests job (4595)
a43b3ea0 [Model Monitoring] Fix labels when calling `list_model_monitoring_functions` (4600)
c0c8a07c Revert "[Datastore] Fixing datastore profile target path resolution in CSV and Parquet targets (4586)" (4596)
4546dc81 [Datastore] Fixing datastore profile target path resolution in CSV and Parquet targets (4586)
9846d0f0 [Images] Use libmamba solver for mlrun image (4592)
aea26681 [Runtimes] Add set categories method to runtimes (4589)
353a6cd8 [Databricks] Accept `state_threshold` param (4591)
6ebca8bb [Secrets] Fix v3io access-key that remained in cleartext when creating function through UI (4587)
a29c1a93 [Docs] Update change log with open issues and limitations (4577)
6043953e [Tests] Move Databricks env config to env template (4583)
cd776370 [Requirments] bump storey to v1.6.6 (4584)
2c219804 [Datastore] Support Dbfs datastore profiles (4579)
a0396e85 [Nuclio] Allow not creating an http trigger for Nuclio functions by default (4552)
0ab7bc65 [Requirements] Fix Apple silicon conda dependencies (4580)
81888c19 [Feature Store] Fix `CSVTarget` in ingestion service, fix feature order in target with `after_step` (4582)
9d09e4ea [API] Change minimum state threshold to 1 minute (4578)
47b8c733 [Notifications] Remote Pipeline Notifications (4567)
14acb754 [Builder] Fix resolving secret name for using mlrun default registry (4575)
90006a5e [Dask] Fix dask CLI command for old mlrun clients (4576)

Features / Enhancements
* **Config**: Refactor runs monitoring structure, 4572, alonmr
* **Pipelines**: Clean up kubeflow specific data from function definitions after workflow execution, 4547, laurybueno
* **CLI**: Run - when image is not provided, use mlrun/mlrun by default, 4563, Yacouby
* **API**: Add mpijob and spark state threshold monitoring, 4562, alonmr
* **Docs**: Improve deletionstrategy description, 4569, jillnogold
* **Docs**: Add warning banner to old versions that this is not the latest version, 4558, jillnogold
* **Docs**: Add jars via spark conf, 4561, jillnogold
* **Server**: Avoid setting workdir to target dir, 4560, gtopper
* **Model Monitoring**: Create tsdb schema for batch infer, 4546, jond01
* **UI**: [Features & enhancement](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc6#features-and-enhancements)

Bug fixes
* **LogCollector**: Fix big stop logs, 4555, liranbg
* **Model Monitoring**: Fix the labels aggregation in `list_model_monitoring_functions`, 4573, jond01
* **Datastore**: Fix s3 bugs introduced in previous pr, 4570, alxtkr77
* **Datastore**: Fix support for 'credential prefix', 4568, alxtkr77
* **Model Monitoring**: Fix monitoring functions sdk usage, 4565, jond01
* **Datastore**: Fix error in redis ds profile handling, 4559, alxtkr77
* **Tests**: Fix expected system tests strings, 4557, alonmr
* **HTTPDB**: Fix abort run default project, 4556, alonmr
* **Tests**: Fix databricks tests, 4554, tomerm-iguazio
* **UI**: [Bug fixes](https://github.com/mlrun/ui/releases/tag/v1.6.0-rc6#bug-fixes)


Pull requests:
ec8b7141 [Config] Refactor runs monitoring structure (4572)
ce281124 [Pipelines] Clean up Kubeflow specific data from Function definitions after workflow execution (4547)
897c2e5d [LogCollector] Fix big stop logs (4555)
1f7d4c23 [CLI] Run - when image is not provided, use mlrun/mlrun by default (4563)
8e073efe [API] Add mpijob and spark state threshold monitoring (4562)
16c4983a [Model Monitoring] Fix the labels aggregation in `list_model_monitoring_functions` (4573)
947d19a2 [Datastore] Fix s3 bugs introduced in previous PR (4570)
176b8f56 [Datastore] Fix support for 'credential prefix' (4568)
45b985b3 [Docs] Improve Deletionstrategy description (4569)
f981e7cc [Docs] Add warning banner to old versions that this is not the latest version (4558)
d1011657 [Docs] Add jars via spark conf (4561)
b1ca92a1 [Model Monitoring] Fix monitoring functions SDK usage (4565)
afe7b105 [Datastore] Fix error in Redis DS profile handling (4559)
9eeed2a6 [Server] Avoid setting workdir to target dir (4560)
178b68a2 [Model Monitoring] Create TSDB schema for batch infer (4546)
51a0c93b [Tests] Fix expected system tests strings (4557)
7d4753b7 [HTTPDB] Fix abort run default project (4556)
8e296637 [Tests] Fix databricks tests (4554)