Hopsworks

[HWORKS-909](https://hopsworks.atlassian.net/browse/HWORKS-909) Add hopsworks-api opensearch and kafka workflow tests

[HWORKS-912](https://hopsworks.atlassian.net/browse/HWORKS-912) Upload staging dir does not need to be configurable.

[HWORKS-919](https://hopsworks.atlassian.net/browse/HWORKS-919) Create database on the online feature store on-demand

[HWORKS-934](https://hopsworks.atlassian.net/browse/HWORKS-934) Use more appropriate data types for statistics

Bug

[HOPSWORKS-2359](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2359) Always show GPU configuration fields for Jobs and Jupyter
[HOPSWORKS-2358](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2358) Remove ROCM GPU monitoring code
[HOPSWORKS-2346](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2346) Identifying if a project is using the base image on ECR does not work
[HOPSWORKS-2338](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2338) handle empty tfrecord files in TFDataEngine
[HOPSWORKS-2333](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2333) create\_training\_dataset method throws Connector not found exception
[HOPSWORKS-2311](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2311) Template required ec2-init parameters for the unmanaged version
[HOPSWORKS-2307](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2307) Docker garbage collection should not remove base or python37 images for any hopsworks version
[HOPSWORKS-2302](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2302) Hide python pip search bar as underlying pip search is now deprecated
[HOPSWORKS-2294](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2294) Installing a python library from git should enclose the url in quotes
[HOPSWORKS-2283](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2283) Reading from the online feature store doesn't work on Databricks
[HOPSWORKS-2249](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2249) provenance elasticsearch - count and aggregations should be separate methods
[HOPSWORKS-2248](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2248) Python file path is not shown in Job details
[HOPSWORKS-2238](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2238) Feature Group schema primary key info not set for partition keys
[HOPSWORKS-2219](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2219) Remove NaN from statistics
[HOPSWORKS-2202](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2202) Docker registry caches deleted manifests
[HOPSWORKS-2199](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2199) Glassfish consul domain should be included in Kafka super.users
[HOPSWORKS-2198](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2198) Cannot get the schema of a shared kafka topic
[HOPSWORKS-2187](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2187) Add anaconda user to hadoop group only if the group exists. Create hops-system Anaconda env conditionally
[HOPSWORKS-2182](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2182) Disable unattended-upgrades in Ubuntu
[HOPSWORKS-2174](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2174) issues attaching xattr to shared files
[HOPSWORKS-2164](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2164) Add API Key annotation on the commits endpoint
[HOPSWORKS-2159](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2159) ruby tests fix
[HOPSWORKS-2158](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2158) Query constructor ignores more than 2 joining conditions
[HOPSWORKS-2151](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2151) Tag validation doesn't work for key only tags
[HOPSWORKS-2145](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2145) Flask \(sklearn\) race condition when reading pid file
[HOPSWORKS-2102](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2102) Conversion of Jupyter notebooks to Python programs should consider notebook kernel
[HOPSWORKS-1984](https://logicalclocks.atlassian.net/browse/HOPSWORKS-1984) Exception when creating online feature group without primary key

Task

[HOPSWORKS-2348](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2348) Disable storage and compute quota by default
[HOPSWORKS-2337](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2337) Upgrade hudi to 0.7.0 version
[HOPSWORKS-2327](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2327) Ui fixed for generating training datasets
[HOPSWORKS-2324](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2324) Display alert if project is using older docker image
[HOPSWORKS-2320](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2320) Set PARQUET as default for non-hudi offline feature groups
[HOPSWORKS-2287](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2287) Bump up [docker.io](http://docker.io) version for ubuntu
[HOPSWORKS-2275](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2275) Remove InodeEncodingStatus Bean
[HOPSWORKS-2274](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2274) Feature store "NOT FOUND" errors should return 404
[HOPSWORKS-2241](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2241) Consul security recipe should not try to get Hopsworks IP if not needed
[HOPSWORKS-2206](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2206) HSFS profile to install with and without Hive dependencies
[HOPSWORKS-2196](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2196) Remove hudiEnabled flag and sync endpoint from hopsworks
[HOPSWORKS-2189](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2189) Update spark-avro package for to\_avro/from\_avro methods
[HOPSWORKS-2188](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2188) Remove partition key requirements for HUDI feature groups and allow users to set precombine keys
[HOPSWORKS-2178](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2178) Remove leftover ldconfig in hops::install
[HOPSWORKS-2163](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2163) Add missing chef changes of HOPSWORKS-2057 in master

[HWORKS-897](https://hopsworks.atlassian.net/browse/HWORKS-897) Upgrade project python environment libraries

Task

[HOPSWORKS-2287](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2287) Bump up [docker.io](http://docker.io) version for ubuntu
[HOPSWORKS-2186](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2186) Remove installation of slack operator in Airflow cookbook
[HOPSWORKS-2161](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2161) Option to disable just the Registration UI
[HOPSWORKS-2156](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2156) Pin all TensorFlow projects in Docker image
[HOPSWORKS-2126](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2126) Cleanup old hops-util\(-py\) feature store notebooks

[FSTORE-891](https://hopsworks.atlassian.net/browse/FSTORE-891) SQL and expat migrations for Statistics

[FSTORE-899](https://hopsworks.atlassian.net/browse/FSTORE-899) Capability Write Up - Feature Engineering in Flink

[FSTORE-903](https://hopsworks.atlassian.net/browse/FSTORE-903) External Warehouse Access Flyingduck

[FSTORE-904](https://hopsworks.atlassian.net/browse/FSTORE-904) Flying Duck - Write Blog Article

[FSTORE-908](https://hopsworks.atlassian.net/browse/FSTORE-908) Upgrade scheduler to near real-time standard and cron frequencies

[FSTORE-926](https://hopsworks.atlassian.net/browse/FSTORE-926) Hopsworks BigQuery Storage Connector can't create and write data to external FG from Dataproc cluster

[FSTORE-938](https://hopsworks.atlassian.net/browse/FSTORE-938) Remove enabled flag on Feature Monitoring Configuration to only use scheduler flag

[FSTORE-939](https://hopsworks.atlassian.net/browse/FSTORE-939) Change name of Stream Feature Group's DeltaStreamer Job

[FSTORE-940](https://hopsworks.atlassian.net/browse/FSTORE-940) Constraints for job and scheduler deletion

[FSTORE-947](https://hopsworks.atlassian.net/browse/FSTORE-947) Tutorials Streamlit Fix

[FSTORE-948](https://hopsworks.atlassian.net/browse/FSTORE-948) Add ASOF Joins to Hopsworks Query Constructor

[FSTORE-952](https://hopsworks.atlassian.net/browse/FSTORE-952) Single Kafka topic per project

[FSTORE-956](https://hopsworks.atlassian.net/browse/FSTORE-956) FS REST API logging and metric

[FSTORE-958](https://hopsworks.atlassian.net/browse/FSTORE-958) Rename and split tutorial notebooks as feature, training and inference pipelines

[FSTORE-961](https://hopsworks.atlassian.net/browse/FSTORE-961) tutorial for bytewax / hsfs integration

[FSTORE-964](https://hopsworks.atlassian.net/browse/FSTORE-964) Exclude extra data from get all feature groups api

[FSTORE-970](https://hopsworks.atlassian.net/browse/FSTORE-970) Include schedule in the job DTO

[FSTORE-977](https://hopsworks.atlassian.net/browse/FSTORE-977) FlyingDuck - Slides Pydata

[FSTORE-978](https://hopsworks.atlassian.net/browse/FSTORE-978) Hopsworks recommender system tutorial

[FSTORE-979](https://hopsworks.atlassian.net/browse/FSTORE-979) Flink hsfs client workflow test

[FSTORE-981](https://hopsworks.atlassian.net/browse/FSTORE-981) Make clear that from python env only stream=True is supported

[FSTORE-982](https://hopsworks.atlassian.net/browse/FSTORE-982) Adopt serving key in feature store Rest API

[FSTORE-984](https://hopsworks.atlassian.net/browse/FSTORE-984) FlyingDuck BigQuery Connector

[FSTORE-988](https://hopsworks.atlassian.net/browse/FSTORE-988) Add PIT join tests to workflow/load tests

[FSTORE-994](https://hopsworks.atlassian.net/browse/FSTORE-994) Use non-ascii characters in user first and last name in nightly tests

[FSTORE-995](https://hopsworks.atlassian.net/browse/FSTORE-995) Return feature value in get\_feature\_vectors when entry is str instead of int

[FSTORE-996](https://hopsworks.atlassian.net/browse/FSTORE-996) FS REST API Alert

[FSTORE-1010](https://hopsworks.atlassian.net/browse/FSTORE-1010) Don't leave orphaned subjects when deleting online enabled fg

[FSTORE-1013](https://hopsworks.atlassian.net/browse/FSTORE-1013) Hopsworks-api allow to specify feature\_store\_topic when creating project

[FSTORE-1014](https://hopsworks.atlassian.net/browse/FSTORE-1014) \[Online FS REST API\] Failed to return features when only primary key is selected

[FSTORE-1020](https://hopsworks.atlassian.net/browse/FSTORE-1020) Unable to delete a FeatureGroup with a FeatureView with filters

[FSTORE-1027](https://hopsworks.atlassian.net/browse/FSTORE-1027) FS docs fix

[FSTORE-1028](https://hopsworks.atlassian.net/browse/FSTORE-1028) POC for prospect customer

[FSTORE-1031](https://hopsworks.atlassian.net/browse/FSTORE-1031) Update Colab links in docs

[HOPSWORKS-2621](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2621) Bump pip in hops-system to last version that supports python 2.7

[HOPSWORKS-2601](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2601) Update swagger code to be compatible with the latest swagger hub and enable for demo cluster

[HOPSWORKS-2498](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2498) Remove filebeat beam recipes

[HOPSWORKS-2424](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2424) Attach transformation function to training dataset

[HOPSWORKS-2420](https://logicalclocks.atlassian.net/browse/HOPSWORKS-2420) Bump hops dependencies to 3.2.0.3-SNAPSHOT

[HOPSWORKS-1942](https://logicalclocks.atlassian.net/browse/HOPSWORKS-1942) Upgrade to Spark 3.x