Bug
[HWORKS-224](https://hopsworks.atlassian.net/browse/HWORKS-224) hopsworks python SDK opensearch link broken
[HWORKS-267](https://hopsworks.atlassian.net/browse/HWORKS-267) After fail checkout, branch is empty on pull
[HWORKS-269](https://hopsworks.atlassian.net/browse/HWORKS-269) Don't show the same in file status and in UI table
[HWORKS-309](https://hopsworks.atlassian.net/browse/HWORKS-309) Bugs in searching
[HWORKS-331](https://hopsworks.atlassian.net/browse/HWORKS-331) Hopsworks metrics not showing after payara restart
[HWORKS-374](https://hopsworks.atlassian.net/browse/HWORKS-374) opensearch\_api.get\_default\_py\_config\(\) returns public IP of host, should return consul fqdn
[HWORKS-386](https://hopsworks.atlassian.net/browse/HWORKS-386) Spark job does not accept space in arguments
[HWORKS-469](https://hopsworks.atlassian.net/browse/HWORKS-469) Dependent conda commands not handled correctly in case of failure
[HWORKS-475](https://hopsworks.atlassian.net/browse/HWORKS-475) hopsworks Python library can't connect in sklearn legacy deployments
[HWORKS-492](https://hopsworks.atlassian.net/browse/HWORKS-492) Servings logs are not shown in Kibana
[HWORKS-493](https://hopsworks.atlassian.net/browse/HWORKS-493) Request batching configuration is broken in the deployment creation form
[HWORKS-496](https://hopsworks.atlassian.net/browse/HWORKS-496) Exception not raised if other than duplicated deployment entry
[HWORKS-498](https://hopsworks.atlassian.net/browse/HWORKS-498) Transformer resources in dict format are not properly deserialized
[HWORKS-499](https://hopsworks.atlassian.net/browse/HWORKS-499) Support for jupyter notebooks as predictor scripts
[HWORKS-561](https://hopsworks.atlassian.net/browse/HWORKS-561) Hopsworks Kafka Authorizer does not allow IDEMPOTENT\_WRITE operations
[HWORKS-569](https://hopsworks.atlassian.net/browse/HWORKS-569) Filebeat cannot list nodes which is required for scraping container logs
[HWORKS-602](https://hopsworks.atlassian.net/browse/HWORKS-602) HA cluster could not connect to live logs, rm UI, spark UI, monitor, onlineFS
[HWORKS-630](https://hopsworks.atlassian.net/browse/HWORKS-630) Project creation doesn't fail on invalid project name
[HWORKS-633](https://hopsworks.atlassian.net/browse/HWORKS-633) Failing deployments are shown as Starting when Kubernetes stops restarting pods
[HWORKS-636](https://hopsworks.atlassian.net/browse/HWORKS-636) Inference logging default value is overwritten when updating a deployment using .save\(\)
[HWORKS-656](https://hopsworks.atlassian.net/browse/HWORKS-656) NP trying to force delete a partially created project
[HWORKS-693](https://hopsworks.atlassian.net/browse/HWORKS-693) Server logs not collected when the predictor class fails to initialize
[HWORKS-786](https://hopsworks.atlassian.net/browse/HWORKS-786) Storage connector specs failure for flag disabled case
[HWORKS-787](https://hopsworks.atlassian.net/browse/HWORKS-787) Git repository current commit and branch not being updated
[HWORKS-792](https://hopsworks.atlassian.net/browse/HWORKS-792) Hopsworks job schedule update doesn't update cron expression
[HWORKS-809](https://hopsworks.atlassian.net/browse/HWORKS-809) Project creation fails when the serving api key k8s secret takes too long to be available
[HWORKS-815](https://hopsworks.atlassian.net/browse/HWORKS-815) Judge service should be restarted when docker daemon restarts
[HWORKS-817](https://hopsworks.atlassian.net/browse/HWORKS-817) Log level in RonDB replication scripts is wrong
[HWORKS-819](https://hopsworks.atlassian.net/browse/HWORKS-819) The request URI port is absent in MultiRegionFilter
[HWORKS-820](https://hopsworks.atlassian.net/browse/HWORKS-820) In MultiRegionFilter the Primary region is null if invoked too early
[HWORKS-821](https://hopsworks.atlassian.net/browse/HWORKS-821) Kubernetes does not accept serving names starting with a number
[HWORKS-824](https://hopsworks.atlassian.net/browse/HWORKS-824) Only Primary MySQLs should be able to update RonDB replication metadata table
[HWORKS-833](https://hopsworks.atlassian.net/browse/HWORKS-833) Consul datacenter is hardcoded in Prometheus configuration
[HWORKS-834](https://hopsworks.atlassian.net/browse/HWORKS-834) Undeploy hopsworks on upgrade
[HWORKS-837](https://hopsworks.atlassian.net/browse/HWORKS-837) RonDB native backup script fails when there is no database to backup
[HWORKS-848](https://hopsworks.atlassian.net/browse/HWORKS-848) AlertManager EJB does not take Datacenter into consideration in Multiregion clusters
[HWORKS-851](https://hopsworks.atlassian.net/browse/HWORKS-851) Check in hops-hadoop-chef for creating RonDB undo and log files is wrong
[HWORKS-865](https://hopsworks.atlassian.net/browse/HWORKS-865) NullPointerException when monitoring execution of a deleted job
Task
[HWORKS-135](https://hopsworks.atlassian.net/browse/HWORKS-135) Models backend should store metadata in tables instead of opensearch
[HWORKS-164](https://hopsworks.atlassian.net/browse/HWORKS-164) Add Airflow documentation
[HWORKS-198](https://hopsworks.atlassian.net/browse/HWORKS-198) Remove hopsworks::image recipe
[HWORKS-226](https://hopsworks.atlassian.net/browse/HWORKS-226) python api should throw informative error if user tries to download a folder
[HWORKS-258](https://hopsworks.atlassian.net/browse/HWORKS-258) Add hopsworks python sdk dataset upload/download to workflow testing
[HWORKS-262](https://hopsworks.atlassian.net/browse/HWORKS-262) Remove user email FK from hopsworks schema.
[HWORKS-266](https://hopsworks.atlassian.net/browse/HWORKS-266) Checking out a branch that exist on remote repo does not work
[HWORKS-284](https://hopsworks.atlassian.net/browse/HWORKS-284) Documentation to export cluster logs
[HWORKS-302](https://hopsworks.atlassian.net/browse/HWORKS-302) Enhancement Request: Ability to disable access to Anaconda package repository and hide the functionality at a system defined level
[HWORKS-314](https://hopsworks.atlassian.net/browse/HWORKS-314) Add support for git fetch and reset
[HWORKS-422](https://hopsworks.atlassian.net/browse/HWORKS-422) Exclude old versions of the documentation from being indexed by search engines
[HWORKS-491](https://hopsworks.atlassian.net/browse/HWORKS-491) Replace hsml with hopsworks in the code snippets shown in the UI
[HWORKS-494](https://hopsworks.atlassian.net/browse/HWORKS-494) Support local script path in model.deploy\(\)
[HWORKS-511](https://hopsworks.atlassian.net/browse/HWORKS-511) Add number of GPUs to Python resource usage
[HWORKS-520](https://hopsworks.atlassian.net/browse/HWORKS-520) Make oauth claims configurable
[HWORKS-536](https://hopsworks.atlassian.net/browse/HWORKS-536) Move master encryption password to KMS
[HWORKS-559](https://hopsworks.atlassian.net/browse/HWORKS-559) Users should be able to specify a directory when using the Hopsworks-api to download logs
[HWORKS-565](https://hopsworks.atlassian.net/browse/HWORKS-565) Hopsworks api dataset upload should support parallel chunk upload
[HWORKS-571](https://hopsworks.atlassian.net/browse/HWORKS-571) Run more integration tests in parallel
[HWORKS-590](https://hopsworks.atlassian.net/browse/HWORKS-590) Make jenkins test use HA cluster
[HWORKS-591](https://hopsworks.atlassian.net/browse/HWORKS-591) Show a better log for when pods are killed by Kubernetes with OOM
[HWORKS-617](https://hopsworks.atlassian.net/browse/HWORKS-617) Make load balancer use https
[HWORKS-631](https://hopsworks.atlassian.net/browse/HWORKS-631) Support info exchange between preprocess and postprocess methods in transformers
[HWORKS-635](https://hopsworks.atlassian.net/browse/HWORKS-635) Add unit tests and loadtest for hsml deployments
[HWORKS-641](https://hopsworks.atlassian.net/browse/HWORKS-641) Add loadtests for hsml and model registry
[HWORKS-662](https://hopsworks.atlassian.net/browse/HWORKS-662) Fix versioning and build automation for hopsify
[HWORKS-676](https://hopsworks.atlassian.net/browse/HWORKS-676) Supress CVE-2023-33265 in payara-embedded-web-5.2022.5.jar/META-INF/maven/com.hazelcast/hazelcast/pom.xml:
[HWORKS-677](https://hopsworks.atlassian.net/browse/HWORKS-677) Supress multiple severe CVE in hive-storage-api-2.6.1.2.jar
[HWORKS-679](https://hopsworks.atlassian.net/browse/HWORKS-679) Add option to set python environment
[HWORKS-690](https://hopsworks.atlassian.net/browse/HWORKS-690) Airflow project dags sharing
[HWORKS-697](https://hopsworks.atlassian.net/browse/HWORKS-697) Remove airflow JWT
[HWORKS-707](https://hopsworks.atlassian.net/browse/HWORKS-707) Separate statistics from feature monitoring PR
[HWORKS-729](https://hopsworks.atlassian.net/browse/HWORKS-729) Nightly loadtest cluster should have an admin user that is member of all projects
[HWORKS-745](https://hopsworks.atlassian.net/browse/HWORKS-745) Support python 3.11 for the client APIs
[HWORKS-746](https://hopsworks.atlassian.net/browse/HWORKS-746) Documentation for python environment history and custom commands
[HWORKS-751](https://hopsworks.atlassian.net/browse/HWORKS-751) Add webhook support for Hopsworks alerts
[HWORKS-756](https://hopsworks.atlassian.net/browse/HWORKS-756) Remove cmake dependency in hops-hadoop-chef
[HWORKS-759](https://hopsworks.atlassian.net/browse/HWORKS-759) Show custom commands file that was used to build the environment in the environment history
[HWORKS-764](https://hopsworks.atlassian.net/browse/HWORKS-764) Reduce JWT lifetime in nightly tests
[HWORKS-774](https://hopsworks.atlassian.net/browse/HWORKS-774) Set hopsworks\_rest\_log\_level to DEV in the nightly tests
[HWORKS-775](https://hopsworks.atlassian.net/browse/HWORKS-775) Nightly tests should reload random module
[HWORKS-777](https://hopsworks.atlassian.net/browse/HWORKS-777) Fix integration test
[HWORKS-780](https://hopsworks.atlassian.net/browse/HWORKS-780) Increase number of concurrent load tests
[HWORKS-783](https://hopsworks.atlassian.net/browse/HWORKS-783) Cleanup PIA functionality
[HWORKS-784](https://hopsworks.atlassian.net/browse/HWORKS-784) Remove airflow jdbc metric from payara
[HWORKS-789](https://hopsworks.atlassian.net/browse/HWORKS-789) Fix git tests
[HWORKS-791](https://hopsworks.atlassian.net/browse/HWORKS-791) Add Python API support to schedule jobs
[HWORKS-795](https://hopsworks.atlassian.net/browse/HWORKS-795) Hopsworks CA should generate Java keystores instead of a shell script on the client
[HWORKS-798](https://hopsworks.atlassian.net/browse/HWORKS-798) Add \*\*kwargs to python client libraries
[HWORKS-800](https://hopsworks.atlassian.net/browse/HWORKS-800) Implement dataset.copy and dataset.move API
[HWORKS-801](https://hopsworks.atlassian.net/browse/HWORKS-801) Export onlinefs/default/private\_ips in metadata
[HWORKS-803](https://hopsworks.atlassian.net/browse/HWORKS-803) Remove airflow from old UI
[HWORKS-804](https://hopsworks.atlassian.net/browse/HWORKS-804) upgrade zookeeper 3.8.2 to 3.8.3 CVE-2023-44981
[HWORKS-805](https://hopsworks.atlassian.net/browse/HWORKS-805) Environment history compares versions lexicographically instead of numerically per component
[HWORKS-807](https://hopsworks.atlassian.net/browse/HWORKS-807) Add wait\_until\_finished method to execution object in the hopsworks python apis
[HWORKS-829](https://hopsworks.atlassian.net/browse/HWORKS-829) Register model files without compressing them
[HWORKS-830](https://hopsworks.atlassian.net/browse/HWORKS-830) Configurable glassfish tmpdir property
[HWORKS-832](https://hopsworks.atlassian.net/browse/HWORKS-832) Set size limit in Consul log files
[HWORKS-835](https://hopsworks.atlassian.net/browse/HWORKS-835) Download model artifact without compressing model files
[HWORKS-841](https://hopsworks.atlassian.net/browse/HWORKS-841) Add 'See resource usage' button in the deployment overview page
[HWORKS-847](https://hopsworks.atlassian.net/browse/HWORKS-847) RonDB Global replication alert
[HWORKS-849](https://hopsworks.atlassian.net/browse/HWORKS-849) Check if dag has access\_control is not None before syncing dag permissions in airflow
[HWORKS-854](https://hopsworks.atlassian.net/browse/HWORKS-854) hops-system should use the minimal env definition without pydoop
Feature Store
Epic
[FSTORE-473](https://hopsworks.atlassian.net/browse/FSTORE-473) Data Management
[FSTORE-612](https://hopsworks.atlassian.net/browse/FSTORE-612) Feature Monitoring
[FSTORE-1047](https://hopsworks.atlassian.net/browse/FSTORE-1047) Support Similarity Search in the Feature Store
Bug
[FSTORE-55](https://hopsworks.atlassian.net/browse/FSTORE-55) Backfill job gets stuck
[FSTORE-537](https://hopsworks.atlassian.net/browse/FSTORE-537) Explicit provenance ClassCastException thrown
[FSTORE-754](https://hopsworks.atlassian.net/browse/FSTORE-754) Feature Group creation during RonDB reconfig leaves behind broken feature group
[FSTORE-755](https://hopsworks.atlassian.net/browse/FSTORE-755) Uninformative error when deleting feature group during RonDB reconfig
[FSTORE-799](https://hopsworks.atlassian.net/browse/FSTORE-799) HSFS should reinstantiate sqlalchemy connectrion pool if MySQL node disappears
[FSTORE-813](https://hopsworks.atlassian.net/browse/FSTORE-813) Unclear error when the feature group contains duplicated columns
[FSTORE-830](https://hopsworks.atlassian.net/browse/FSTORE-830) Error fetching feature statistics from feature view UI - but statistics exists with a different timestamp
[FSTORE-856](https://hopsworks.atlassian.net/browse/FSTORE-856) Only one OnlineFS instance running
[FSTORE-892](https://hopsworks.atlassian.net/browse/FSTORE-892) FM jobs are not removed when the corresponding FM configuration is deleted
[FSTORE-916](https://hopsworks.atlassian.net/browse/FSTORE-916) FeatureGroup.insert does not retry in case of a failure
[FSTORE-932](https://hopsworks.atlassian.net/browse/FSTORE-932) Pyarrow dependency is required even if the python profile is not installed
[FSTORE-943](https://hopsworks.atlassian.net/browse/FSTORE-943) NullPointerException in SchematizedTagHelper
[FSTORE-944](https://hopsworks.atlassian.net/browse/FSTORE-944) Nested filter statements are not handled correctly when attached to a feature view
[FSTORE-968](https://hopsworks.atlassian.net/browse/FSTORE-968) Add workflow test for get\_batch\_data
[FSTORE-976](https://hopsworks.atlassian.net/browse/FSTORE-976) Feature group filter in PIT join gets pushed down on the temporary table that doesn't contain the feature group
[FSTORE-987](https://hopsworks.atlassian.net/browse/FSTORE-987) Failed to read data when there is a self-join
[FSTORE-989](https://hopsworks.atlassian.net/browse/FSTORE-989) GCS connector-Encryption fields and secrets update issues
[FSTORE-992](https://hopsworks.atlassian.net/browse/FSTORE-992) create\_train\_validation\_test\_split fails with unexpected keyword argument 'pit\_query\_asof'
[FSTORE-998](https://hopsworks.atlassian.net/browse/FSTORE-998) Can't read from a shared feature store
[FSTORE-1000](https://hopsworks.atlassian.net/browse/FSTORE-1000) Tags parameter is missing in TrainingDataset class
[FSTORE-1005](https://hopsworks.atlassian.net/browse/FSTORE-1005) Training datasets are written and read using a wrong path
[FSTORE-1009](https://hopsworks.atlassian.net/browse/FSTORE-1009) hsfs flink consumer doesn't work with parallelism greater than 1.
[FSTORE-1011](https://hopsworks.atlassian.net/browse/FSTORE-1011) Validation message during data ingestion to FG when there are no kafka topics configured
[FSTORE-1017](https://hopsworks.atlassian.net/browse/FSTORE-1017) Updating job schedule's cron expression doesn't work
[FSTORE-1023](https://hopsworks.atlassian.net/browse/FSTORE-1023) Mix up of subjectId and schemaId
[FSTORE-1024](https://hopsworks.atlassian.net/browse/FSTORE-1024) Concurrency can cause SQLIntegrityConstraintViolationException when creating topic
[FSTORE-1026](https://hopsworks.atlassian.net/browse/FSTORE-1026) Wrong td file path created by ArrowFlight
[FSTORE-1029](https://hopsworks.atlassian.net/browse/FSTORE-1029) Unexplained OnlineFS exceptions in logs
[FSTORE-1034](https://hopsworks.atlassian.net/browse/FSTORE-1034) .select\(\) method should not default to empty list
[FSTORE-1035](https://hopsworks.atlassian.net/browse/FSTORE-1035) No error if a user tries to create a feature view without features
[FSTORE-1037](https://hopsworks.atlassian.net/browse/FSTORE-1037) ArrayIndexOutOfBounds in TrainingDatasetController
[FSTORE-1038](https://hopsworks.atlassian.net/browse/FSTORE-1038) Misleading exception for ArrowFlight when not waiting for ingestion job to finish
[FSTORE-1044](https://hopsworks.atlassian.net/browse/FSTORE-1044) Insert fails with Feature 'id': dtype 'O' \(arrow\_type 'null'\) not supported
[FSTORE-1049](https://hopsworks.atlassian.net/browse/FSTORE-1049) FlyingDuck gets NoneType in evaluate\_filter\_expression
[FSTORE-1050](https://hopsworks.atlassian.net/browse/FSTORE-1050) FlyingDuck gets Permission denied trying to read hoodie.properties file
[FSTORE-1051](https://hopsworks.atlassian.net/browse/FSTORE-1051) NotImplementedException when querying FeatureView based on external feature group
[FSTORE-1053](https://hopsworks.atlassian.net/browse/FSTORE-1053) Never ending materialization job
[FSTORE-1057](https://hopsworks.atlassian.net/browse/FSTORE-1057) Schema inconsistency in DuckDB after upsert
[FSTORE-1058](https://hopsworks.atlassian.net/browse/FSTORE-1058) if feature view query doesn't include event time feature add for get\_batch\_data\(start\_time, end\_time\)
[FSTORE-1059](https://hopsworks.atlassian.net/browse/FSTORE-1059) ArrowFlight server hangs after some requests
[FSTORE-1062](https://hopsworks.atlassian.net/browse/FSTORE-1062) Materialisation job does not work with new Hopsworks scheduler
[FSTORE-1068](https://hopsworks.atlassian.net/browse/FSTORE-1068) Fixed failed load test
[FSTORE-1071](https://hopsworks.atlassian.net/browse/FSTORE-1071) Flyingduck crashes when joining shared feature stores
[FSTORE-1072](https://hopsworks.atlassian.net/browse/FSTORE-1072) Connection timeout not set in the arrow\_flight\_client.py
[FSTORE-1073](https://hopsworks.atlassian.net/browse/FSTORE-1073) OnlineFS getting offsets fails when project name has upper case letters
[FSTORE-1074](https://hopsworks.atlassian.net/browse/FSTORE-1074) Expand documentation on filter logic
[FSTORE-1075](https://hopsworks.atlassian.net/browse/FSTORE-1075) Cannot get schema from shared project
[FSTORE-1084](https://hopsworks.atlassian.net/browse/FSTORE-1084) Cannot run multiple insert\_stream query on the same project by default
[FSTORE-1086](https://hopsworks.atlassian.net/browse/FSTORE-1086) Broken query when adding a filter on the non-label feature group without selecting the feature
[FSTORE-1087](https://hopsworks.atlassian.net/browse/FSTORE-1087) feature\_view.get\_batch\_data timing out when using ArrowFlight
[FSTORE-1089](https://hopsworks.atlassian.net/browse/FSTORE-1089) Remove copying of application code from databricks integration
[FSTORE-1094](https://hopsworks.atlassian.net/browse/FSTORE-1094) pyarrow.\_flight.FlightServerError: 'fileIdAndRelativePaths' when reading from ArrowFlight
[FSTORE-1095](https://hopsworks.atlassian.net/browse/FSTORE-1095) JDBC storage connector missing driver option in the documentation
Subtask
[FSTORE-1007](https://hopsworks.atlassian.net/browse/FSTORE-1007) Fix training dataset paths on the backend and adjust them on hsfs
Task
[FSTORE-26](https://hopsworks.atlassian.net/browse/FSTORE-26) Print Warning that backfill job had to be started manually, when PySpark engine stream=True and using .insert\_stream\(\)
[FSTORE-335](https://hopsworks.atlassian.net/browse/FSTORE-335) Track GIT commit that was used to create/insert into a feature group
[FSTORE-362](https://hopsworks.atlassian.net/browse/FSTORE-362) Replace Deequ statistics with more lightweight module
[FSTORE-363](https://hopsworks.atlassian.net/browse/FSTORE-363) Make HSFS API typesafe
[FSTORE-367](https://hopsworks.atlassian.net/browse/FSTORE-367) Create internal guide for hsfs unit test
[FSTORE-382](https://hopsworks.atlassian.net/browse/FSTORE-382) Feature store v2.5 documentation - broken links
[FSTORE-394](https://hopsworks.atlassian.net/browse/FSTORE-394) Unify metadata update API in hsfs
[FSTORE-404](https://hopsworks.atlassian.net/browse/FSTORE-404) Add rollback action for Fs job creation.
[FSTORE-471](https://hopsworks.atlassian.net/browse/FSTORE-471) Allow tranformation functions for label features
[FSTORE-550](https://hopsworks.atlassian.net/browse/FSTORE-550) Ensure type consistency between for online/offline/feature view reads in python and spark engine
[FSTORE-665](https://hopsworks.atlassian.net/browse/FSTORE-665) Design log: Kafka topic deletion
[FSTORE-722](https://hopsworks.atlassian.net/browse/FSTORE-722) Hopsworks should not retrieve metadata from the Hive metastore
[FSTORE-725](https://hopsworks.atlassian.net/browse/FSTORE-725) Transformation Function unit test should test the default behaviour of the pre-registered function in the backend
[FSTORE-738](https://hopsworks.atlassian.net/browse/FSTORE-738) UI should show commit timestamps in UTC format
[FSTORE-753](https://hopsworks.atlassian.net/browse/FSTORE-753) Improve user error message for feature group creation and append during RonDB reconfig
[FSTORE-790](https://hopsworks.atlassian.net/browse/FSTORE-790) Replace hsfs with hopsworks in the code snippets shown in the UI
[FSTORE-823](https://hopsworks.atlassian.net/browse/FSTORE-823) Statistics Engine - Design Log
[FSTORE-825](https://hopsworks.atlassian.net/browse/FSTORE-825) Deleting an online feature group doesn't revoke mysql privileges
[FSTORE-826](https://hopsworks.atlassian.net/browse/FSTORE-826) Add test support for AWS and GCP connectors
[FSTORE-828](https://hopsworks.atlassian.net/browse/FSTORE-828) Automatically add prefix if there are duplicate columns in the query join.
[FSTORE-834](https://hopsworks.atlassian.net/browse/FSTORE-834) Relative path in absolute URI - External FG