Featurebyte

🛑 Breaking Changes

+ `sdk-api` Skip filling null value by default for aggregated features.
+ `service` Rename FeatureJobSetting attributes to match the new naming convention.

💡 Enhancements

+ `service` Perform sampling operations without sorting tables
+ `service` Support offset parameter in aggregate_over and forward_aggregate
+ `service` Add default feature job settings to the SCDTable.
+ `dependencies` Bumped `freeware` to 0.2.18 to support new feature job settings
+ `service` Relax constraint that key has to be a lookup feature in dictionary operations
+ `dependencies` bump snowflake-connector-python

🐛 Bug Fixes

+ `service` Fix incorrect type casting in most frequent value UDF for Databricks Unity

💡 Enhancements

+ `service` Backfill only required tiles for offline store tables when enabling a deployment
+ `service` Fix view and table describe method error on invalid datetime values
+ `service` Cast type for features with float dtype
+ `docker` Bump base docker image to python 3.10
+ `api` Introduce databricks accessor to deployment API object.
+ `api` Support specifying the target column when creating an observation table.
+ This change allows users to specify the target column when creating an observation table.
+ The target column is the column that contains the target values for the observations.
+ The target column name must match a valid target namespace name in the catalog.
+ The primary entities of the target namespace must match that of the observation table.
+ `service` Run feature computation queries in parallel
+ `service` Cast features with integer dtype BIGINT explicitly in feature queries
+ `api` Use async task for table / view / column describe to avoid timeout on large datasets.
+ `gh-actions` Migration to pytest-split to github actions
+ Databricks tests
+ Spark tests
+ `service` Avoid repeated graph flattening in GraphInterpreter and improve tile sql generation efficiency
+ `service` Skip casting data to string in describe query if not required
+ `sdk-api` Prevent users from creating a UDF feature that is not deployable.
+ `service` Run on demand tile computation concurrently
+ `service` Validate point in time and entity columns do not contain missing values in observation table
+ `service` Validate internal row index column is valid after features computation
+ `service` Improve precomputed lookup feature tables handling
+ `service` Support creating Target objects using forward_aggregate_asat
+ `service` Handle duplicate rows when looking up SCD and dimension tables
+ `service` Calculate entropy using absolute count values
+ `models` Limit asset names to 255 characters in length to ensure they can be referenced as identifiers in SQL queries
+ This change ensures that asset names are compatible with the maximum length of identifiers in SQL queries + This change will prevent errors when querying assets with long names
+ `dependencies` Bump dependencies to latest version
1. snowflake-connector-python
2. databricks-sdk
3. databricks-sql-connector
+ `api` Add more associated objects to historical feature table objects.
+ `service` Create tile cache working tables in parallel

⚠️ Deprecations

+ `redis` Dropping aioredis as redis client library

🐛 Bug Fixes

+ `service` Fix offline store feature table name construction logic to avoid name collisions
+ `service` Fix ambiguous column name error when concatenating serving names
+ `service` Fix target SCD lookup code definition generation bug when the target name contains special characters.
+ `deps` Pinning pyopenssl to 24.X.X as client requirement
+ `service` Databricks integration is not working as expected.
+ `service` Fix KeyError caused by precomputed_lookup_feature_table_info due to backward compatibility issue
+ `session` Set active schema for the snowflake explicitly. The connector does not set the active schema specified.
+ `service` Fix an error when submitting data describe task payload
+ `session` Fix dtype detected wrongly for MAP type in Spark session
+ `api` Make dtype mandatory when create a target namespace
+ `session` Fix DataBricks relative frequency UDF to return None when all counts are 0
+ `service` Handle missing values in SCD effective timestamp and point in time columns
+ `session` Fix DataBricks entropy UDF to return 0 when all counts are 0
+ `udf` Fix division by zero in count dict cosine similarity UDFs
+ `dependencies` Bumping vulnerable dependencies
+ orjson
+ cryptography
+ ~~fastapi~~ (Need to bump to pydantic 2.X.X)
+ python-multipart
+ aiohttp
+ jupyterlab
+ black
+ pymongo
+ pillow
+ `session` Set ownership of created tables to the session group. This is a fix for the issue where the tables created cannot be updated by other users in the group.

🐛 Bug Fixes

+ `service` Databricks integration fix

💡 Enhancements

+ `api` Support description specification during table creation.
+ `api` Create api to manage online stores
+ `session` Specify role and group in Snowflake and Databricks details to enforce permissions for accessing source and output tables
+ `service` Simplify user defined function route creation schema
+ `online_serving` Implement FEAST offline stores for Spark Thrift and DataBricks for online serving support
+ `service` Compute data description in batches of columns
+ `service` Support offset parameter for aggregate_asat
+ `profile` Create a profile from databricks secrets to simplify access from a Databricks workspace.
+ `service` Improve efficiency of feature table cache checks for saved feature lists
+ `session` Add client_session_keep_alive to snowflake connector to keep the session alive
+ `service` Support cancellation for historical features table creation task

🐛 Bug Fixes

+ `service` Updates output variable type of count aggregation to be integer instead of float
+ `service` Fix FeatureList online_enabled_feature_ids attribute not updated correctly in some cases
+ `session` Fix snowflake session using wrong role if the user's default role does not match role in feature store details
+ `session` Fix count dictionary entropy UDF behavior for edge cases
+ `deployment` Fix getting sample entity serving names for deployment fails when entity has null values
+ `service` Fix ambiguous column name error when using SCD lookup features with different offsets

💡 Enhancements

+ `session` Implement missing UDFs for DataBricks clusters that support Unity Catalog.
+ `storage` Support azure blob storage for file storage.

🐛 Bug Fixes

+ `service` Fixes a bug where the feature saving would fail if the feature or colum name contains quotes.
+ `deployment` Fix an issue where periodic tasks were not disabled when reverting a failed deployment

🛑 Breaking Changes

+ `api` Support using observation tables in feature, target and featurelist preview
+ Parameter `observation_set` in `Feature.preview`, `Target.preview` and `FeatureList.preview` now accepts `ObservationTable` object or pandas dataframe
+ Breaking change: Parameter `observation_table` in `FeatureList.compute_historical_feature_table` is renamed to `observation_set`
+ `feature_list` Change feature list catalog output dataframe column name from `primary_entities` to `primary_entity`

💡 Enhancements

+ `databricks-unity` Add session for databricks unity cluster, and migrate one UDF to python for databricks unity cluster.
+ `target` Allow users to create observation table with just a target id, but no graph.
+ `service` Support latest aggregation for vector columns
+ `service` Update repeated columns validation logic to handle excluded columns.
+ `endpoints` Enable observation table to associate with multiple use cases from endpoints
+ `target` Derive window for lookup targets as well
+ `service` Add critical data info validation logic
+ `api` Implement remove observation table from context
+ `service` Support rename of context, use case, observation table and historical feature table
+ `target_table` Persist primary entity IDs for the target observation table
+ `observation_table` Update observation table creation check to make sure primary entity is set
+ `service` Implement service to materialize features to be published to external feature store
+ `service` Add feature definition hash to new feature model to allow duplicated features to be detected
+ `observation_table` Track uploaded file name when creating an observation table from an uploaded file.
+ `observation_table` Add way to update purpose for observation table.
+ `tests` Use published featurebyte library in notebook tests.
+ `service` Reduce complexity of describe query to avoid memory issue during query compilation
+ `session` Use DBFS for Databricks session storage to simplify setup
+ `target_namespace` Add support for target namespace deletion
+ `observation_table` add minimum interval between entities to observation table
+ `api` Implement delete observation table from use case
+ `api` Implement removal of default preview and eda table for context
+ `api` Enable observation table to associate with multiple use cases from api
+ `api` Implement removal of default preview and eda table for use case

🐛 Bug Fixes

+ `observation_table` fix validation around primary entity IDs when creating observation tables
+ `worker` Use cpu worker for feature job setting analysis to avoid blocking io worker async loop
+ `session` Make data warehouse session creation asynchronous with a timeout to avoid blocking the asyncio main thread. This prevents the API service from being unresponsive when certain compute clusters takes a long time to start up.
+ `service` Fix observation table sampling so that it is always uniform over the input
+ `worker` Fix feature job setting analysis fails for databricks feature store
+ `session` Fix spark session failing with spark version >= 3.4.1
+ `service` Fix observation table file upload error
+ `target` Support value_column=None for count in forward_aggregate/target operations.
+ `service` Fix division by zero error when calling describe on empty views
+ `worker` Fix bug where feature job setting analysis backtest fails when the analysis is missing an optional histogram
+ `service` Fixes a view join issue that causes the generated feature not savable due to graph inconsistency.
+ `use_case` Allow use cases to be created with descriptive only targets
+ `service` Fixes an error when rendering FeatureJobStatusResult in notebooks when matplotlib package is not available.
+ `feature` Fix feature saving bug when the feature contains timestamp filtering