Aperture-py

Changelog

List of aperture PRs merged since 2.16.0 release. For the full list of changes, see [list of changes][changes]

Fixes in blueprints (2669)

This PR contains following changes:

1. Fixed the hard-coded setpoint in Postgresql, Elasticsearch and JMX
blueprints
2. Moved the config update logic to `bundle.libsonnet` as it would be
needed for dashboard generation
3. Moved the `overload_condition` flag to root of component and fixed it
based on use-case in blueprints

- New Feature: Introduced a new configuration option
`service_protection_core` with a `setpoint` field and an
`overload_condition` field across multiple modules. This allows users to
specify the criteria for determining overload state.
- Refactor: Simplified code by removing unnecessary updates of
configuration objects and directly passing original configurations to
functions.
- Documentation: Updated documentation to reflect changes in
configuration parameters, including the addition of
`service_protection_core.setpoint` and removal of `overload_condition`
from various parameters.
- Chore: Adjusted internal handling of `overload_condition`, moving it
from `Parameters` to top-level in `AIADLoadScheduler`.

Added workload_index label in request_in_queue_duration metrics (2666)


- New Feature: Replaced the `quota_checks` module with the
`workload_decisions` module in Grafana panels, providing more detailed
insights into workload decisions.
- Refactor: Updated the query in the `request_in_queue_duration` panel
to include the `workload_index` field, allowing for more granular
aggregation of metrics.
- Style: Renamed the time series panel from "Quota Checks" to "Workload
Decisions" in Grafana, improving clarity and consistency.
- Refactor: Enhanced Prometheus metrics creation in the workload
scheduler by specifying additional label keys, enabling more precise
tracking and analysis.
- Test: Updated tests in `wfq_test.go` to reflect changes in metrics and
labels, ensuring accurate testing conditions.

blueprints: use remote yaml schema when uri has github.com (2664)

- Refactor: Enhanced the `createValuesFile` function in
`values-common.go`. The update introduces a more robust and flexible
method for constructing the `schemaURL` variable, including improved
handling of GitHub repositories. This change ensures more reliable URL
generation under various scenarios, enhancing the overall stability of
the application.

blueprints generate: add name and uri to the values file (2663)


Fix promql query in postgres blueprint (2661)

- Bug Fix: Corrected a syntax error in the `promqlQuery` variable
assignment within the `policy.libsonnet` file under the
`load-scheduling/postgresql` blueprint. This fix ensures that the query
is properly formed, preventing potential execution errors and enhancing
the reliability of load scheduling for PostgreSQL databases.

Add PGSQL Dashboard to GracefulJs Scenario (2609)

- Created pgsql panels
- Created pgsql dashboard
- Modified current dashboard creation process to add creation for receiver dashboards per infra meter

- New Feature: Added a comprehensive Grafana dashboard for PostgreSQL,
providing detailed insights into various database statistics and
operations.
- New Feature: Introduced several new panels for PostgreSQL metrics,
enhancing the visualization options for users.
- New Feature: Implemented customization options for the legend
displayed in the panel, improving user experience.
- New Feature: Added a new dashboard called "%s-receiver" with its
namespace, labels, and datasource, expanding the monitoring
capabilities.
- Refactor: Improved code modularity by introducing new functions to
generate Prometheus panels for different PostgreSQL metrics.
- Refactor: Enhanced the logic of dashboard addition and generation,
making it more efficient and maintainable.

Fixed incorrect message on policy delete (2659)

Fixes: 2638
"Bug Fix":
- Enhanced error handling in the `deletePolicyUsingAPI` function across
multiple modules. Now, instead of just logging a warning message, the
function returns a detailed error using `fmt.Errorf`. This change allows
for better error context and enables the caller to handle errors more
effectively.

Range throttling strategy for Adaptive Load Scheduler (2603)

- New Feature: Introduced two new throttling strategies,
`AIMDThrottlingStrategy` and `RangeThrottlingStrategy`, to the
`AdaptiveLoadScheduler` for more efficient token rate adjustment.
- New Feature: Added support for defining and using polynomial range
functions with the introduction of the `PolynomialRangeFunction`
component.
- Documentation: Updated the reference configuration specification to
include details about the new throttling strategies and the polynomial
range function.
- Refactor: Modified the `ParseAdaptiveLoadScheduler` function to handle
the new throttling strategies and updated the nested circuit components
accordingly.

Added request in queue duration metrics (2655)


- New Feature: Added a new Prometheus summary metric
`RequestInQueueDurationMetricName` to track the duration of requests in
the queue, enhancing monitoring capabilities.
- New Feature: Introduced a new Grafana panel for workload latency using
the newly added Prometheus metric, providing better visibility into
system performance.
- Refactor: Updated the `Factory` struct and its associated methods in
`scheduler.go` to initialize, retrieve, and delete the new metric.
- Test: Modified tests in `wfq_test.go` to include the new metric in the
`WFQMetrics` struct, ensuring accurate testing of the new feature.

Add more logs when applying/deleting policies (2653)

Controller now logs:
* When policy is created/updated/deleted from etcd via API.
* When policy is actually loaded/unloaded from controller.

Drive-by: Deleting policies via API is now synchronous and propagates
errors.

Enforce single controller per project via leader election (2644)

Now the controller won't start until it wins a leader election (note:
it's a separate leader election than the one agents are doing). The
`Election` was refactored so that it has configurable key and works for
both controllers' and agents' elections.

Drive-by:
* Simplified election package by removing trackers dependency.
* Added some structured concurrency helpers in `panichandler` (and used
when appropriate):
* `WaitGroup` – like `conc.WaitGroup`, but uses our panichandler, No
more `wg.Add(1)` and `defer wg.Done()`.
* `CancellableWaitGroup` – a `WaitGroup` with context and cancellation.
Also provides `GoOnDone` to schedule a callback when a given channel is
done (closed).

Resolves 2588

Introduce Ramp mode (2623)


Always use etcd in GetPolicies (2639)

GetPolicies (used by `aperturectl policies`) now always queries etcd and
cross-checks with the local state and reports policy as VALID only if
the policy is both running and exactly matches the version present in
etcd (config/api/policies). GetPolicy was altered to be consistent with
GetPolicies.

Note: The k8s-managed policies are treated differently – for them
it's expected they're not present in etcd. For consistency with
etcd-managed policies it would be nice if GetPolicies would cross-check
with state of k8s (as it cross-checks with etcd), but this commit
doesn't change this asymmetry.

UpsertPolicy was also rewritten to not depend on local state (almost,
there's a check to avoid accidentally messing with k8s-managed
policies).

Also: Load scheduler constructor now doesn't mutate the policy proto
object (so that policy in the factory now matches policy in etcd).

Resolves: 2587

[changes]: https://github.com/fluxninja/aperture/compare/releases/aperture-controller/v2.16.0...releases/aperture-controller/v2.17.0

Changelog

List of aperture PRs merged since 2.16.0 release. For the full list of changes, see [list of changes][changes]

Fixes in blueprints (2669)

This PR contains following changes:

1. Fixed the hard-coded setpoint in Postgresql, Elasticsearch and JMX
blueprints
2. Moved the config update logic to `bundle.libsonnet` as it would be
needed for dashboard generation
3. Moved the `overload_condition` flag to root of component and fixed it
based on use-case in blueprints

- New Feature: Introduced a new configuration option
`service_protection_core` with a `setpoint` field and an
`overload_condition` field across multiple modules. This allows users to
specify the criteria for determining overload state.
- Refactor: Simplified code by removing unnecessary updates of
configuration objects and directly passing original configurations to
functions.
- Documentation: Updated documentation to reflect changes in
configuration parameters, including the addition of
`service_protection_core.setpoint` and removal of `overload_condition`
from various parameters.
- Chore: Adjusted internal handling of `overload_condition`, moving it
from `Parameters` to top-level in `AIADLoadScheduler`.

Added workload_index label in request_in_queue_duration metrics (2666)


- New Feature: Replaced the `quota_checks` module with the
`workload_decisions` module in Grafana panels, providing more detailed
insights into workload decisions.
- Refactor: Updated the query in the `request_in_queue_duration` panel
to include the `workload_index` field, allowing for more granular
aggregation of metrics.
- Style: Renamed the time series panel from "Quota Checks" to "Workload
Decisions" in Grafana, improving clarity and consistency.
- Refactor: Enhanced Prometheus metrics creation in the workload
scheduler by specifying additional label keys, enabling more precise
tracking and analysis.
- Test: Updated tests in `wfq_test.go` to reflect changes in metrics and
labels, ensuring accurate testing conditions.

blueprints: use remote yaml schema when uri has github.com (2664)

- Refactor: Enhanced the `createValuesFile` function in
`values-common.go`. The update introduces a more robust and flexible
method for constructing the `schemaURL` variable, including improved
handling of GitHub repositories. This change ensures more reliable URL
generation under various scenarios, enhancing the overall stability of
the application.

blueprints generate: add name and uri to the values file (2663)


Fix promql query in postgres blueprint (2661)

- Bug Fix: Corrected a syntax error in the `promqlQuery` variable
assignment within the `policy.libsonnet` file under the
`load-scheduling/postgresql` blueprint. This fix ensures that the query
is properly formed, preventing potential execution errors and enhancing
the reliability of load scheduling for PostgreSQL databases.

Add PGSQL Dashboard to GracefulJs Scenario (2609)

- Created pgsql panels
- Created pgsql dashboard
- Modified current dashboard creation process to add creation for receiver dashboards per infra meter

- New Feature: Added a comprehensive Grafana dashboard for PostgreSQL,
providing detailed insights into various database statistics and
operations.
- New Feature: Introduced several new panels for PostgreSQL metrics,
enhancing the visualization options for users.
- New Feature: Implemented customization options for the legend
displayed in the panel, improving user experience.
- New Feature: Added a new dashboard called "%s-receiver" with its
namespace, labels, and datasource, expanding the monitoring
capabilities.
- Refactor: Improved code modularity by introducing new functions to
generate Prometheus panels for different PostgreSQL metrics.
- Refactor: Enhanced the logic of dashboard addition and generation,
making it more efficient and maintainable.

Fixed incorrect message on policy delete (2659)

Fixes: 2638
"Bug Fix":
- Enhanced error handling in the `deletePolicyUsingAPI` function across
multiple modules. Now, instead of just logging a warning message, the
function returns a detailed error using `fmt.Errorf`. This change allows
for better error context and enables the caller to handle errors more
effectively.

Range throttling strategy for Adaptive Load Scheduler (2603)

- New Feature: Introduced two new throttling strategies,
`AIMDThrottlingStrategy` and `RangeThrottlingStrategy`, to the
`AdaptiveLoadScheduler` for more efficient token rate adjustment.
- New Feature: Added support for defining and using polynomial range
functions with the introduction of the `PolynomialRangeFunction`
component.
- Documentation: Updated the reference configuration specification to
include details about the new throttling strategies and the polynomial
range function.
- Refactor: Modified the `ParseAdaptiveLoadScheduler` function to handle
the new throttling strategies and updated the nested circuit components
accordingly.

Added request in queue duration metrics (2655)


- New Feature: Added a new Prometheus summary metric
`RequestInQueueDurationMetricName` to track the duration of requests in
the queue, enhancing monitoring capabilities.
- New Feature: Introduced a new Grafana panel for workload latency using
the newly added Prometheus metric, providing better visibility into
system performance.
- Refactor: Updated the `Factory` struct and its associated methods in
`scheduler.go` to initialize, retrieve, and delete the new metric.
- Test: Modified tests in `wfq_test.go` to include the new metric in the
`WFQMetrics` struct, ensuring accurate testing of the new feature.

Add more logs when applying/deleting policies (2653)

Controller now logs:
* When policy is created/updated/deleted from etcd via API.
* When policy is actually loaded/unloaded from controller.

Drive-by: Deleting policies via API is now synchronous and propagates
errors.

Enforce single controller per project via leader election (2644)

Now the controller won't start until it wins a leader election (note:
it's a separate leader election than the one agents are doing). The
`Election` was refactored so that it has configurable key and works for
both controllers' and agents' elections.

Drive-by:
* Simplified election package by removing trackers dependency.
* Added some structured concurrency helpers in `panichandler` (and used
when appropriate):
* `WaitGroup` – like `conc.WaitGroup`, but uses our panichandler, No
more `wg.Add(1)` and `defer wg.Done()`.
* `CancellableWaitGroup` – a `WaitGroup` with context and cancellation.
Also provides `GoOnDone` to schedule a callback when a given channel is
done (closed).

Resolves 2588

Introduce Ramp mode (2623)


Always use etcd in GetPolicies (2639)

GetPolicies (used by `aperturectl policies`) now always queries etcd and
cross-checks with the local state and reports policy as VALID only if
the policy is both running and exactly matches the version present in
etcd (config/api/policies). GetPolicy was altered to be consistent with
GetPolicies.

Note: The k8s-managed policies are treated differently – for them
it's expected they're not present in etcd. For consistency with
etcd-managed policies it would be nice if GetPolicies would cross-check
with state of k8s (as it cross-checks with etcd), but this commit
doesn't change this asymmetry.

UpsertPolicy was also rewritten to not depend on local state (almost,
there's a check to avoid accidentally messing with k8s-managed
policies).

Also: Load scheduler constructor now doesn't mutate the policy proto
object (so that policy in the factory now matches policy in etcd).

Resolves: 2587

[changes]: https://github.com/fluxninja/aperture/compare/releases/aperture-controller/v2.16.0...releases/aperture-controller/v2.17.0-rc.2

Changelog

List of aperture PRs merged since 2.16.0 release. For the full list of changes, see [list of changes][changes]

Add PGSQL Dashboard to GracefulJs Scenario (2609)

- Created pgsql panels
- Created pgsql dashboard
- Modified current dashboard creation process to add creation for receiver dashboards per infra meter

- New Feature: Added a comprehensive Grafana dashboard for PostgreSQL,
providing detailed insights into various database statistics and
operations.
- New Feature: Introduced several new panels for PostgreSQL metrics,
enhancing the visualization options for users.
- New Feature: Implemented customization options for the legend
displayed in the panel, improving user experience.
- New Feature: Added a new dashboard called "%s-receiver" with its
namespace, labels, and datasource, expanding the monitoring
capabilities.
- Refactor: Improved code modularity by introducing new functions to
generate Prometheus panels for different PostgreSQL metrics.
- Refactor: Enhanced the logic of dashboard addition and generation,
making it more efficient and maintainable.

Fixed incorrect message on policy delete (2659)

Fixes: 2638
"Bug Fix":
- Enhanced error handling in the `deletePolicyUsingAPI` function across
multiple modules. Now, instead of just logging a warning message, the
function returns a detailed error using `fmt.Errorf`. This change allows
for better error context and enables the caller to handle errors more
effectively.

Range throttling strategy for Adaptive Load Scheduler (2603)

- New Feature: Introduced two new throttling strategies,
`AIMDThrottlingStrategy` and `RangeThrottlingStrategy`, to the
`AdaptiveLoadScheduler` for more efficient token rate adjustment.
- New Feature: Added support for defining and using polynomial range
functions with the introduction of the `PolynomialRangeFunction`
component.
- Documentation: Updated the reference configuration specification to
include details about the new throttling strategies and the polynomial
range function.
- Refactor: Modified the `ParseAdaptiveLoadScheduler` function to handle
the new throttling strategies and updated the nested circuit components
accordingly.

Added request in queue duration metrics (2655)


- New Feature: Added a new Prometheus summary metric
`RequestInQueueDurationMetricName` to track the duration of requests in
the queue, enhancing monitoring capabilities.
- New Feature: Introduced a new Grafana panel for workload latency using
the newly added Prometheus metric, providing better visibility into
system performance.
- Refactor: Updated the `Factory` struct and its associated methods in
`scheduler.go` to initialize, retrieve, and delete the new metric.
- Test: Modified tests in `wfq_test.go` to include the new metric in the
`WFQMetrics` struct, ensuring accurate testing of the new feature.

Add more logs when applying/deleting policies (2653)

Controller now logs:
* When policy is created/updated/deleted from etcd via API.
* When policy is actually loaded/unloaded from controller.

Drive-by: Deleting policies via API is now synchronous and propagates
errors.

Enforce single controller per project via leader election (2644)

Now the controller won't start until it wins a leader election (note:
it's a separate leader election than the one agents are doing). The
`Election` was refactored so that it has configurable key and works for
both controllers' and agents' elections.

Drive-by:
* Simplified election package by removing trackers dependency.
* Added some structured concurrency helpers in `panichandler` (and used
when appropriate):
* `WaitGroup` – like `conc.WaitGroup`, but uses our panichandler, No
more `wg.Add(1)` and `defer wg.Done()`.
* `CancellableWaitGroup` – a `WaitGroup` with context and cancellation.
Also provides `GoOnDone` to schedule a callback when a given channel is
done (closed).

Resolves 2588

Introduce Ramp mode (2623)


Always use etcd in GetPolicies (2639)

GetPolicies (used by `aperturectl policies`) now always queries etcd and
cross-checks with the local state and reports policy as VALID only if
the policy is both running and exactly matches the version present in
etcd (config/api/policies). GetPolicy was altered to be consistent with
GetPolicies.

Note: The k8s-managed policies are treated differently – for them
it's expected they're not present in etcd. For consistency with
etcd-managed policies it would be nice if GetPolicies would cross-check
with state of k8s (as it cross-checks with etcd), but this commit
doesn't change this asymmetry.

UpsertPolicy was also rewritten to not depend on local state (almost,
there's a check to avoid accidentally messing with k8s-managed
policies).

Also: Load scheduler constructor now doesn't mutate the policy proto
object (so that policy in the factory now matches policy in etcd).

Resolves: 2587

Set log timestamp format to include nanoseconds (2641)

- Refactor: Updated the timestamp format in the logging system from
`time.RFC3339` to `time.RFC3339Nano`. This change will provide more
precise timestamps in the logs, aiding in better tracking and debugging.

[changes]: https://github.com/fluxninja/aperture/compare/releases/aperture-controller/v2.16.0...releases/aperture-controller/v2.17.0-rc.1

Changelog

List of aperture PRs merged since 2.15.0 release. For the full list of changes, see [list of changes][changes]

Revert "Added metrics for Flow Duration" (2635)


refactor rate limiter policy spec (2633)


add priorities to checkresponse (2629)


Fixed job name conflict in background scheduler when multiple policies are deployed (2627)

- Refactor: Updated the `background-scheduler.go` in the control plane
runtime to enhance job tracking. Now, each job is assigned a unique name
derived from the policy hash, improving the visibility and traceability
of individual jobs within the system. This change will not affect the
end-user experience but will significantly aid in system maintenance and
debugging processes.

global token bucket: delay filling when bucket has tokens (2626)

<img width="1399" alt="Screenshot 2023-09-14 at 5 02 48 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/f10009bd-e226-4119-b568-b20f379382f1">
<img width="1400" alt="Screenshot 2023-09-14 at 5 02 34 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/cb46078d-5f83-4725-90dc-9843f55f1cf3">



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- Bug Fix: Improved the functionality of the `GlobalTokenBucket` in the
rate-limiter package. The update introduces a mechanism to control token
generation based on the start fill time, preventing more tokens than the
fill rate in a given time window while using burst capacity.
- Documentation: Updated comments for the `bucket_capacity` field in the
`RateLimiter` component and the "Capacity of the bucket" parameter in
the configuration specification. The new descriptions provide better
clarity on how the capacity allows for bursty traffic and gives the
bucket a chance to empty out before filling starts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Added metrics for Flow Duration (2625)


- New Feature: Added a new `FlowDurationMetricName` for grouping
durations of Check calls, enhancing the granularity of metrics.
- New Feature: Introduced a new panel in Grafana dashboard for
visualizing workload latency using the newly added
`FlowDurationMetricName`.
- Refactor: Updated the `metricsProcessor` struct to handle flow control
metrics separately, improving the efficiency of metrics processing.
- Test: Enhanced the test cases in `processor_test.go` to accommodate
the new flow duration metric, ensuring the robustness of the new
feature.
- Chore: Added a new method `GetFlowDurationSummary` across various
components (like `quotaScheduler`, `Factory`, `Scheduler`) for fetching
flow duration summary based on provided labels.

scheduler: provide remaining/current tokens back in check call (2620)

<img width="1392" alt="Screenshot 2023-09-13 at 10 35 55 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/3c5aef8f-970c-4cc1-904e-8d1007b0eaa4">
<img width="1384" alt="Screenshot 2023-09-13 at 10 35 40 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/f13dade6-63b6-40a9-83da-7b6109c1884d">



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- New Feature: Enhanced the `LimiterDecision` message in protobuf
definitions to include a nested `TokensInfo` message, providing more
detailed information about token consumption.
- Refactor: Updated various functions across multiple files to return
additional values related to token availability and wait time, improving
the granularity of operation details.
- Refactor: Modified the `Decide` function in several structs to return
a pointer to `flowcontrolv1.LimiterDecision`, simplifying the code and
improving consistency.
- Test: Adjusted the duration of stages in the load generator test
script, affecting the target number of virtual users for these stages.
- Chore: Renamed and updated the `updateMetricsAndReturnDecision` method
in the `WFQScheduler` struct to return additional values, enhancing
metrics tracking.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Fixed aperturectl cloud command addition (2619)

- New Feature: Added two new commands to the `aperturectl` CLI tool,
`apply.ApplyCmd` and `delete.DeleteCmd`, enhancing user control over
policy application and resource deletion in the Cloud Controller.
- Documentation: Updated and added comprehensive documentation for the
new commands, including usage examples and options. Renamed `aperturectl
cloud policy` to `aperturectl cloud apply policy` for clarity and
consistency.
- User Impact: These changes provide users with more precise command
options for managing Aperture policies, improving usability and
efficiency of the `aperturectl` tool.

Added cloud sub-command for policy apply and delete (2618)

- Added new `aperturectl cloud` sub-command for policy apply and delete.
We will still keep the older way for a release so that the cloud gets
updated in the mean time.
- Created new proto for the cloud cmd services which will be used in the
cloud
- We will eventually migrate all the cloud controller sub-commands like
`preview` etc to this and have a clear separation between OSS and Cloud

- New Feature: Added `ApplyPolicyCmd` and `DeletePolicyCmd` commands to
the `aperturectl` CLI tool for managing Aperture policies on the Cloud
Controller.
- New Feature: Introduced a terminal user interface (TUI) for selecting
which policies to apply from a directory.
- Refactor: Moved policy-related functionality to a new `utils` package,
improving modularity and maintainability.
- Documentation: Added comprehensive documentation for the new "cloud"
command in the `aperturectl` CLI tool, including usage examples and
options.
- Chore: Updated protocol buffer definitions for communication between
`aperturectl` and the cloud controller.

Make logging in FX apps structured (2616)

Description of change

Checklist

- [x] Tested in playground or other setup
- [ ] Screenshot (Grafana) from playground added to PR for 15+ minute
run
- [ ] Documentation is changed or added
- [ ] Tests and/or benchmarks are included
- [ ] Breaking changes



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

"Refactor: Enhanced Logging Mechanism"
- The update introduces a more robust and configurable logging system
using `go.uber.org/zap`.
- This change allows for better tracking of application events, aiding
in debugging and monitoring.
- The new logger is integrated with the FX application via
`fx.WithLogger()`, ensuring consistent logging across all components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

token bucket: allow 0 capacity buckets (2617)

0 capacity buckets should be allowed, and there is no reason in the code
that this leads to any issues.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- New Feature: Added logging for request deadline and timeout in
`cmd/sdk-validator/validator/flowcontrol.go` to improve debugging
capabilities.
- Refactor: Simplified the logic in
`pkg/rate-limiter/global-token-bucket/global-token-bucket.go` by
removing unnecessary check for bucket capacity, enhancing code
readability and maintainability.
- New Feature: Increased the timeout for requests made by the
`ApertureClient` in `sdks/aperture-js/example/routes/use_aperture.ts`,
improving user experience during network latency.
- Style: Improved code readability in `sdks/aperture-js/sdk/client.ts`
by reformatting import statements and overall code structure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Revert "Set log timestamp format to include nanoseconds (2614)" (2615)

This reverts commit d6547ee1cd90b4339b95ac5b0e739ce68e7a1bfd.
Unfortunately this breaks log format in LogsExplorer in gcloud.

Set log timestamp format to include nanoseconds (2614)

- Refactor: Updated the global logger instance to use `time.RFC3339Nano`
for more precise timestamp logging.
- Refactor: Enhanced the `zerolog.ErrorStackMarshaler` and
`zerolog.CallerMarshalFunc` functions for improved error tracking and
caller information.

Spec: rename priority label key (2611)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- Refactor: Renamed `priorities_label_key` to `priority_label_key` in
the `Scheduler` message definition and updated all references in the
codebase.
- Refactor: Simplified the `Decide` function in the `Scheduler` struct
by removing unnecessary variables and updating label checks.
- Refactor: Improved code readability in `client.ts` by reformatting
imports, using destructuring and shorthand syntax, and extracting
callback functions.
- Change: Updated the default value of `timeoutMilliseconds` parameter
in the constructor from 200 to 0 in `client.ts`.
- Refactor: Simplified request object creation and gRPC parameters in
`client.ts`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Use buf's central proto and grpc packages instead of copying them over in sdk (2607)

Description of change

Checklist

- [ ] Tested in playground or other setup
- [ ] Screenshot (Grafana) from playground added to PR for 15+ minute
run
- [ ] Documentation is changed or added
- [ ] Tests and/or benchmarks are included
- [ ] Breaking changes



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

**Release Notes:**

- **Refactor**: Updated the build process and SDK generation in
`api/Makefile`. Now supports Java, JavaScript, and Python SDKs.
- **New Feature**: Introduced new packages and updated existing ones in
`cmd/sdk-validator/main.go` to enhance functionality and improve code
organization.
- **Refactor**: Updated import paths and data structures in
`sdks/aperture-go/sdk/client.go`, `sdks/aperture-go/sdk/flow.go`,
`sdks/aperture-go/sdk/httpflow.go`,
`sdks/aperture-go/sdk/middleware/grpc.go`,
`sdks/aperture-go/sdk/middleware/utils.go` for better compatibility with
new package paths.
- **Style**: Improved readability of function signatures in
`cmd/sdk-validator/validator/common.go`.
- **Chore**: Renamed "middlewares" package to "middleware" in
`sdks/aperture-go/sdk/middleware/http.go` for consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

aperturectl: Fail early on unsupported commands with Cloud Controller (2606)


Make default GRPC server name not empty (2589)

Description of change
Having the name of default GRPC Server empty messed up the fx tags. As
we use `group:<server_name>` tags, fx read this as no group tag at all
as it was empty. Trying to bring up another GRPC server with a name
resulted in an error that groups can't be optional.

This is breaking change for someone using this as a library. It won't
affect anyone else.

Checklist

- [ ] Tested in playground or other setup
- [ ] Screenshot (Grafana) from playground added to PR for 15+ minute
run
- [x] Breaking changes (only for library users)




<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- Refactor: Introduced new struct types to bundle and annotate
parameters for various functions across multiple packages. This change
improves code modularity, readability, and maintainability.
- Refactor: Updated the `ServerModule` and `GMuxServerModule` functions
in `pkg/net/grpc/server.go` to improve configurability and consistency.
- Chore: Removed unnecessary imports and reorganized code in
`cmd/aperture-agent/agent/otel-component.go` and
`test/aperture_suite_test.go`.

These changes do not introduce any new features or bug fixes, but they
significantly enhance the code quality and maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Hardik Shingala <34568645+hdkshingalausers.noreply.github.com>
Co-authored-by: Hardik Shingala <hardik.shingalafluxninja.com>

[changes]: https://github.com/fluxninja/aperture/compare/releases/aperture-controller/v2.15.0...releases/aperture-controller/v2.16.0

Changelog

List of aperture PRs merged since 2.15.0 release. For the full list of changes, see [list of changes][changes]

Revert "Added metrics for Flow Duration" (2635)


refactor rate limiter policy spec (2633)


add priorities to checkresponse (2629)


Fixed job name conflict in background scheduler when multiple policies are deployed (2627)

- Refactor: Updated the `background-scheduler.go` in the control plane
runtime to enhance job tracking. Now, each job is assigned a unique name
derived from the policy hash, improving the visibility and traceability
of individual jobs within the system. This change will not affect the
end-user experience but will significantly aid in system maintenance and
debugging processes.

global token bucket: delay filling when bucket has tokens (2626)

<img width="1399" alt="Screenshot 2023-09-14 at 5 02 48 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/f10009bd-e226-4119-b568-b20f379382f1">
<img width="1400" alt="Screenshot 2023-09-14 at 5 02 34 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/cb46078d-5f83-4725-90dc-9843f55f1cf3">



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- Bug Fix: Improved the functionality of the `GlobalTokenBucket` in the
rate-limiter package. The update introduces a mechanism to control token
generation based on the start fill time, preventing more tokens than the
fill rate in a given time window while using burst capacity.
- Documentation: Updated comments for the `bucket_capacity` field in the
`RateLimiter` component and the "Capacity of the bucket" parameter in
the configuration specification. The new descriptions provide better
clarity on how the capacity allows for bursty traffic and gives the
bucket a chance to empty out before filling starts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Added metrics for Flow Duration (2625)


- New Feature: Added a new `FlowDurationMetricName` for grouping
durations of Check calls, enhancing the granularity of metrics.
- New Feature: Introduced a new panel in Grafana dashboard for
visualizing workload latency using the newly added
`FlowDurationMetricName`.
- Refactor: Updated the `metricsProcessor` struct to handle flow control
metrics separately, improving the efficiency of metrics processing.
- Test: Enhanced the test cases in `processor_test.go` to accommodate
the new flow duration metric, ensuring the robustness of the new
feature.
- Chore: Added a new method `GetFlowDurationSummary` across various
components (like `quotaScheduler`, `Factory`, `Scheduler`) for fetching
flow duration summary based on provided labels.

scheduler: provide remaining/current tokens back in check call (2620)

<img width="1392" alt="Screenshot 2023-09-13 at 10 35 55 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/3c5aef8f-970c-4cc1-904e-8d1007b0eaa4">
<img width="1384" alt="Screenshot 2023-09-13 at 10 35 40 PM"
src="https://github.com/fluxninja/aperture/assets/18579817/f13dade6-63b6-40a9-83da-7b6109c1884d">



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- New Feature: Enhanced the `LimiterDecision` message in protobuf
definitions to include a nested `TokensInfo` message, providing more
detailed information about token consumption.
- Refactor: Updated various functions across multiple files to return
additional values related to token availability and wait time, improving
the granularity of operation details.
- Refactor: Modified the `Decide` function in several structs to return
a pointer to `flowcontrolv1.LimiterDecision`, simplifying the code and
improving consistency.
- Test: Adjusted the duration of stages in the load generator test
script, affecting the target number of virtual users for these stages.
- Chore: Renamed and updated the `updateMetricsAndReturnDecision` method
in the `WFQScheduler` struct to return additional values, enhancing
metrics tracking.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Fixed aperturectl cloud command addition (2619)

- New Feature: Added two new commands to the `aperturectl` CLI tool,
`apply.ApplyCmd` and `delete.DeleteCmd`, enhancing user control over
policy application and resource deletion in the Cloud Controller.
- Documentation: Updated and added comprehensive documentation for the
new commands, including usage examples and options. Renamed `aperturectl
cloud policy` to `aperturectl cloud apply policy` for clarity and
consistency.
- User Impact: These changes provide users with more precise command
options for managing Aperture policies, improving usability and
efficiency of the `aperturectl` tool.

Added cloud sub-command for policy apply and delete (2618)

- Added new `aperturectl cloud` sub-command for policy apply and delete.
We will still keep the older way for a release so that the cloud gets
updated in the mean time.
- Created new proto for the cloud cmd services which will be used in the
cloud
- We will eventually migrate all the cloud controller sub-commands like
`preview` etc to this and have a clear separation between OSS and Cloud

- New Feature: Added `ApplyPolicyCmd` and `DeletePolicyCmd` commands to
the `aperturectl` CLI tool for managing Aperture policies on the Cloud
Controller.
- New Feature: Introduced a terminal user interface (TUI) for selecting
which policies to apply from a directory.
- Refactor: Moved policy-related functionality to a new `utils` package,
improving modularity and maintainability.
- Documentation: Added comprehensive documentation for the new "cloud"
command in the `aperturectl` CLI tool, including usage examples and
options.
- Chore: Updated protocol buffer definitions for communication between
`aperturectl` and the cloud controller.

Make logging in FX apps structured (2616)

Description of change

Checklist

- [x] Tested in playground or other setup
- [ ] Screenshot (Grafana) from playground added to PR for 15+ minute
run
- [ ] Documentation is changed or added
- [ ] Tests and/or benchmarks are included
- [ ] Breaking changes



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

"Refactor: Enhanced Logging Mechanism"
- The update introduces a more robust and configurable logging system
using `go.uber.org/zap`.
- This change allows for better tracking of application events, aiding
in debugging and monitoring.
- The new logger is integrated with the FX application via
`fx.WithLogger()`, ensuring consistent logging across all components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

token bucket: allow 0 capacity buckets (2617)

0 capacity buckets should be allowed, and there is no reason in the code
that this leads to any issues.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- New Feature: Added logging for request deadline and timeout in
`cmd/sdk-validator/validator/flowcontrol.go` to improve debugging
capabilities.
- Refactor: Simplified the logic in
`pkg/rate-limiter/global-token-bucket/global-token-bucket.go` by
removing unnecessary check for bucket capacity, enhancing code
readability and maintainability.
- New Feature: Increased the timeout for requests made by the
`ApertureClient` in `sdks/aperture-js/example/routes/use_aperture.ts`,
improving user experience during network latency.
- Style: Improved code readability in `sdks/aperture-js/sdk/client.ts`
by reformatting import statements and overall code structure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Revert "Set log timestamp format to include nanoseconds (2614)" (2615)

This reverts commit d6547ee1cd90b4339b95ac5b0e739ce68e7a1bfd.
Unfortunately this breaks log format in LogsExplorer in gcloud.

Set log timestamp format to include nanoseconds (2614)

- Refactor: Updated the global logger instance to use `time.RFC3339Nano`
for more precise timestamp logging.
- Refactor: Enhanced the `zerolog.ErrorStackMarshaler` and
`zerolog.CallerMarshalFunc` functions for improved error tracking and
caller information.

Spec: rename priority label key (2611)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- Refactor: Renamed `priorities_label_key` to `priority_label_key` in
the `Scheduler` message definition and updated all references in the
codebase.
- Refactor: Simplified the `Decide` function in the `Scheduler` struct
by removing unnecessary variables and updating label checks.
- Refactor: Improved code readability in `client.ts` by reformatting
imports, using destructuring and shorthand syntax, and extracting
callback functions.
- Change: Updated the default value of `timeoutMilliseconds` parameter
in the constructor from 200 to 0 in `client.ts`.
- Refactor: Simplified request object creation and gRPC parameters in
`client.ts`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Use buf's central proto and grpc packages instead of copying them over in sdk (2607)

Description of change

Checklist

- [ ] Tested in playground or other setup
- [ ] Screenshot (Grafana) from playground added to PR for 15+ minute
run
- [ ] Documentation is changed or added
- [ ] Tests and/or benchmarks are included
- [ ] Breaking changes



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

**Release Notes:**

- **Refactor**: Updated the build process and SDK generation in
`api/Makefile`. Now supports Java, JavaScript, and Python SDKs.
- **New Feature**: Introduced new packages and updated existing ones in
`cmd/sdk-validator/main.go` to enhance functionality and improve code
organization.
- **Refactor**: Updated import paths and data structures in
`sdks/aperture-go/sdk/client.go`, `sdks/aperture-go/sdk/flow.go`,
`sdks/aperture-go/sdk/httpflow.go`,
`sdks/aperture-go/sdk/middleware/grpc.go`,
`sdks/aperture-go/sdk/middleware/utils.go` for better compatibility with
new package paths.
- **Style**: Improved readability of function signatures in
`cmd/sdk-validator/validator/common.go`.
- **Chore**: Renamed "middlewares" package to "middleware" in
`sdks/aperture-go/sdk/middleware/http.go` for consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

aperturectl: Fail early on unsupported commands with Cloud Controller (2606)


Make default GRPC server name not empty (2589)

Description of change
Having the name of default GRPC Server empty messed up the fx tags. As
we use `group:<server_name>` tags, fx read this as no group tag at all
as it was empty. Trying to bring up another GRPC server with a name
resulted in an error that groups can't be optional.

This is breaking change for someone using this as a library. It won't
affect anyone else.

Checklist

- [ ] Tested in playground or other setup
- [ ] Screenshot (Grafana) from playground added to PR for 15+ minute
run
- [x] Breaking changes (only for library users)




<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
Summary by CodeRabbit

- Refactor: Introduced new struct types to bundle and annotate
parameters for various functions across multiple packages. This change
improves code modularity, readability, and maintainability.
- Refactor: Updated the `ServerModule` and `GMuxServerModule` functions
in `pkg/net/grpc/server.go` to improve configurability and consistency.
- Chore: Removed unnecessary imports and reorganized code in
`cmd/aperture-agent/agent/otel-component.go` and
`test/aperture_suite_test.go`.

These changes do not introduce any new features or bug fixes, but they
significantly enhance the code quality and maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Hardik Shingala <34568645+hdkshingalausers.noreply.github.com>
Co-authored-by: Hardik Shingala <hardik.shingalafluxninja.com>

[changes]: https://github.com/fluxninja/aperture/compare/releases/aperture-controller/v2.15.0...releases/aperture-controller/v2.16.0-rc.1

Changelog

List of aperture PRs merged since 2.15.0 release. For the full list of changes, see [list of changes][changes]

Fixed job name conflict in background scheduler when multiple policies are deployed (2627)

- Refactor: Updated the `background-scheduler.go` in the control plane
runtime to enhance job tracking. Now, each job is assigned a unique name
derived from the policy hash, improving the visibility and traceability
of individual jobs within the system. This change will not affect the
end-user experience but will significantly aid in system maintenance and
debugging processes.

[changes]: https://github.com/fluxninja/aperture/compare/releases/aperture-controller/v2.15.0...releases/aperture-controller/v2.15.1