Cortex

**New features**

* Add support for the Classic Load Balancer for APIs; the Network Load Balancer remains the default ([docs](https://docs.cortex.dev/clusters/management/create#cluster.yaml)) https://github.com/cortexlabs/cortex/pull/2413 https://github.com/cortexlabs/cortex/issues/2414 ([RobertLucian](https://github.com/RobertLucian))

**Bug fixes**

* Fix Async API http/tcp probes when probing the empty root path (`/`) https://github.com/cortexlabs/cortex/pull/2407 ([RobertLucian](https://github.com/RobertLucian))
* Fix nil pointer exception in the `cortex cluster export` command https://github.com/cortexlabs/cortex/pull/2415 https://github.com/cortexlabs/cortex/issues/2414 ([RobertLucian](https://github.com/RobertLucian))
* Ensure that user-specified environment variables are ordered deterministically in the Kubernetes deployment spec https://github.com/cortexlabs/cortex/pull/2411 ([deliahu](https://github.com/deliahu))

**Misc**

* Ensure that the batch on-job-complete request contains a valid JSON body https://github.com/cortexlabs/cortex/pull/2409 ([RobertLucian](https://github.com/RobertLucian))

**New features**

* Support configurable `pre_stop` command for containers https://github.com/cortexlabs/cortex/pull/2403 ([docs](https://docs.cortex.dev/workloads/realtime/configuration)) ([deliahu](https://github.com/deliahu))

**Misc**

* Support m6i instance types https://github.com/cortexlabs/cortex/pull/2398 ([deliahu](https://github.com/deliahu))
* Update to Kubernetes v1.21 https://github.com/cortexlabs/cortex/pull/2398 ([deliahu](https://github.com/deliahu))

**Bug fixes**

* Wait for in-flight requests to reach zero before terminating the proxy container https://github.com/cortexlabs/cortex/pull/2402 ([deliahu](https://github.com/deliahu))
* Fix `cortex get --env` command https://github.com/cortexlabs/cortex/pull/2404 ([deliahu](https://github.com/deliahu))
* Fix cluster price estimate during `cortex cluster up` for spot node groups with on-demand base capacity https://github.com/cortexlabs/cortex/pull/2406 ([RobertLucian](https://github.com/RobertLucian))

Nucleus Model Server

We have released v0.1.0 of the [Nucleus model server](https://github.com/cortexlabs/nucleus)!

Nucleus is a model server for TensorFlow and generic Python models. It is compatible with Cortex clusters, Kubernetes clusters, and any other container-based deployment platforms. Nucleus can also be run locally via Docker compose.

Some of Nucleus's features include:

* Generic Python models (PyTorch, ONNX, Sklearn, MLFlow, Numpy, Pandas, etc)
* TensorFlow models
* CPU and GPU support
* Serve models directly from S3 paths
* Configurable multiprocessing and multithreadding
* Multi-model endpoints
* Dynamic server-side request batching
* Automatic model reloading when new model versions are uploaded to S3
* Model caching based on LRU policy (on disk and memory)
* HTTP and gRPC support

**New features**

* Support concurrency for Async APIs (via the `max_concurrency` field) https://github.com/cortexlabs/cortex/pull/2376 https://github.com/cortexlabs/cortex/issues/2200 ([miguelvr](https://github.com/miguelvr))
* Add graphs for cluster-wide and per-API cost breakdowns to the cluster metrics dashboard https://github.com/cortexlabs/cortex/pull/2382 https://github.com/cortexlabs/cortex/issues/1962 ([RobertLucian](https://github.com/RobertLucian))
* Allow worker nodes containing Async APIs to scale to zero (now a shared async gateway is used, which runs on the operator node group) https://github.com/cortexlabs/cortex/pull/2380 https://github.com/cortexlabs/cortex/issues/2279 ([vishalbollu](https://github.com/vishalbollu))
* Add `cortex describe API_NAME` command for Realtime and Async APIs https://github.com/cortexlabs/cortex/pull/2368 https://github.com/cortexlabs/cortex/issues/2320 https://github.com/cortexlabs/cortex/issues/2359 ([RobertLucian](https://github.com/RobertLucian))
* Support updating the priority of an existing node group https://github.com/cortexlabs/cortex/pull/2369 https://github.com/cortexlabs/cortex/issues/2254 ([vishalbollu](https://github.com/vishalbollu))

**Misc**

* Improve the reporting of API statuses https://github.com/cortexlabs/cortex/pull/2368 https://github.com/cortexlabs/cortex/issues/2320 https://github.com/cortexlabs/cortex/issues/2359 ([RobertLucian](https://github.com/RobertLucian))
* Remove the default readiness probe on the target port if a custom readiness probe is specified in the API spec https://github.com/cortexlabs/cortex/pull/2379 ([RobertLucian](https://github.com/RobertLucian))

**Bug fixes**

* Remove an unnecessary cluster validation which limited the IP ranges that could be used in `api_load_balancer_cidr_white_list` and `operator_load_balancer_cidr_white_list` https://github.com/cortexlabs/cortex/pull/2363 ([RobertLucian](https://github.com/RobertLucian))

**New features**

* Add `cortex cluster health` command to show the health of the cluster's components https://github.com/cortexlabs/cortex/pull/2313 https://github.com/cortexlabs/cortex/issues/2029 ([miguelvr](https://github.com/miguelvr))
* Forward request headers to AsyncAPIs https://github.com/cortexlabs/cortex/pull/2329 https://github.com/cortexlabs/cortex/issues/2296 ([miguelvr](https://github.com/miguelvr))
* Add metrics dashboard for Task APIs https://github.com/cortexlabs/cortex/pull/2311 https://github.com/cortexlabs/cortex/pull/2322 ([RobertLucian](https://github.com/RobertLucian))

**Reliability**

* Enable larger cluster sizes (up to 1000 nodes with 10000 pods) by enabling IPVS https://github.com/cortexlabs/cortex/pull/2357 https://github.com/cortexlabs/cortex/issues/1834 ([RobertLucian](https://github.com/RobertLucian))
* Automatically limit the rate at which nodes are added to avoid overloading the Kubernetes API server https://github.com/cortexlabs/cortex/pull/2331 https://github.com/cortexlabs/cortex/pull/2338 https://github.com/cortexlabs/cortex/issues/2314 ([RobertLucian](https://github.com/RobertLucian))
* Ensure cluster autoscaler availability https://github.com/cortexlabs/cortex/pull/2347 https://github.com/cortexlabs/cortex/issues/2346 ([RobertLucian](https://github.com/RobertLucian))
* Improve istiod availability at large scale https://github.com/cortexlabs/cortex/pull/2342 https://github.com/cortexlabs/cortex/issues/2332 ([RobertLucian](https://github.com/RobertLucian))
* Reduce metrics shown in `cortex get` to improve scalability and reliability of the command https://github.com/cortexlabs/cortex/pull/2333 https://github.com/cortexlabs/cortex/issues/2319 ([vishalbollu](https://github.com/vishalbollu))
* Show aggregated node statistics in the cluster dashboard https://github.com/cortexlabs/cortex/pull/2336 https://github.com/cortexlabs/cortex/issues/2318 ([RobertLucian](https://github.com/RobertLucian))

**Bug fixes**

* Ensure that the `Content-Type` header is properly set to `application/json` for responses to Async API submissions https://github.com/cortexlabs/cortex/pull/2323 ([vishalbollu](https://github.com/vishalbollu))
* Fix pod autoscaler scale-to-zero edge cases https://github.com/cortexlabs/cortex/pull/2350 ([miguelvr](https://github.com/miguelvr))
* Allow autoscaling configuration to be updated on a running API https://github.com/cortexlabs/cortex/pull/2355 ([RobertLucian](https://github.com/RobertLucian))
* Fix node group priority calculation for the cluster autoscaler https://github.com/cortexlabs/cortex/pull/2358 https://github.com/cortexlabs/cortex/pull/2343 ([RobertLucian](https://github.com/RobertLucian), [deliahu](https://github.com/deliahu))
* Allow the `node_groups` selector to be updated in a running API https://github.com/cortexlabs/cortex/pull/2354 ([RobertLucian](https://github.com/RobertLucian))
* Fix the active replicas graph on the Async API dashboard https://github.com/cortexlabs/cortex/pull/2328 ([RobertLucian](https://github.com/RobertLucian))

**Docs**

* Add a guide for [running in production](https://docs.cortex.dev/clusters/management/production) https://github.com/cortexlabs/cortex/pull/2334 https://github.com/cortexlabs/cortex/issues/2317 ([vishalbollu](https://github.com/vishalbollu))
* Add a guide for [configuring an HTTP API Gateway](https://docs.cortex.dev/clusters/networking/api-gateway#http-api) https://github.com/cortexlabs/cortex/pull/2341 ([deliahu](https://github.com/deliahu))

**Misc**

* Add a graph of the number of active and queued requests to the Async API dashboard https://github.com/cortexlabs/cortex/pull/2326 https://github.com/cortexlabs/cortex/issues/1960 ([deliahu](https://github.com/deliahu))
* Add a graph of the number of instances to the cluster dashboard https://github.com/cortexlabs/cortex/pull/2336 https://github.com/cortexlabs/cortex/issues/2318 ([RobertLucian](https://github.com/RobertLucian))
* Ensure that `cortex cluster info --print-config` displays YAML that is consumable by `cortex cluster configure` https://github.com/cortexlabs/cortex/pull/2324 ([vishalbollu](https://github.com/vishalbollu))

**New features**

* Support autoscaling down to zero replicas for Realtime APIs https://github.com/cortexlabs/cortex/pull/2298 https://github.com/cortexlabs/cortex/issues/445 ([miguelvr](https://github.com/miguelvr))
* Allow `ssl_certificate_arn`, `api_load_balancer_cidr_white_list`, and `operator_load_balancer_cidr_white_list` to be updated on an existing cluster (via the `cortex cluster configure` command) https://github.com/cortexlabs/cortex/pull/2305 https://github.com/cortexlabs/cortex/issues/2107 ([vishalbollu](https://github.com/vishalbollu))
* Allow Prometheus's instance type to be configured ([docs](https://docs.cortex.dev/clusters/management/create#cluster-yaml)) https://github.com/cortexlabs/cortex/pull/2307 https://github.com/cortexlabs/cortex/issues/2285 ([RobertLucian](https://github.com/RobertLucian))
* Allow multiple Inferentia chips to be assigned to a single container https://github.com/cortexlabs/cortex/pull/2304 https://github.com/cortexlabs/cortex/issues/1123 ([deliahu](https://github.com/deliahu))

**Bug fixes**

* Fix cluster autoscaler's nodegroup priority calculation https://github.com/cortexlabs/cortex/pull/2309 ([RobertLucian](https://github.com/RobertLucian))

**Misc**

* Various scalability improvements https://github.com/cortexlabs/cortex/pull/2307 https://github.com/cortexlabs/cortex/pull/2304 https://github.com/cortexlabs/cortex/issues/2297 https://github.com/cortexlabs/cortex/issues/2278 https://github.com/cortexlabs/cortex/issues/2285
* Allow setting a nodegroup's `max_instances` to `0` https://github.com/cortexlabs/cortex/pull/2310 ([RobertLucian](https://github.com/RobertLucian))