Cortex

**New features**

* Support ARM instance types https://github.com/cortexlabs/cortex/pull/2268 https://github.com/cortexlabs/cortex/issues/1528 ([RobertLucian](https://github.com/RobertLucian))
* Add `cortex cluster configure` command to add, remove, or scale nodegroups on a running cluster https://github.com/cortexlabs/cortex/pull/2246 https://github.com/cortexlabs/cortex/issues/2096 ([RobertLucian](https://github.com/RobertLucian))
* Add `cortex cluster info --print-config` command to print the current configuration of a running cluster https://github.com/cortexlabs/cortex/pull/2246 ([RobertLucian](https://github.com/RobertLucian))
* Add metrics dashboard for Async APIs https://github.com/cortexlabs/cortex/pull/2242 https://github.com/cortexlabs/cortex/issues/1958 ([miguelvr](https://github.com/miguelvr))
* Support `cortex refresh` command for Async APIs https://github.com/cortexlabs/cortex/pull/2265 https://github.com/cortexlabs/cortex/issues/2237 ([deliahu](https://github.com/deliahu))

**Breaking changes**

* The `cortex cluster scale` command has been replaced by the `cortex cluster configure` command.

**Bug fixes**

* Fix Async API metrics reporting for non-200 response status codes https://github.com/cortexlabs/cortex/pull/2266 ([miguelvr](https://github.com/miguelvr))
* Make batch job metrics persistence resilient to instance termination https://github.com/cortexlabs/cortex/pull/2247 https://github.com/cortexlabs/cortex/issues/2041 ([vishalbollu](https://github.com/vishalbollu))
* Make network validations during `cortex cluster up` more permissive (to avoid unnecessarily failing checks on GovCloud) https://github.com/cortexlabs/cortex/pull/2248 ([vishalbollu](https://github.com/vishalbollu))
* Fix Inferentia resource requests https://github.com/cortexlabs/cortex/pull/2250 ([RobertLucian](https://github.com/RobertLucian))

**Docs**

* Add instructions for exporting [logs](https://docs.cortex.dev/clusters/observability/logging#exporting-logs) and [metrics](https://docs.cortex.dev/clusters/observability/metrics#exporting-metrics-to-monitoring-solutions) to external tools ([vishalbollu](https://github.com/vishalbollu))

**Misc**

* Improve output of `cortex cluster info` for running batch jobs https://github.com/cortexlabs/cortex/pull/2270 ([deliahu](https://github.com/deliahu))
* Persist Batch job metrics regardless of job status https://github.com/cortexlabs/cortex/pull/2244 ([miguelvr](https://github.com/miguelvr))
* Support creating clusters with no node groups https://github.com/cortexlabs/cortex/pull/2269 ([deliahu](https://github.com/deliahu))
* Improve handling of container startup errors in batch jobs with multiple containers https://github.com/cortexlabs/cortex/pull/2260 https://github.com/cortexlabs/cortex/issues/2217 ([vishalbollu](https://github.com/vishalbollu))
* Add CPU and memory resource requests to the proxy and dequeuer containers https://github.com/cortexlabs/cortex/pull/2252 ([deliahu](https://github.com/deliahu))

**New features**

* Support running arbitrary Docker containers in all workload types (Realtime, Async, Batch, Task) https://github.com/cortexlabs/cortex/pull/2173 ([RobertLucian](https://github.com/RobertLucian), [miguelvr](https://github.com/miguelvr), [vishalbollu](https://github.com/vishalbollu), [deliahu](https://github.com/deliahu), [ospillinger](https://github.com/ospillinger))
* Support autoscaling Async APIs to zero replicas https://github.com/cortexlabs/cortex/pull/2224 https://github.com/cortexlabs/cortex/issues/2199 ([RobertLucian](https://github.com/RobertLucian))

**Breaking changes**

* With this release, we have generalized Cortex to exclusively support running arbitrary Docker containers for all workload types (Realtime, Async, Batch, and Task). This enables the use of any model server, programming language, etc. As a result, the API configuration has been updated: the `predictor` section has been removed, the `pod` section has been added, and the `autoscaling` parameters have been modified slightly (depending on the workload type). See updated docs for [Realtime](https://docs.cortex.dev/workloads/realtime), [Async](https://docs.cortex.dev/workloads/async), [Batch](https://docs.cortex.dev/workloads/batch), and [Task](https://docs.cortex.dev/workloads/task). If you'd like to to see examples of Dockerizing Python applications, see our [test/apis](https://github.com/cortexlabs/cortex/tree/0.36/test/apis) folder.
* The `cortex prepare-debug` command has been removed; Cortex now exclusively runs Docker containers, which can be run locally via `docker run`.
* The `cortex patch` command as been removed; its behavior is now identical to `cortex deploy`.
* The `cortex logs` command now prints a CloudWatch Insights URL with a pre-populated query which can be executed to show logs from your workloads, since this is the recommended approach in production. If you wish to stream logs from a pod at random, you can use `cortex logs --random-pod` (keep in mind that these logs will not include some system logs related to your workload).
* gRPC support has been temporarily removed; we are working on adding it back in v0.37.

**Bug fixes**

* Handle exception when initializing the Python client when the default environment is not set https://github.com/cortexlabs/cortex/pull/2225 https://github.com/cortexlabs/cortex/issues/2223 ([deliahu](https://github.com/deliahu))

**Docs**

* Document how to configure SMTP in Grafana (e.g to enable email alerts) https://github.com/cortexlabs/cortex/pull/2219 ([RobertLucian](https://github.com/RobertLucian))

**Misc**

* Show CloudWatch Insights URL with a pre-populated query in the output of `cortex logs` https://github.com/cortexlabs/cortex/issues/2085 ([vishalbollu](https://github.com/vishalbollu))
* Improve efficiency of batch job submission validations https://github.com/cortexlabs/cortex/pull/2179 https://github.com/cortexlabs/cortex/issues/2178 ([deliahu](https://github.com/deliahu))

**New features**

* Avoid processing HTTP requests that have been cancelled by the client https://github.com/cortexlabs/cortex/pull/2135 https://github.com/cortexlabs/cortex/issues/1453 ([vishalbollu](https://github.com/vishalbollu))
* Support GP3 volumes (and make GP3 the default volume type) https://github.com/cortexlabs/cortex/pull/2130 https://github.com/cortexlabs/cortex/issues/1843 ([RobertLucian](https://github.com/RobertLucian))
* Allow setting the shared memory (shm) size for Task APIs https://github.com/cortexlabs/cortex/pull/2132 https://github.com/cortexlabs/cortex/issues/2115 ([RobertLucian](https://github.com/RobertLucian))
* Implement automatic 7-day expiration for Async API responses https://github.com/cortexlabs/cortex/pull/2151 ([RobertLucian](https://github.com/RobertLucian))
* Add `cortex env rename` command https://github.com/cortexlabs/cortex/pull/2165 https://github.com/cortexlabs/cortex/issues/1773 ([deliahu](https://github.com/deliahu))

**Breaking changes**

* The Python client methods which deploy Python classes have been separated from the `deploy()` method. Now, `deploy()` is used only to deploy project folders, and `deploy_realtime_api()`, `deploy_async_api()`, `deploy_batch_api()`, and `deploy_task_api()` are for deploying Python classes. ([docs](https://docs.cortex.dev/clients/python))
* The name of the bucket that Cortex uses for internal purposes is no longer configurable. During cluster creation, Cortex will auto-generate the bucket name (and create the bucket if it doesn't exist). During cluster deletion, the bucket will be emptied (unless the `--keep-aws-resources` flag is provided to `cortex cluster down`). Users' files should not be stored in the Cortex internal bucket.

**Bug fixes**

* Fix the number of Async API replicas shown in `cortex cluster info` https://github.com/cortexlabs/cortex/pull/2140 https://github.com/cortexlabs/cortex/issues/2129 ([RobertLucian](https://github.com/RobertLucian))

**Misc**

* Delete all cortex-created AWS resources when deleting a cluster, and support the `--keep-aws-resources` flag with `cortex cluster down` to preserve AWS resources https://github.com/cortexlabs/cortex/pull/2161 https://github.com/cortexlabs/cortex/issues/1612 ([RobertLucian](https://github.com/RobertLucian))
* Validate the user's AWS service quota for number of security groups and in/out rules during cluster creation https://github.com/cortexlabs/cortex/pull/2127 https://github.com/cortexlabs/cortex/issues/2087 ([RobertLucian](https://github.com/RobertLucian))
* Allow specifying only one of `--min-instances` or `--max-instances` with `cortex cluster scale` https://github.com/cortexlabs/cortex/pull/2149 ([RobertLucian](https://github.com/RobertLucian))
* Use 405 status code for un-implemented Realtime API methods https://github.com/cortexlabs/cortex/pull/2158 ([RobertLucian](https://github.com/RobertLucian))
* Decrease file size and project size limits https://github.com/cortexlabs/cortex/pull/2152 ([deliahu](https://github.com/deliahu))
* Set the default environment name to the cluster name when creating a cluster https://github.com/cortexlabs/cortex/pull/2164 https://github.com/cortexlabs/cortex/issues/1546 ([deliahu](https://github.com/deliahu))

**New features**

* Support handling `GET`, `PUT`, `PATCH`, and `DELETE` HTTP requests in Realtime APIs ([docs](https://docs.cortex.dev/workloads/realtime-apis/handler#http)) https://github.com/cortexlabs/cortex/pull/2111 https://github.com/cortexlabs/cortex/issues/2063 ([RobertLucian](https://github.com/RobertLucian))
* Support running realtime API containers locally for debugging / development purposes ([docs](https://docs.cortex.dev/workloads/debugging)) https://github.com/cortexlabs/cortex/pull/2112 https://github.com/cortexlabs/cortex/issues/2077 ([vishalbollu](https://github.com/vishalbollu))
* Support multiple gRPC services / methods (which can be named arbitrarily) in a single Realtime API ([docs](https://docs.cortex.dev/workloads/realtime-apis/handler#grpc)) https://github.com/cortexlabs/cortex/pull/2111 https://github.com/cortexlabs/cortex/issues/2063 ([RobertLucian](https://github.com/RobertLucian))
* Support specifying a list of node groups on which a workload is allowed to run (see configuration docs for [Realtime](https://docs.cortex.dev/workloads/realtime-apis/configuration), [Async](https://docs.cortex.dev/workloads/async-apis/configuration), [Batch](https://docs.cortex.dev/workloads/batch-apis/configuration), or [Task](https://docs.cortex.dev/workloads/task-apis/configuration) APIs) https://github.com/cortexlabs/cortex/pull/2098 https://github.com/cortexlabs/cortex/issues/2034 ([RobertLucian](https://github.com/RobertLucian))
* Support AWS GovCloud regions https://github.com/cortexlabs/cortex/pull/2118 https://github.com/cortexlabs/cortex/issues/2103 ([vishalbollu](https://github.com/vishalbollu))

**Breaking changes**

* "predictor" has been renamed to "handler" throughout the product (API configuration and Python APIs). In addition, as a result of supporting additional HTTP method verbs, `predict()` has been renamed to `handle_post()` in Realtime APIs (`handle_get()`, `handle_put()`, `handle_patch()`, and `handle_delete()` are now also supported). For consistency, `predict()` has been renamed to `handle_async()` for Async APIs, and `handle_batch()` for Batch APIs. See the examples for [Realtime](https://docs.cortex.dev/workloads/realtime-apis/example), [Async](https://docs.cortex.dev/workloads/async-apis/example), and [Batch](https://docs.cortex.dev/workloads/batch-apis/example) APIs. Task APIs have not been changed.

**Bug fixes**

* Fix invalid Async workload status during processing https://github.com/cortexlabs/cortex/pull/2106 https://github.com/cortexlabs/cortex/issues/2104 ([RobertLucian](https://github.com/RobertLucian))

**Docs**

* Add docs for [configuring Grafana alerts](https://docs.cortex.dev/clusters/observability/alerting) ([RobertLucian](https://github.com/RobertLucian))
* Document how to [create a Cortex cluster without administrator IAM access](https://docs.cortex.dev/clusters/management/auth#minimum-iam-policy) ([vishalbollu](https://github.com/vishalbollu))
* Add docs for [mirroring Cortex's docker images to a private repo](https://docs.cortex.dev/clusters/advanced/self-hosted-images) ([vishalbollu](https://github.com/vishalbollu))

**Misc**

* Support json output for the `cortex cluster info` command https://github.com/cortexlabs/cortex/pull/2089 https://github.com/cortexlabs/cortex/issues/2062 ([RobertLucian](https://github.com/RobertLucian))
* Allow nodegroups to be scaled down to `max_instances` == 0 https://github.com/cortexlabs/cortex/pull/2095 ([deliahu](https://github.com/deliahu))

**New features**

* Allow specifying a CIDR range whitelist for APIs and the operator ([docs](https://docs.cortex.dev/clusters/management/create)) https://github.com/cortexlabs/cortex/pull/2071 https://github.com/cortexlabs/cortex/issues/2003 ([vishalbollu](https://github.com/vishalbollu))
* Enable CORS for async, batch, and task APIs https://github.com/cortexlabs/cortex/pull/2082 https://github.com/cortexlabs/cortex/issues/2073 ([deliahu](https://github.com/deliahu))

**Breaking changes**

* The onnx predictor type has been replaced by the python predictor type; please use the python predictor type instead (all onnx models are fully supported by the python predictor type)

**Bug fixes**

* Fix bug affecting async api consistency during heavy traffic https://github.com/cortexlabs/cortex/pull/2072 ([RobertLucian](https://github.com/RobertLucian))
* Fix bug affecting async api updates https://github.com/cortexlabs/cortex/pull/2067 ([vishalbollu](https://github.com/vishalbollu))

**Misc**

* Rename `cortex cluster configure` command to `cortex cluster scale` https://github.com/cortexlabs/cortex/pull/2040 https://github.com/cortexlabs/cortex/issues/1972 ([RobertLucian](https://github.com/RobertLucian))
* Disable AZRebalance autoscaling group process https://github.com/cortexlabs/cortex/pull/2042 https://github.com/cortexlabs/cortex/issues/1349 ([RobertLucian](https://github.com/RobertLucian))
* Add horizontal pod autoscaler to async API gateway https://github.com/cortexlabs/cortex/pull/2079 https://github.com/cortexlabs/cortex/issues/2078 ([RobertLucian](https://github.com/RobertLucian))
* Rename async modules to `async_api` to avoid name collision with the reserved keyword in Python 3.7+ https://github.com/cortexlabs/cortex/pull/2066 https://github.com/cortexlabs/cortex/issues/2052 ([vishalbollu](https://github.com/vishalbollu))
* Backup images to dockerhub https://github.com/cortexlabs/cortex/pull/2081 ([vishalbollu](https://github.com/vishalbollu))
* Add additional debugging info for `cluster up` failures https://github.com/cortexlabs/cortex/pull/2080 https://github.com/cortexlabs/cortex/issues/2027 ([vishalbollu](https://github.com/vishalbollu))

**New features**

* Add gRPC support to realtime APIs ([docs](https://docs.cortex.dev/workloads/realtime-apis/predictors#grpc)) https://github.com/cortexlabs/cortex/pull/1997 https://github.com/cortexlabs/cortex/issues/1056 ([RobertLucian](https://github.com/RobertLucian))
* Add support for ONNX and TensorFlow predictor types in async APIs ([docs](https://docs.cortex.dev/workloads/async-apis/predictors)) https://github.com/cortexlabs/cortex/pull/1996 https://github.com/cortexlabs/cortex/issues/1980 ([miguelvr](https://github.com/miguelvr))
* Support using ECR images from other AWS accounts and regions https://github.com/cortexlabs/cortex/pull/2011 https://github.com/cortexlabs/cortex/issues/1988 ([vishalbollu](https://github.com/vishalbollu))

**Breaking changes**

* GCP support has been removed so that we can focus our efforts on improving the scalability, reliability, and security for Cortex on AWS. Cortex on GCP will still be available in v0.31. If you are currently using Cortex on GCP, our team will be happy to help you migrate to AWS or work with you to find alternative solutions. Please feel free to reach out to us on [slack](https://community.cortex.dev/) or email us at hellocortex.dev if you're interested.

**Bug fixes**

* Fix memory plots on Grafana dashboards for realtime and batch APIs https://github.com/cortexlabs/cortex/pull/2024 https://github.com/cortexlabs/cortex/pull/2014 https://github.com/cortexlabs/cortex/issues/1970 ([RobertLucian](https://github.com/RobertLucian))

**Docs**

* Misc docs improvements https://github.com/cortexlabs/cortex/pull/1994 ([ospillinger](https://github.com/ospillinger))

**Misc**

* Increase kubelet's `registryPullQPS` limit from 5 to 10 https://github.com/cortexlabs/cortex/pull/2023 https://github.com/cortexlabs/cortex/issues/1989 ([miguelvr](https://github.com/miguelvr))
* Pin the AMI version https://github.com/cortexlabs/cortex/pull/2010 https://github.com/cortexlabs/cortex/issues/1975 https://github.com/cortexlabs/cortex/issues/1615 ([vishalbollu](https://github.com/vishalbollu))