Cortex

**Bug fixes**

* Preemptible node pools on GCP aren't autoscaling https://github.com/cortexlabs/cortex/pull/1981 ([vishalbollu](https://github.com/vishalbollu))
* Replica autoscaler targets incorrect deployments on operator restart https://github.com/cortexlabs/cortex/pull/1982 ([miguelvr](https://github.com/miguelvr))
* Replica autoscaler is not reinitialized for running APIs on operator restart on GCP https://github.com/cortexlabs/cortex/pull/1984 ([vishalbollu](https://github.com/vishalbollu))

**New features**

* Add support for AsyncAPI (experimental) ([docs](https://docs.cortex.dev/workloads/introduction)) https://github.com/cortexlabs/cortex/pull/1935 https://github.com/cortexlabs/cortex/issues/1610 ([miguelvr](https://github.com/miguelvr))
* Add support for multi-instance-type clusters to AWS/GCP providers (experimental) ([aws](https://docs.cortex.dev/clusters/aws/multi-instance-type)/[gcp](https://docs.cortex.dev/clusters/gcp/multi-instance-type) docs) https://github.com/cortexlabs/cortex/pull/1951 ([RobertLucian](https://github.com/RobertLucian))
* Allow users to duplicate/mirror traffic using shadow pipelines https://github.com/cortexlabs/cortex/pull/1948 https://github.com/cortexlabs/cortex/issues/1889 ([docs](https://docs.cortex.dev/workloads/realtime-apis/traffic-splitter/configuration)) ([vishalbollu](https://github.com/vishalbollu))

**Breaking changes**

* `on_demand_backup` in cluster configuration has been removed in favour of using a cluster with a mixture of spot and on-demand nodegroups. See multi-instance documentation for [aws](https://docs.cortex.dev/clusters/aws/multi-instance-type) and [gcp](https://docs.cortex.dev/clusters/gcp/multi-instance-type) for more details.

**Bug fixes**

* Fix Python client not respecting CORTEX_CLI_CONFIG_DIR environment variable for client-id.txt https://github.com/cortexlabs/cortex/pull/1953 ([jackmpcollins](https://github.com/jackmpcollins))
* Prevent threads from being stuck in DynamicBatcher https://github.com/cortexlabs/cortex/pull/1915 ([cbensimon](https://github.com/cbensimon))
* Fix unexpected cortex logs termination by increasing buffer size https://github.com/cortexlabs/cortex/pull/1939 ([vishalbollu](https://github.com/vishalbollu))
* Decouple cluster deletion from EBS volume deletion for cortex cluster down https://github.com/cortexlabs/cortex/pull/1954 ([deliahu](https://github.com/deliahu))
* Fix spot/on-demand GPU instances not joining the cluster by upgrading to eksctl 0.40.0 https://github.com/cortexlabs/cortex/pull/1955 ([vishalbollu](https://github.com/vishalbollu))
* Prevent premature queue not found errors by preserving the SQS for minutes till after the job has completed https://github.com/cortexlabs/cortex/pull/1952 ([vishalbollu](https://github.com/vishalbollu))

**Docs**

* Update docs https://github.com/cortexlabs/cortex/pull/1949 ([ospillinger](https://github.com/ospillinger))

**Misc**

* Configure a default cortex client to manage APIs from with cortex workloads https://github.com/cortexlabs/cortex/pull/1942 https://github.com/cortexlabs/cortex/issues/1644 ([RobertLucian](https://github.com/RobertLucian))
* Save batch metrics to cloud to preserve job metrics history https://github.com/cortexlabs/cortex/pull/1940 ([vishalbollu](https://github.com/vishalbollu))

**New features**

* Record custom metrics from predictors and view them in Grafana ([docs](https://docs.cortex.dev/workloads/observability/metrics#custom-user-metrics)) https://github.com/cortexlabs/cortex/pull/1910 https://github.com/cortexlabs/cortex/issues/1897 ([miguelvr](https://github.com/miguelvr))
* Add granular pod metrics to the Grafana dashboards https://github.com/cortexlabs/cortex/pull/1905 ([RobertLucian](https://github.com/RobertLucian))
* Add node metrics to Grafana dashboards https://github.com/cortexlabs/cortex/pull/1900 ([miguelvr](https://github.com/miguelvr))

**Breaking changes**

* Remove support for installing Cortex on your own Kubernetes Cluster https://github.com/cortexlabs/cortex/pull/1921 ([RobertLucian](https://github.com/RobertLucian))

**Bug fixes**

* Fix bug where successfully completed jobs were marked as completed with errors https://github.com/cortexlabs/cortex/pull/1913 ([vishalbollu](https://github.com/vishalbollu))
* Fix bug where batch jobs were being terminated unnecessarily https://github.com/cortexlabs/cortex/pull/1917 ([vishalbollu](https://github.com/vishalbollu))
* Prevent cluster autoscaler from reallocating job pods https://github.com/cortexlabs/cortex/pull/1919 ([vishalbollu](https://github.com/vishalbollu))
* Address AWS cluster up quota issues such not enough NAT Gateways or EIPs https://github.com/cortexlabs/cortex/pull/1912 ([RobertLucian](https://github.com/RobertLucian))
* Delete unused prometheus volume on cluster down https://github.com/cortexlabs/cortex/pull/1863 ([miguelvr](https://github.com/miguelvr))
* Create .cortex dir if not present https://github.com/cortexlabs/cortex/pull/1909 ([RobertLucian](https://github.com/RobertLucian))

**Docs**

* Add docs for accessing dashboard through private load balancer ([docs](https://docs.cortex.dev/workloads/observability/metrics#accessing-the-dashboard)) https://github.com/cortexlabs/cortex/pull/1907 ([deliahu](https://github.com/deliahu))

**Misc**

* Allow specifying paths for requirements.txt, conda-packages.txt & dependencies.sh ([docs](https://docs.cortex.dev/workloads/dependencies/python-packages#customizing-dependency-paths)) https://github.com/cortexlabs/cortex/pull/1896 https://github.com/cortexlabs/cortex/pull/1927 https://github.com/cortexlabs/cortex/issues/1777 ([miguelvr](https://github.com/miguelvr))
* Log relevant kubernetes events to API specific log streams https://github.com/cortexlabs/cortex/pull/1906 https://github.com/cortexlabs/cortex/issues/833 ([miguelvr](https://github.com/miguelvr))
* Support credentials using AWS_SESSION_TOKEN with the CLI/Client ([docs](https://docs.cortex.dev/clusters/aws/auth#cortex-client)) https://github.com/cortexlabs/cortex/pull/1908 https://github.com/cortexlabs/cortex/pull/1920 https://github.com/cortexlabs/cortex/issues/1134 https://github.com/cortexlabs/cortex/issues/1865 ([vishalbollu](https://github.com/vishalbollu))
* Provide auth to Operator and APIs by attaching IAM policies to the cluster ([docs](https://docs.cortex.dev/clusters/aws/auth#authorizing-your-apis)) https://github.com/cortexlabs/cortex/pull/1908 https://github.com/cortexlabs/cortex/issues/1858 ([vishalbollu](https://github.com/vishalbollu))

**New features**

* Add Grafana dashboard for APIs ([docs](https://www.docs.cortex.dev/workloads/realtime-apis/metrics)) https://github.com/cortexlabs/cortex/pull/1867 https://github.com/cortexlabs/cortex/pull/1885 https://github.com/cortexlabs/cortex/pull/1890 https://github.com/cortexlabs/cortex/pull/1887 ([miguelvr](https://github.com/miguelvr))
* Support API autoscaling in GCP clusters ([docs](https://www.docs.cortex.dev/workloads/realtime-apis/autoscaling)) https://github.com/cortexlabs/cortex/pull/1814 https://github.com/cortexlabs/cortex/pull/1879 https://github.com/cortexlabs/cortex/issues/1601 ([miguelvr](https://github.com/miguelvr))
* Support traffic splitting in GCP clusters ([docs](https://www.docs.cortex.dev/workloads/realtime-apis/traffic-splitter/example)) https://github.com/cortexlabs/cortex/pull/1892 https://github.com/cortexlabs/cortex/issues/1660 ([miguelvr](https://github.com/miguelvr))

**Breaking changes**

* The default Docker images for APIs have been slimmed down to not include packages other than what Cortex requires to function. Therefore, when deploying APIs, it is now necessary to include the dependencies that your predictor needs in `requirements.txt` ([docs](https://www.docs.cortex.dev/workloads/dependencies/python-packages)) and/or `dependencies.sh` ([docs](https://www.docs.cortex.dev/workloads/dependencies/system-packages)).

**Bug fixes**

* Disable dynamic batcher for TensorFlow predictor type https://github.com/cortexlabs/cortex/pull/1888 ([miguelvr](https://github.com/miguelvr))
* Support empty directory objects for models saved in S3/GCS https://github.com/cortexlabs/cortex/pull/1830 https://github.com/cortexlabs/cortex/issues/1829 ([RobertLucian](https://github.com/RobertLucian))
* Fix bug which prevented Task APIs on GCP from being cleaned up after completion https://github.com/cortexlabs/cortex/pull/1871 ([RobertLucian](https://github.com/RobertLucian))

**Docs**

* Add documentation for using a version of Python other than the default via `dependencies.sh` ([docs](https://www.docs.cortex.dev/workloads/dependencies/system-packages)) or custom images ([docs](https://www.docs.cortex.dev/workloads/dependencies/images)) https://github.com/cortexlabs/cortex/pull/1862 https://github.com/cortexlabs/cortex/issues/1779 ([RobertLucian](https://github.com/RobertLucian))

**Misc**

* Support deploying predictor Python classes from more environments (e.g. from separate Python files, AWS Lambda) https://github.com/cortexlabs/cortex/pull/1883 https://github.com/cortexlabs/cortex/commit/3a1b777d06e660a49b6223badda4c5e8b1fe4ec1 https://github.com/cortexlabs/cortex/issues/1824 https://github.com/cortexlabs/cortex/issues/1826 ([vishalbollu](https://github.com/vishalbollu))
* Improve error logging for Batch and Task APIs https://github.com/cortexlabs/cortex/pull/1866 https://github.com/cortexlabs/cortex/issues/1833 ([RobertLucian](https://github.com/RobertLucian))

**New features**

* Support installing Cortex on an existing Kubernetes cluster (on AWS or GCP) ([docs](https://docs.cortex.dev/clusters/cortex-core-on-kubernetes/install)) https://github.com/cortexlabs/cortex/pull/1837 https://github.com/cortexlabs/cortex/issues/1808 ([vishalbollu](https://github.com/vishalbollu))

**Breaking changes**

* The cloudwatch dashboard has been removed as a result of our switch to Prometheus for metrics aggregation. The dashboard will be replaced with an alternative in an upcoming release.

**Bug fixes**

* Fix bug which can cause requests to APIs from a Python client to timeout during cluster autoscaling https://github.com/cortexlabs/cortex/pull/1841 https://github.com/cortexlabs/cortex/issues/1840 ([RobertLucian](https://github.com/RobertLucian))
* Fix bug which can cause `downscale_stabilization_period` to be disregarded during downscaling https://github.com/cortexlabs/cortex/pull/1847 https://github.com/cortexlabs/cortex/issues/1846 ([RobertLucian](https://github.com/RobertLucian))

**Misc**

* AWS credentials are no longer required to connect the CLI to the cluster operator. If you need to restrict access to your cluster operator, configure the operator's load balancer to be private by setting `operator_load_balancer_scheme: internal` in your [cluster configuration file](https://docs.cortex.dev/clusters/cortex-cloud-on-aws/install#configure-cortex), and set up [VPC Peering](https://docs.cortex.dev/clusters/cortex-cloud-on-aws/index/vpc-peering). We plan in supporting a new auth strategy in an upcoming release.
* Improve S6 error code/signal handling https://github.com/cortexlabs/cortex/pull/1825 https://github.com/cortexlabs/cortex/issues/1703 ([RobertLucian](https://github.com/RobertLucian))

**New features**

* Add new API type `TaskAPI` for running arbitrary Python jobs ([docs](https://docs.cortex.dev/workloads/task/example)) https://github.com/cortexlabs/cortex/pull/1717 https://github.com/cortexlabs/cortex/issues/253 ([miguelvr](https://github.com/miguelvr), [RobertLucian](https://github.com/RobertLucian))
* Write Cortex's logs as structured logs, and allow use of Cortex's structured logger in predictors (supports adding extra fields) ([aws docs](https://docs.cortex.dev/clusters/aws/logging), [gcp docs](https://docs.cortex.dev/clusters/gcp/logging)) https://github.com/cortexlabs/cortex/pull/1778 https://github.com/cortexlabs/cortex/pull/1803 https://github.com/cortexlabs/cortex/pull/1804 https://github.com/cortexlabs/cortex/issues/1732 https://github.com/cortexlabs/cortex/issues/1563 ([vishalbollu](https://github.com/vishalbollu))
* Support preemptible instances on GCP ([docs](https://docs.cortex.dev/clusters/gcp/install)) https://github.com/cortexlabs/cortex/pull/1791 https://github.com/cortexlabs/cortex/issues/1631 ([RobertLucian](https://github.com/RobertLucian))
* Support private load balancers on GCP ([docs](https://docs.cortex.dev/clusters/gcp/install)) https://github.com/cortexlabs/cortex/pull/1786 https://github.com/cortexlabs/cortex/issues/1621 ([deliahu](https://github.com/deliahu))
* Support GCP instances with multiple GPUs ([docs](https://docs.cortex.dev/clusters/gcp/install)) https://github.com/cortexlabs/cortex/pull/1789 https://github.com/cortexlabs/cortex/issues/1784 ([deliahu](https://github.com/deliahu))

**Breaking changes**

* `cortex logs` now streams logs from a single replica at random when there are multiple replicas for an API. The recommended way to analyze production logs is via a dedicated logging tool (by default, logs are sent to [CloudWatch](https://us-west-2.console.aws.amazon.com/cloudwatch/home) on AWS and [StackDriver](https://console.cloud.google.com/logs/query) on GCP)

**Bug fixes**

* Misc Python client fixes https://github.com/cortexlabs/cortex/pull/1798 https://github.com/cortexlabs/cortex/pull/1782 https://github.com/cortexlabs/cortex/pull/1772 ([vishalbollu](https://github.com/vishalbollu), [RobertLucian](https://github.com/RobertLucian))

**Docs**

* Document the shared `/mnt` directory for TensorFlow predictors https://github.com/cortexlabs/cortex/pull/1802 https://github.com/cortexlabs/cortex/issues/1792 ([deliahu](https://github.com/deliahu))
* Misc GCP docs improvements https://github.com/cortexlabs/cortex/pull/1799 ([deliahu](https://github.com/deliahu))

**Misc**

* Improve out-of-memory status reporting ([RobertLucian](https://github.com/RobertLucian))
* Improve batch job cleanup process https://github.com/cortexlabs/cortex/pull/1797 https://github.com/cortexlabs/cortex/pull/1796 ([vishalbollu](https://github.com/vishalbollu))
* Remove grpc msg send/receive limit https://github.com/cortexlabs/cortex/pull/1769 https://github.com/cortexlabs/cortex/issues/1740 ([RobertLucian](https://github.com/RobertLucian))