Codalab

Features
Worker
* Workers now have a `--exit-when-idle` flag which kills the worker process if it is ever idle after checking in with the server. This is part of an ongoing effort to allow ephemeral/one-use workers in shared computation environments like Slurm-managed clusters.

Bug fixes
Worksheets web app
* Fixed a bug where consecutive Markdown blocks were not merged, leading to rendering issues

Worker
* Fixed worker dependency manager initialization so the worker can correctly pick up existing cached dependencies upon restart
* Fixed a bug in worker dependency manager state saving that prevented any dependency from getting cleaned up, eventually making the dependency manager hit the disk limit and hang all dependency downloads.
* Fixed a bug where if the Docker container for a run cannot be started, the worker would forever keep attempting instead of failing the bundle. The bundles are failed immediately now if their Docker containers cannot be started.

Bundle server
* Fixed a bug that failed worker check-ins if the check-in included any run that had since been deleted server-side. Now the deleted run is ignored.

Misc changes
* codalab-cli python code is now auto formatted with [Black](https://github.com/ambv/black) with the exception of string normalization and a more liberal 100 character line limit to reduce the amount of formatting changes
* The repo comes with a `pyproject.toml` file to configure Black with these settings and a `.pre-commit-config.yaml` file to run Black before each commit if you have pre-commit installed.
* This also means CodaLab code is now written in a Python3 compatible syntax, but no further work has been done for Python3 compatibility beyond fixing the syntax differences.
* Updated package dependency versions to newer versions as long as there were no breaking changes, mainly as the previous version of gunicorn used had security vulnerabilities.

This version has mainly new worker focused bugfixes as well as a couple tweaks to the CLI
Server Bugfixes & Tweaks
* Fails bundles with killed dependencies unless `--allow-failed-dependencies` specified to fix 930
* Moves bundles in FINALIZING state but that are not in `worker_run` to `worker_offline` state to fix 930

Worker Bugfixes & Tweaks
* Fixed a KeyError in LocalFileSystemDependencyManager so Workers can clean up their dependencies successfully.
* Removed the `Touched dependency` log line from LocalFileSystemDependencyManager to make logs more readable and compact

CLI Bugfixes and tweaks
* Add a missing default value to the `fields` argument of `print_user_info` to fix 928

Bugfixes

* A breaking change that made CLI <= v0.2.26 incompatible with the new server when trying to upload was fixed. As of v0.2.31 CLI <= v0.2.26 should still work fine for uploads.
* Alleviation to an existing bug where trying to include bundle tags in a table schema would break the table rendering.
* Fix to a bug that made dashboard worksheets broken
* Update PyPi dependencies so that `cl install codalab` leads to a full setup again.

If you are on v0.2.27, please update to v0.2.31 as v0.2.27 has these bugs that make it incompatible with previous or later versions.

** Warning, this version had some bugs and is now incompatible with the Public instance. If you have the CLI running against the public CodaLab server, please update to v0.2.28 or later**

Features

* `.shared` and `group=<group spec>` directives for worksheet and bundle searches. `.shared` lists any (bundle|worksheet) that any of the groups you are a member of has READ or higher permissions, while `group=<group spec>` lists all (bundles|worksheets) that the given group has READ or higher permissions to.
* Support for the new `worksheet//bundle` style bundle specs when issuing macros or mimic commands. For backwards compatibility we accept both the `worksheet/bundle` and `worksheet//bundle` forms for mimic and macro commands.

Tweaks

* A big rewrite of our Worker code means job state management will be much more robust now, and any failures much easier to debug.
* A major overhaul to worksheet rendering code to make future extensions to the front end easier.

Breaking changes

* The new worker code only works with the new server code (due to API changes). If you're running your own CodaLab instance, please make sure to update both your workers and server to version 0.2.27 to avoid failing workers. (If you have workers connected to the public CodaLab instance, please update them)

* The format of the worker state files have changed. If you're updating your worker, please delete the old `dependencies` directory and `dependencies-state.json`, `worker-state.json` and `images-state.json` files.

Features
* New admin command "workers" to retrieve info about workers.
* New admin command "ufarewell" to permanently delete a user.
* Added fields printing for "uinfo" command.
* Enabled usage of command "cat" in the web command line interface, albeit with a limit of 100 lines.
* Added ".shared" and "group=<group_spec>" args to "wls" worksheet search command, which allows you to find worksheets that are shared you through any group, or a specific group, respectively.

Tweaks
* Added a more descriptive error message when bundle content path is invalid.
* A request for 0 cpus or less will be bumped up to a request for 1 cpu.

Bugfixes
* The owner of a group cannot be removed as an admin.
* Added missing import to upload_manager.
* Fixed autocomplete for some functions that used TargetsCompleter.
* Fixed root user creation for clean install of server.

- Macros can now take named optional arguments alongside positional arguments
- Global namespacing for bundles: bundles can now be referred to with `host-worksheet//bundle-name` from other worksheets
- Resource management fixes: GPU, CPU and memory requests now act as exclusive request guarantees, and the limits are enforced properly. Removes the free-for-all memory workers
- URL and Git based bundle uploads can now be made from the Web CLI (`cl up https://example.com/test.txt`)
- `--allow-failed-dependencies` now also allows `killed` dependencies
- Fixes and improvements around worker job management: jobs are resumed better after worker restarts, and long running jobs heartbeat the server, preventing unnecessary `worker_offline` states, offline runs are removed from worker_run
- Bugfix: content bundle uploads `cl upload -c <bundle-contents>` now respect their name argument if given
- Bugfix: content-disposition headers now include `attachment` as previously they were in violation of the RFC