Visidata

- commands overhaul; see `commands.tsv` (command longnames should now be largely stable)
- add quantile aggregators (q3/q4/q5/q10)
- add `z;` to add new column from bash *expr*, with `$`columnNames as variables
- keyboard layout (thanks to deinspanjer for the inspiration)
- `O` launches sheet-specific options (see design/169.md); `gO` launches global OptionsSheet
- options.wrap now defaults to False
- options.quitguard enables confirmation before quitting last sheet
- options.safety_first makes loading/saving more robust, at the cost of performance
- currently only removing NULs from csv input
- dedup, sort, color status messages by "priority" (thanks to jsvine for suggestion)
- remove menu system
- can now edit source values from FreqSheet

- Command changes
- `^H` is now main command to open the manpage! `z^H` opens a list of all commands for this sheet.
- `R` (`random-sheet`) pushes sheet instead of selecting (reverting to former behavior)
- `za` (`addcol-empty`) asks for column name
- `zd` (`delete-cell`) moves value to clipboard ("cut", like other delete commands)
- add `gI` (`describe-all`) like `gC` (`columns-all`)
- add `gS` (`sheets-graveyard`)
- add `g(`, `z(`, `gz(` variants of `(` 'expand-column'
- add `z|` and `z\` to un/select by python expr (thanks to jsvine for suggestion)
- add `z` to set type of current column to `len`
- add `z;` to get the stdout/stderr from running a cmdline program with $colname args
- `Space` is now bound to exec-longname (was `menu`; `^A` was exec-longname previously)

- Loaders:
- add pandas adapter
- add xml loader
- add pcap loader (thanks to vbrown608 and TemperedNetworks)
- add yaml loader (thanks to robcarrington, JKiely, anjakefala at PyCon Sprints for making this happen)
- add jsonl saver
- remove `tsv_safe_char` and split into `tsv_safe_newline` and `tsv_safe_tab`

- initial commit of a task warrior app (vtask)

minor changes
- more portable system clipboard handling (thanks chocolateboy for PR)
- [json] no more incremental display during loading (need better json parser than stdlib)
- `date` supports adding a number of days (or `6*hours`, `9*months`, etc)
- hidden columns are darkened on columns sheet
- exception rollup
- dev/commands.tsv table of commands
- motd default url uses https
- improve ProfileSheet
- [DirSheet] editable `mode` (set to octal like `0o0644`)

- python 3.7
- Change `async` decorator to `asyncthread` and rename `async.py` to avoid using Python 3.7 keyword

- macro system
- `gD` goes to directory browser of `options.visidata_dir` (default to `~/.visidata/`) which contains saved commandlogs and macros
- `z^S` on CommandLog saves selected rows to macro for given keystroke, saving to `.visidata/macro/command-longname.vd`
- macro list saved at `.visidata/macros.vd` (keystroke, filename)
- `options.cmdlog_histfile` for auto-appended (default: empty means disabled)
- [DirSheet] edits make deferred changes to any field
- add `directory` and `filetype` columns
- note: only 256 changes maintained per column (same as column cache size)
- `^S` saves all deferred changes
- `z^S` saves changes for the current file only
- `^R` clears all changes (reload)
- `z^R` clears changes on the current file only
- `d`/`gd` marks the current/selected file for deletion
- if `directory` is edited, on `^S` (save) file is moved (if directory not existing, a new directory is created)
- [New conda package](https://github.com/conda-forge/visidata-feedstock)
- add .visidatarc [snippets](https://github.com/saulpw/visidata/tree/stable/snippets) with examples of extra functionality
- add replayable options [97](https://github.com/saulpw/visidata/issues/97)
- `g^S` for multisave to single file (`.html`, `.md` and `.txt` are currently supported) or directory
- `z^S` to save selected rows of current column only (along with key columns)
- `T` to transpose rows and columns [129](https://github.com/saulpw/visidata/issues/129)
- `^A` to specify a command longname to execute
- `^O`/`g^O` to open current/selected files in external editor
- `g^R` on SheetsSheet to reload all [selected] sheets
- `options.error_is_null` to treat errors as nulls when applicable
- `g,` fixed to compare by visible column values, not by row objects
- `gv` to unhide all columns
- `gM` open melted sheet (unpivot) with key columns retained and *regex* capture groups determining how the non-key columns will be reduced to Variable-Value rows
- `g*` replace selected row cells in current column with regex transform
- Shift-Up/Down aliases for mac [135](https://github.com/saulpw/visidata/issues/135)
- options.wrap now true by default on TextSheet (`v` to toggle)
- `save_txt` with single column concatenates all values to single file
- `+` can add multiple aggregators
- ^X bugfix: use evalexpr over cursorRow
- `z`/`gz` `s`/`t`/`u` to select to/from cursorRow
- `z<` and `z>` to move up/down to next null cell
- `"` no longer reselects all rows
- `sheet-set-diff` command to act like `--diff`
- math functions (like sin/cos) now at toplevel
- bugfix: freeze
- all `csv_` options sent to csv.reader/writer
- `options.tsv_safe_char` to replace \t and \n in tsv files; set to empty for speedup during tsv save
- loaders and savers
- support bz2 and xz (LZMA) compression (thanks jpgrayson)
- add loaders for:
- sas7bda (SAS; requires `sas7bdat`)
- xpt (SAS; requires `xport`)
- sav (SPSS; requires `savReaderWriter`)
- dta (Stata; requires `pandas`)
- .shp can save as .geojson
- add htm as alias for html filetype
- json bugfix: fix [133](https://github.com/saulpw/visidata/issues/133) json loader column deduction
- [experimental] bin/vsh initial commit

- VisiData will be included in the [next debian repository release](https://tracker.debian.org/pkg/visidata)!
- remove all install dependencies
- additional libraries must be installed manually for certain loaders; see requirements.txt
- experimental hierarchical menu system with SPACE to explore commands
- use standard movement keys (`hjkl`/`arrows`) to navigate within a command level
- Use `Enter`/`q` to navigate down/up a command tree
- abort with `gq` or `^C`
- existing chooseOne selections (aggregators/joins) still use simple input() for now
- most longnames changed
- let me know if anyone is using any longnames at all, and we will stabilize the names
- if you do end up playing with it, please let me know what did and didn't work for you
- randomized message/announcement/tip on startup; disable with `options.motd_url = None`
- cache messages in `$HOME/.visidata/`

Command additions/changes:

- add `za` and `gza` to add 1/N new blanks column
- add `(` and `)` commands to expand/collapse list/dict columns (e.g. in nested json)
- add `Backspace` command to drop sheet like `q` and also scrub its history from the cmdlog
- [canvas] add `d` and `gd` to delete points from source sheet
- remove `!$%-_` special actions on columns sheet
- alias Shift+Arrows to `HJKL` (may not work in all environments)
- alias `ENTER` to modify-edit-cell by default
- add `Y`/`gY`/`zY` to copy row/selected/cell to system clipboard (with options.clipboard_copy_cmd)

- filename `-` works to specify stdin/stdout (`-b -o -` will dump final sheet to stdout)
- search/select uses most recent when not given any (as in vim/etc)
- annotate None with disp_note_none ('∅'); previously was not visually distinguishable from empty string

- save to .md org-mode compatible table format
- load/view/edit/save png, edit pixels in data form
- load/view ttf/otf font files
- [canvas] draw quadratic curves with qcurve([(x,y)...])
- improvements/bugfixes: pivot, describe, melt, sqlite, shp, html

- date.__sub__ returns timedelta object (was int days)
- pivot table bugfixes
- many cosmetic fixes
- disable default profiling for perf improvements
- remove .visidatarc support in PWD or XDG; only $HOME/.visidatarc supported now
- website and docs complete overhaul
- do not execute .py files
- apt/brew packages submitted

- tab completion for filename and python expr
- `v` now 'visibility toggle' (moved from `w`)
- `^W` to erase a word in the line editor
- `gC`
- `--version` (thanks to jsvine)
- `options.use_default_colors` (thanks to wavexx)
- `median` aggregator
- .html loads tables (requires lxml)
- simple http works (requires requests)
- json save
- json incremental load
- [cmdlog] use rowkey if available instead of row number; options.rowkey_prefix
- [cmdlog] only set row/col when relevant
- [vdtui] task renamed to thread
- /howto/dev/loader
- /design/graphics