Csvkit

----------------------

Changes:

- Drop Python 3.3 support (end-of-life was September 29, 2017).

Improvements:

- :doc:`/scripts/csvsql` adds a :code:`--chunk-size` option to set the chunk size when batch inserting into a table.
- csvkit is tested against Python 3.7.

Fixes:

- :code:`--names` works with :code:`--skip-lines`.
- Dates and datetimes without punctuation can be parsed with :code:`--date-format` and :code:`datetime-format`.
- Error messages about column indices use 1-based numbering unless :code:`--zero` is set.
- :doc:`/scripts/csvcut` no longer errors on :code:`--delete-empty-rows` with short rows.
- :doc:`/scripts/csvjoin` no longer errors if given a single file.
- :doc:`/scripts/csvsql` supports UPDATE commands.
- :doc:`/scripts/csvstat` no longer errors on non-finite numbers.
- :doc:`/scripts/csvstat` respects all command-line arguments when :code:`--count` is set.
- :doc:`/scripts/in2csv` CSV-to-CSV conversion respects :code:`--linenumbers` when buffering.
- :doc:`/scripts/in2csv` writes XLS sheets without encoding errors in Python 2.

----------------------

Improvements:

- :doc:`/scripts/csvgrep` adds a :code:`--any-match` (:code:`-a`) flag to select rows where any column matches instead of all columns.
- :doc:`/scripts/csvjson` no longer emits a property if its value is null.
- :doc:`/scripts/csvjson` adds :code:`--type` and :code:`--geometry` options to emit non-Point GeoJSON features.
- :doc:`/scripts/csvjson` adds a :code:`--no-bbox` option to disable the calculation of a bounding box.
- :doc:`/scripts/csvjson` supports :code:`--stream` for newline-delimited GeoJSON.
- :doc:`/scripts/csvsql` adds a :code:`--unique-constraint` option to list names of columns to include in a UNIQUE constraint.
- :doc:`/scripts/csvsql` adds :code:`--before-insert` and :code:`--after-insert` options to run commands before and after the INSERT command.
- :doc:`/scripts/csvpy` reports an error message if input is provided via STDIN.
- :doc:`/scripts/in2csv` adds a :code:`--encoding-xls` option to specify the encoding of the input XLS file.
- :doc:`/scripts/in2csv` supports :code:`--no-header-row` on XLS and XLSX files.
- Suppress agate warning about column names not specified when using :code:`--no-header-row`.
- Prompt the user if additional input is expected (i.e. if no input file or piped data is provided).
- Update to `agate-excel 0.2.2 <https://agate-excel.readthedocs.io/en/latest/#changelog>`_, `agate-sql 0.5.3 <https://agate-sql.readthedocs.io/en/latest/#changelog>`_.

Fixes:

- :doc:`/scripts/csvgrep` accepts utf-8 arguments to the :code:`--match` and :code:`--regex` options in Python 2.
- :doc:`/scripts/csvjson` streams input and output only if :code:`--snifflimit` is :code:`0`.
- :doc:`/scripts/csvsql` sets a DECIMAL's precision and scale and a VARCHAR's length to avoid dialect-specific errors.
- :doc:`/scripts/csvstack` no longer opens all files at once.
- :doc:`/scripts/in2csv` respects :code:`--no-header-row` when :code:`--no-inference` is set.
- :doc:`/scripts/in2csv` CSV-to-CSV conversion streams input and output only if :code:`--snifflimit` is :code:`0`.
- :doc:`/scripts/in2csv` supports GeoJSON files with: ``geometry`` set to ``null``, missing Point ``coordinates``, altitude coordinate values.

csvkit is no longer tested on PyPy.

----------------------

Improvements:

- Add a :code:`--version` flag.
- Add a :code:`--skip-lines` option to skip initial lines (e.g. comments, copyright notices, empty rows).
- Add a :code:`--locale` option to set the locale of any formatted numbers.
- Add a :code:`--date-format` option to set a strptime date format string.
- Add a :code:`--datetime-format` option to set a strptime datetime format string.
- Make :code:`--blanks` a common argument across all tools.
- :code:`-I` is the short option for :code:`--no-inference`.
- :doc:`/scripts/csvclean`, :doc:`/scripts/csvformat`, :doc:`/scripts/csvjson`, :doc:`/scripts/csvpy` support :code:`--no-header-row`.
- :doc:`/scripts/csvclean` is faster and no longer requires exponential time in the worst case.
- :doc:`/scripts/csvformat` supports :code:`--linenumbers` and `--zero` (no-op).
- :doc:`/scripts/csvjoin` supports :code:`--snifflimit` and :code:`--no-inference`.
- :doc:`/scripts/csvpy` supports :code:`--linenumbers` (no-op) and :code:`--zero` (no-op).
- :doc:`/scripts/csvsql` adds a :code:`--prefix` option to add expressions like OR IGNORE or OR REPLACE following the INSERT keyword.
- :doc:`/scripts/csvsql` adds a :code:`--overwrite` flag to drop any existing table with the same name before creating.
- :doc:`/scripts/csvsql` accepts a file name for the :code:`--query` option.
- :doc:`/scripts/csvsql` supports :code:`--linenumbers` (no-op).
- :doc:`/scripts/csvsql` adds a :code:`--create-if-not-exists` flag to not abort if the table already exists.
- :doc:`/scripts/csvstat` adds a :code:`--freq-count` option to set the maximum number of frequent values to display.
- :doc:`/scripts/csvstat` supports :code:`--linenumbers` (no-op).
- :doc:`/scripts/in2csv` adds a :code:`--names` flag to print Excel sheet names.
- :doc:`/scripts/in2csv` adds a :code:`--write-sheets` option to write the named Excel sheets to files.
- :doc:`/scripts/sql2csv` adds an :code:`--encoding` option to specify the encoding of the input query file.

Fixes:

- :doc:`/scripts/csvgrep` no longer ignores common arguments if :code:`--linenumbers` is set.
- :doc:`/scripts/csvjson` supports Decimal.
- :doc:`/scripts/csvpy` again supports IPython.
- :doc:`/scripts/csvsql` restores support for :code:`--no-constraints` and :code:`--db-schema`.
- :doc:`/scripts/csvstat` no longer crashes when :code:`--freq` is set.
- :doc:`/scripts/in2csv` restores support for :code:`--no-inference` for Excel files.
- :doc:`/scripts/in2csv` restores support for converting Excel files from standard input.
- :doc:`/scripts/in2csv` accepts utf-8 arguments to the :code:`--sheet` option in Python 2.

-------------------------

This is a minor release which fixes several bugs reported in the :code:`1.0.0` release earlier this week. It also significantly improves the output of :doc:`/scripts/csvstat` and adds a :code:`--csv` output option to that command.

- :doc:`/scripts/csvstat` no longer crashes when a :code:`Number` column has :code:`None` as a frequent value. (738)
- :doc:`/scripts/csvlook` documents that output tables are Markdown-compatible. (734)
- :doc:`/scripts/csvstat` adds a :code:`--csv` flag for tabular output. (584)
- :doc:`/scripts/csvstat` output is easier to read. (714)
- :doc:`/scripts/csvpy` has a better description when using the :code:`--agate` flag. (729)
- Fix a Python 2.6 bug preventing :doc:`/scripts/csvjson` from parsing utf-8 files. (732)
- Update required version of unittest to latest. (727)

-------------------------

This is the first major release of csvkit in a very long time. The entire backend has been rewritten to leverage the `agate <https://agate.rtfd.io>`_ data analysis library, which was itself inspired by csvkit. The new backend provides better type detection accuracy, as well as some new features.

Because of the long and complex cycle behind this release, the list of changes should not be considered exhaustive. In particular, the output format of some tools may have changed in small ways. Any existing data pipelines using csvkit should be tested as part of the upgrade.

Much of the credit for this release goes to `James McKinney <https://github.com/jpmckinney>`_, who has almost single-handedly kept the csvkit fire burning for a year. Thanks, James!

Backwards-incompatible changes:

- :doc:`/scripts/csvjoin` renames duplicate columns with integer suffixes to prevent collisions in output.
- :doc:`/scripts/csvsql` generates ``DateTime`` columns instead of ``Time`` columns.
- :doc:`/scripts/csvsql` generates ``Decimal`` columns instead of ``Integer``, ``BigInteger``, and ``Float`` columns.
- :doc:`/scripts/csvsql` no longer generates max-length constraints for text columns.
- The ``--doublequote`` long flag is gone, and the ``-b`` short flag is an alias for ``--no-doublequote``.
- When using the ``--columns`` or ``--not-columns`` options, you must not have spaces around the comma-separated values, unless the column names contain spaces.
- When sorting, null values are greater than other values instead of less than.
- ``CSVKitReader``, ``CSVKitWriter``, ``CSVKitDictReader``, and ``CSVKitDictWriter`` have been removed. Use ``agate.csv.reader``, ``agate.csv.writer``, ``agate.csv.DictReader`` and ``agate.csv.DictWriter``.
- Drop Python 2.6 support (end-of-life was October 29, 2013).
- Drop support for older versions of PyPy.
- If ``--no-header-row`` is set, the output has column names ``a``, ``b``, ``c``, etc. instead of ``column1``, ``column2``, ``column3``, etc.
- csvlook renders a simpler, markdown-compatible table.

Improvements:

- csvkit is tested against Python 3.6. (702)
- ``import csvkit as csv`` defers to agate readers/writers.
- :doc:`/scripts/csvgrep` supports ``--no-header-row``.
- :doc:`/scripts/csvjoin` supports ``--no-header-row``.
- :doc:`/scripts/csvjson` streams input and output if the ``--stream`` and ``--no-inference`` flags are set.
- :doc:`/scripts/csvjson` supports ``--snifflimit`` and ``--no-inference``.
- :doc:`/scripts/csvlook` adds ``--max-rows``, ``--max-columns`` and ``--max-column-width`` options.
- :doc:`/scripts/csvlook` supports ``--snifflimit`` and ``--no-inference``.
- :doc:`/scripts/csvpy` supports ``--agate`` to read a CSV file into an agate table.
- ``csvsql`` supports custom `SQLAlchemy dialects <https://docs.sqlalchemy.org/en/latest/dialects/>`_.
- :doc:`/scripts/csvstat` supports ``--names``.
- :doc:`/scripts/in2csv` CSV-to-CSV conversion streams input and output if the ``--no-inference`` flag is set.
- :doc:`/scripts/in2csv` CSV-to-CSV conversion uses ``agate.Table``.
- :doc:`/scripts/in2csv` GeoJSON conversion adds columns for geometry type, longitude and latitude.
- Documentation: Update tool usage, remove shell prompts, document connection string, correct typos.

Fixes:

- Fixed numerous instances of open files not being closed before utilities exit.
- Change ``-b``, ``--doublequote`` to ``--no-doublequote``, as doublequote is True by default.
- :doc:`/scripts/in2csv` DBF conversion works with Python 3.
- :doc:`/scripts/in2csv` correctly guesses format when file has an uppercase extension.
- :doc:`/scripts/in2csv` correctly interprets ``--no-inference``.
- :doc:`/scripts/in2csv` again supports nested JSON objects (fixes regression).
- :doc:`/scripts/in2csv` with ``--format geojson`` prints a JSON object instead of ``OrderedDict([(...)])``.
- :doc:`/scripts/csvclean` with standard input works on Windows.
- :doc:`/scripts/csvgrep` returns the input file's line numbers if the ``--linenumbers`` flag is set.
- :doc:`/scripts/csvgrep` can match multiline values.
- :doc:`/scripts/csvgrep` correctly operates on ragged rows.
- :doc:`/scripts/csvsql` correctly escapes ``% characters in SQL queries.
- :doc:`/scripts/csvsql` adds standard input only if explicitly requested.
- :doc:`/scripts/csvstack` supports stacking a single file.
- :doc:`/scripts/csvstat` always reports frequencies.
- The ``any_match`` argument of ``FilteringCSVReader`` works correctly.
- All tools handle empty files without error.

----------------------

- Add Antonio Lima to AUTHORS.
- Add support for ndjson. (329)
- Add missing docs for csvcut -C. (227)
- Reorganize docs so TOC works better. (339)
- Render docs locally with RTD theme.
- Fix header in "tricks" docs.
- Add install instructions to tutorial. (331)
- Add killer examples to doc index. (328)
- Reorganize doc index
- Fix broken csvkit module documentation. (327)
- Fix version of openpyxl to work around encoding issue. (391, 288)