Pytest-benchmark

------------------

* Added support for simple ``trial`` x-axis histogram label. Contributed by Ken Crowell in
`95 <https://github.com/ionelmc/pytest-benchmark/pull/95>`_).
* Added support for Pytest 3.3+, Contributed by Julien Nicoulaud in
`103 <https://github.com/ionelmc/pytest-benchmark/pull/103>`_.
* Added support for Pytest 4.0. Contributed by Pablo Aguiar in
`129 <https://github.com/ionelmc/pytest-benchmark/pull/129>`_ and
`130 <https://github.com/ionelmc/pytest-benchmark/pull/130>`_.
* Added support for Pytest 4.1.
* Various formatting, spelling and documentation fixes. Contributed by
Ken Crowell, Ofek Lev, Matthew Feickert, Jose Eduardo, Anton Lodder, Alexander Duryagin and Grygorii Iermolenko in
`97 <https://github.com/ionelmc/pytest-benchmark/pull/97>`_,
`105 <https://github.com/ionelmc/pytest-benchmark/pull/105>`_,
`110 <https://github.com/ionelmc/pytest-benchmark/pull/110>`_,
`111 <https://github.com/ionelmc/pytest-benchmark/pull/111>`_,
`115 <https://github.com/ionelmc/pytest-benchmark/pull/115>`_,
`123 <https://github.com/ionelmc/pytest-benchmark/pull/123>`_,
`131 <https://github.com/ionelmc/pytest-benchmark/pull/131>`_ and
`140 <https://github.com/ionelmc/pytest-benchmark/pull/140>`_.
* Fixed broken ``pytest_benchmark_update_machine_info`` hook. Contributed by Alex Ford in
`109 <https://github.com/ionelmc/pytest-benchmark/pull/109>`_.
* Fixed bogus xdist warning when using ``--benchmark-disable``. Contributed by Francesco Ballarin in
`113 <https://github.com/ionelmc/pytest-benchmark/pull/113>`_.
* Added support for pathlib2. Contributed by Lincoln de Sousa in
`114 <https://github.com/ionelmc/pytest-benchmark/pull/114>`_.
* Changed handling so you can use ``--benchmark-skip`` and ``--benchmark-only``, with the later having priority.
Contributed by Ofek Lev in
`116 <https://github.com/ionelmc/pytest-benchmark/pull/116>`_.
* Fixed various CI/testing issues.
Contributed by Stanislav Levin in
`134 <https://github.com/ionelmc/pytest-benchmark/pull/134>`_,
`136 <https://github.com/ionelmc/pytest-benchmark/pull/136>`_ and
`138 <https://github.com/ionelmc/pytest-benchmark/pull/138>`_.

------------------

* Fixed loading data from old json files (missing ``ops`` field, see
`81 <https://github.com/ionelmc/pytest-benchmark/issues/81>`_).
* Fixed regression on broken SCM (see
`82 <https://github.com/ionelmc/pytest-benchmark/issues/82>`_).

------------------

* Added "operations per second" (``ops`` field in ``Stats``) metric --
shows the call rate of code being tested. Contributed by Alexey Popravka in
`78 <https://github.com/ionelmc/pytest-benchmark/pull/78>`_.
* Added a ``time`` field in ``commit_info``. Contributed by "varac" in
`71 <https://github.com/ionelmc/pytest-benchmark/pull/71>`_.
* Added a ``author_time`` field in ``commit_info``. Contributed by "varac" in
`75 <https://github.com/ionelmc/pytest-benchmark/pull/75>`_.
* Fixed the leaking of credentials by masking the URL printed when storing
data to elasticsearch.
* Added a ``--benchmark-netrc`` option to use credentials from a netrc file when
storing data to elasticsearch. Both contributed by Andre Bianchi in
`73 <https://github.com/ionelmc/pytest-benchmark/pull/73>`_.
* Fixed docs on hooks. Contributed by Andre Bianchi in `74 <https://github.com/ionelmc/pytest-benchmark/pull/74>`_.
* Remove ``git`` and ``hg`` as system dependencies when guessing the project name.

--------------------

* ``machine_info`` now contains more detailed information about the CPU, in
particular the exact model. Contributed by Antonio Cuni in `61 <https://github.com/ionelmc/pytest-benchmark/pull/61>`_.
* Added ``benchmark.extra_info``, which you can use to save arbitrary stuff in
the JSON. Contributed by Antonio Cuni in the same PR as above.
* Fix support for latest PyGal version (histograms). Contributed by Swen Kooij in
`68 <https://github.com/ionelmc/pytest-benchmark/pull/68>`_.
* Added support for getting ``commit_info`` when not running in the root of the repository. Contributed by Vara Canero in
`69 <https://github.com/ionelmc/pytest-benchmark/pull/69>`_.
* Added short form for ``--storage``/``--verbose`` options in CLI.
* Added an alternate ``pytest-benchmark`` CLI bin (in addition to ``py.test-benchmark``) to match the madness in pytest.
* Fix some issues with ``--help`` in CLI.
* Improved git remote parsing (for ``commit_info`` in JSON outputs).
* Fixed default value for ``--benchmark-columns``.
* Fixed comparison mode (loading was done too late).
* Remove the project name from the autosave name. This will get the old brief naming from 3.0 back.

--------------------

* Added ``--benchmark-colums`` command line option. It selects what columns are displayed in the result table. Contributed by
Antonio Cuni in `34 <https://github.com/ionelmc/pytest-benchmark/pull/34>`_.
* Added support for grouping by specific test parametrization (``--benchmark-group-by=param:NAME`` where ``NAME`` is your
param name). Contributed by Antonio Cuni in `37 <https://github.com/ionelmc/pytest-benchmark/pull/37>`__.
* Added support for ``name`` or ``fullname`` in ``--benchmark-sort``.
Contributed by Antonio Cuni in `37 <https://github.com/ionelmc/pytest-benchmark/pull/37>`_.
* Changed signature for ``pytest_benchmark_generate_json`` hook to take 2 new arguments: ``machine_info`` and ``commit_info``.
* Changed ``--benchmark-histogram`` to plot groups instead of name-matching runs.
* Changed ``--benchmark-histogram`` to plot exactly what you compared against. Now it's ``1:1`` with the compare feature.
* Changed ``--benchmark-compare`` to allow globs. You can compare against all the previous runs now.
* Changed ``--benchmark-group-by`` to allow multiple values separated by comma.
Example: ``--benchmark-group-by=param:foo,param:bar``
* Added a command line tool to compare previous data: ``py.test-benchmark``. It has two commands:

* ``list`` - Lists all the available files.
* ``compare`` - Displays result tables. Takes options:

* ``--sort=COL``
* ``--group-by=LABEL``
* ``--columns=LABELS``
* ``--histogram=[FILENAME-PREFIX]``
* Added ``--benchmark-cprofile`` that profiles last run of benchmarked function. Contributed by Petr Šebek.
* Changed ``--benchmark-storage`` so it now allows elasticsearch storage. It allows to store data to elasticsearch instead to
json files. Contributed by Petr Šebek in `58 <https://github.com/ionelmc/pytest-benchmark/pull/58>`_.

------------------

* Improved ``--help`` text for ``--benchmark-histogram``, ``--benchmark-save`` and ``--benchmark-autosave``.
* Benchmarks that raised exceptions during test now have special highlighting in result table (red background).
* Benchmarks that raised exceptions are not included in the saved data anymore (you can still get the old behavior back
by implementing ``pytest_benchmark_generate_json`` in your ``conftest.py``).
* The plugin will use pytest's warning system for warnings. There are 2 categories: ``WBENCHMARK-C`` (compare mode
issues) and ``WBENCHMARK-U`` (usage issues).
* The red warnings are only shown if ``--benchmark-verbose`` is used. They still will be always be shown in the
pytest-warnings section.
* Using the benchmark fixture more than one time is disallowed (will raise exception).
* Not using the benchmark fixture (but requiring it) will issue a warning (``WBENCHMARK-U1``).