Curator-opensearch

---------------------

**New Features**

* Add ``wait_for_rebalance`` as an option for ``shrink`` action. By default
the behavior remains unchanged. You can now set this to False though to
allow the shrink action to only check that the index being shrunk has
finished being relocated and it will not wait for the cluster to
rebalance. 1129 (tschroeder-zendesk)
* Work around for extremely large cluster states. 1142 (rewiko)
* Add CI tests for Elasticsearch versions 6.1 and 6.2 (untergeek)
* Add Elasticsearch datemath support for snapshot names 1078 (untergeek)
* Support ``max_size`` as a rollover condition for Elasticsearch versions
6.1.0 and up. 1140 (untergeek)
* Skip indices with a document count of 0 when using ``source: field_stats``
to do ``age`` or ``period`` type filtering. 1130 (untergeek)

**Bug Fixes**

* Fix missing node information in log line. 1142 (untergeek)
* Fix default options in code that were causing schema validation errors
after ``voluptuous`` upgrade to 0.11.1.
Reported in 1149, fixed in 1156 (untergeek)
* Disallow empty lists as reindex source. Raise exception if that happens.
Reported in 1139 (untergeek)
* Set a ``timeout_override`` for ``delete_snapshots`` to catch cases where
slower repository network and/or disk access can cause a snapshot delete
to take longer than the default 30 second client timeout. 1133 (untergeek)
* Add AWS ES 5.1 support. 1172 (wanix)
* Add missing ``period`` filter arguments for ``delete_snapshots``.
Reported in 1173 (untergeek)
* Fix kibana filtertype to catch newer index names.
Reported in 1171 (untergeek)
* Re-order the closed indices filter for the Replicas action to take place
`before` the empty list check.
Reported in 1180 by ``agomerz`` (untergeek)

**General**

* Deprecate testing for Python 3.4. It is no longer being supported by
Python.
* Increase logging to show error when ``master_only`` is true and there are
multiple hosts.

**Documentation**

* Correct a misunderstanding about the nature of rollover conditions.
1144 (untergeek)
* Correct links to the field_stats API, as it is non-existent in
Elasticsearch 6.x. (untergeek)
* Add a warning about using forcemerge on active indices.
1153 (untergeek)
* Fix select URLs in pip installation from source to not be 404
1133 (untergeek)
* Fix an error in regex filter documentation 1138 (arne-cl)

-----------------------

**Bug Fixes**

* Improve Dockerfile to build from source and produce slimmer image
1111 (mikn)
* Fix ``filter_kibana`` to correctly use ``exclude`` argument
1116 (cjuroz)
* Fix `ssl_no_validate` behavior within AWS ES 1118 (igalarzab)
* Improve command-line exception management 1119 (4383)
* Make ``alias`` action always process ``remove`` before ``add``
to prevent undesired alias removals. 1120 (untergeek)

**General**

* Bump ES versions in Travis CI

**Documentation**

* Remove ``unit_count`` parameter doc for parameter that no longer
exists 1107 (dashford)
* Add missing ``exclude: True`` in ``timestring`` docs 1117 (GregMefford)

------------------------

**Announcement**

* Support for Elasticsearch 6.0!!! Yes!

**New Features**

* The ``field_stats`` API may be gone from Elasticsearch, but its utility
cannot be denied. And so, Curator has replaced the ``field_stats`` API
call with a small aggregation query. This will be perhaps a bit more
costly in performance terms, as this small aggregation query must be made
to each index in sequence, rather than as a one-shot call, like the
``field_stats`` API call. But the benefit will remain available, and
it's the only major API that did not persevere between Elasticsearch 5.x
and 6.x that was needed by Curator.

-----------------------

**New Features**

* With the period filter and field_stats, it is useful to match indices
that fit `within` the period, rather than just their start dates. This
is now possible with ``intersect``. See more in the documentation.
Requested in 1045. (untergeek)
* Add a ``restore`` function to ``curator_cli`` singleton. Mentioned in
851 (alexef)
* Add ``pattern`` to the ``count`` filter. This is particularly useful
when working with rollover indices. Requested in 1044 (untergeek)
* The ``es_repo_mgr create`` command now can take ``skip_repo_fs_check`` as
an argument (default is False) 1072 (alexef)
* Add ``pattern_type`` feature expansion to the ``period`` filter. The
default behavior is ``pattern_type='relative'``, which preserves existing
behaviors so users with existing configurations can continue to use them
without interruption. The new ``pattern_type`` is ``absolute``, which
allows you to specify hard dates for ``date_from`` and ``date_to``, while
``date_from_format`` and ``date_to_format`` are strftime strings to
interpret the from and to dates. Requested in 1047 (untergeek)
* Add ``copy_aliases`` option to the ``shrink`` action. So this option is
only set in the ``shrink`` action. The default value of the option is
``copy_aliases: 'False'`` and it does nothing. If you set to
``copy_aliases: 'True'``, you could copy the aliases from the source index
to the target index. Requested in 1060 (monkey3199)
* IAM Credentials can now be retrieved from the environment using the Boto3
Credentials provider. 1084 (kobuskc)

**Bug Fixes**

* Delete the target index (if it exists) in the event that a shrink fails.
Requested in 1058 (untergeek)
* Fixed an integration test that could fail in the waning days of a month.
* Fix build system anomalies for both unix and windows.

**Documentation**

* Set repository access to be https by default.
* Add documentation for ``copy_aliases`` option.

------------------------

**New Features**

* Shrink action! Apologies to all who have patiently waited for this
feature. It's been a long time coming, but it is hopefully worth the
wait. There are a lot of checks and tests associated with this action,
as there are many conditions that have to be met in order for a shrink
to take place. Curator will try its best to ensure that all of these
conditions are met so you can comfortably rest assured that shrink will
work properly unattended. See the documentation for more information.
* The ``cli`` function has been split into ``cli`` and ``run`` functions.
The behavior of ``cli`` will be indistinguishable from previous releases,
preserving API integrity. The new ``run`` function allows lambda and other
users to `run` Curator from the API with only a client configuration file
and action file as arguments. Requested in 1031 (untergeek)
* Allow use of time/date string interpolation for Rollover index naming.
Added in 1010 (tschroeder-zendesk)
* New ``unit_count_pattern`` allows you to derive the ``unit_count`` from
the index name itself. This involves regular expressions, so be sure to
do lots of testing in ``--dry-run`` mode before deploying to production.
Added by (soenkeliebau) in 997

**Bug Fixes**

* Reindex ``request_body`` allows for 2 different ``size`` options. One
limits the number of documents reindexed. The other is for batch sizing.
The batch sizing option was missing from the schema validator. This has
been corrected. Reported in 1038 (untergeek)
* A few sundry logging and notification changes were made.

----------------------

**Errata**

* An update to Elasticsearch 5.5.0 changes the behavior of
``filter_by_aliases``, differing from previous 5.x versions.

If a list of aliases is provided, indices must appear in `all` listed
aliases or a 404 error will result, leading to no indices being matched.
In older versions, if the index was associated with even one of the
aliases in aliases, it would result in a match.

Tests and documentation have been updated to address these changes.

* Debian 9 changed SSL versions, which means that the pre-built debian
packages no longer work in Debian 9. In the short term, this requires
a new repository. In the long term, I will try to get a better
repository system working for these so they all work together, better.
Requested in 998 (untergeek)

**Bug Fixes**

* Support date math in reindex operations better. It did work previously,
but would report failure because the test was looking for the index with
that name from a list of indices, rather than letting Elasticsearch do
the date math. Reported by DPattee in 1008 (untergeek)
* Under rare circumstances, snapshot delete (or create) actions could fail,
even when there were no snapshots in state ``IN_PROGRESS``. This was
tracked down by JD557 as a collision with a previously deleted snapshot
that hadn't finished deleting. It could be seen in the tasks API. An
additional test for snapshot activity in the tasks API has been added to
cover this scenario. Reported in 999 (untergeek)
* The ``restore_check`` function did not work properly with wildcard index
patterns. This has been rectified, and an integration test added to
satisfy this. Reported in 989 (untergeek)
* Make Curator report the Curator version, and not just reiterate the
elasticsearch version when reporting version incompatibilities. Reported
in 992. (untergeek)
* Fix repository/snapshot name logging issue. 1005 (jpcarey)
* Fix Windows build issue 1014 (untergeek)


**Documentation**

* Fix/improve rST API documentation.
* Thanks to many users who not only found and reported documentation issues,
but also submitted corrections.