Curator-opensearch

-------------

**Announcement**

* As mentioned in 5.8.4's release notes, `boto3` deprecates support for
Python 2.7, and now so does Curator.

**New**

* Add filter by size feature. 1612 (IndraGunawan)
* Update Elasticsearch client to 7.14.0

**Security Fixes**

* Use `urllib3` 1.26.5 or higher 1610 (tsaarni)

---------------------

**Announcement**

* Because Python 2.7 has been EOL for over a year now, many projects are no
longer supporting it. This will also be the case for Curator as its
dependencies cease to support Python 2.7. With `boto3` having announced it
is ceasing support of Python 2.7, deprecated as of 15 Jan 2021, and fully
unsupported on 15 Jul 2021, Curator will follow these same dates. This
means that you will need to use an older version of Curator to continue
using Python 2.7, or upgrade to Python 3.6 or greater.

**Breaking**

* Normally I would not include breaking changes, but users have asked for
Click v7, which changes actions to require hyphens, and not underscores.
Options can still have underscores, but actions can't--well, not strictly
true. You can have underscores, but Click v7 will convert them to hyphens.
This should _only_ affect users of the Curator CLI, and not YAML file
users, and only the actions: `show-indices`, `show-snapshots`,
`delete-indices`, `delete-snapshots`. The actual actions are still named
with underscores, and the code has been updated to work with the hyphenated
action names.

**New**

* Now using `elasticsearch-py` version 7.12.0
* Adding testing for Python 3.9
* Removing testing on Python 3.6
* Tested Elasticsearch versions now include 7.12.0, 7.11.2, 7.10.2, 7.9.3,
7.8.1, 6.8.15, 5.6.16
* Changing `requirements.txt` as follows:
- boto3-1.17.57
- certifi-2020.12.5
- click-7.1.2
- elasticsearch-7.12.0
- pyyaml-5.4.1
- requests-2.25.1
- requests-aws4auth-1.0.1
- six-1.15.0
- urllib3-1.26.4
- voluptuous-0.12.1

**Bug Fixes**

* Alias integration tests needed updating for newer versions of Elasticsearch
that include ILM.
* Click 7.0 now reports an exit code of `1` for schema mismatches where it
yielded a `-1` in the past. Tests needed updating to correct for this.

**Security**

* Address multiple `pyyaml` vulnerabilities by bumping to version 5.4.1.
Contributed in 1596 (tsaarni)

------------------------

**New**

* Determined to test the last 2 major version's final patch releases, plus
the last 5 minor releases in the current major version. Travis CI testing
needs to go faster, and this should suffice. For now, this means versions
5.6.16, 6.8.13, 7.6.2, 7.7.1, 7.8.1, 7.9.3, and 7.10.0

**Bug Fixes**

* Caught a few stale merge failures, and asciidoc documentation problems
which needed fixing in the 5.8 branch, which necessitate this tiny bump
release. No code changes between 5.8.2 and 5.8.3.

------------------------

**Announcement**

* No, Curator isn't going away. But as you can tell, it's not as actively
developed as it once was. I am gratified to find there are still users who
make it a part of their workflow. I intend to continue development in my
spare time. Curator is now a labor of love, not an engineering project I
do during work hours.

**New**

* Testing changes. Only last ES version of 5.x and 6.x are tested, plus the
releases of 7.x since 7.2.
* ``http_auth`` is now deprecated. You can continue to use it, but it will go
away in the next major release. Moving forward, you should use ``username``
and ``password``. This should work in ``curator``, ``curator_cli``, and
``es_repo_mgr``.
* Removed tests for all 5.x branches of Elasticsearch but the final (5.6).
* Added tests for missing 7.x branches of Elasticsearch
* Remove tests for Python 3.5
* Fix hang of Shrink action in ES 7.x in 1528 (jclegras)
* Add ``ecs`` as a ``logformat`` option in 1529 (m1keil)

**Bug Fixes**

* Lots of code cleanup, trying to go PEP-8. All tests are still passing, and
the APIs are not changed (yet—-that comes in the next major release).
* Dockerfile has been updated to produce a working version with Python 3.7
and Curator 5.8.1
* Pin (for now) Elasticsearch Python module to 7.1.0. This will be updated
when an updated release of the module fixes the `cluster.state` API call
regression at https://github.com/elastic/elasticsearch-py/issues/1141
* Fix ``client.tasks.get`` API call to be ``client.tasks.list`` when no index
name is provided. See
https://github.com/elastic/elasticsearch-py/issues/1110
* Pin some pip versions to allow urllib3 and boto to coexist. See 1562
(sethmlarson).

**Documentation**

* Add Freeze/Unfreeze documentation in 1497 (lucabelluccini)
* Update compatibility matrix in 1522 (jibsonline)

-------------------------

**Bug Fixes**

* ``LD_LIBRARY_PATH`` will now be set in ``/usr/bin/curator`` and the
associated scripts rather than set in ``/etc/ld.so.conf.d``

**Other**

* Unsaved logging change in ``utils.py`` that got missed is merged.

-------------------------

**New**

* Require ``elasticsearch-py`` version 7.0.4
* Official support for Python 3.7 — In fact, the pre-built packages are built
using Python 3.7 now.
* Packages bundle OpenSSL 1.1.1c, removing the need for system OpenSSL
* Certifi 2019.9.11 certificates included.
* New client configuration option: api_key - used in the X-Api-key header in
requests to Elasticsearch when set, which may be required if ReadonlyREST
plugin is configured to require api-key. Requested in 1409 (vetler)
* Add ``skip_flush`` option to the ``close`` action. This should be useful
when trying to close indices with unassigned shards (e.g. before restore).
Raised in 1412. (psypuff)
* Use ``RequestsHttpConnection`` class, which permits the use of
``HTTP_PROXY`` and ``HTTPS_PROXY`` environment variables. Raised in 510
and addressed by 1259 (raynigon) in August of 2018. Subsequent changes,
however, required some adaptation, and re-submission as a different PR.
(untergeek)
* ``ignore_existing`` option added to ``CreateIndex``. Will not raise an
error if the index to be created already exists. Raised by (breml) in
1352. (untergeek)
* Add support for ``freeze`` and ``unfreeze`` indexes using curator. Requires
Elasticsearch version 6.6 or greater with xpack enabled. Requested in issue
1399 and rasied in PR 1454. (junmuz)
* Allow the ``close`` action to ignore synced flush failures with the new
``ignore_sync_failures`` option. Raised in 1248. (untergeek)

**Bug Fixes**

* Fix kibana filter to match any and all indices starting with ``.kibana``.
This addresses 1363, and everyone else upgrading to Elasticsearch 7.x.
Update documentation accordingly. (untergeek)
* Fix reindex post-action checks. When the filters do not return documents
to be reindexed, the post-action check to ensure the target index exists
is not needed. This new version will skip that validation if no documents
are processed (issue 1170). (afharo)
* Prevent the ``empty`` filtertype from incorrectly matching against closed
indices 1430 (heyitsmdr)
* Fix ``index_size`` function to be able to report either for either the
``total`` of all shards (default) or just ``primaries``. Added as a keyword
arg to preserve existing behavior. This was needed to fix sizing
calculations for the Shrink action, which should only count ``primaries``.
Raised in 1429 (untergeek).
* Fix ``allow_ilm_indices`` to work with the ``rollover`` action. Reported in
1418 (untergeek)
* Update the client connection logic to be cleaner and log more verbosely in
an attempt to address issues like 1418 and others like it more effectively
as other failures have appeared to be client failures because the last
log message were vague indications that a client connection was attempted.
This is a step in the right direction, as it explicitly exits with a 1 exit
code for different conditions now. (untergeek)
* Catch snapshots without a timestring in the name causing a logic error when
using the ``count`` filter and ``use_age`` with ``source: name``. Reported
by (nerophon) in 1366. (untergeek)
* Ensure authentication (401), authorization (403), and other 400 errors are
logged properly. Reported by (rfalke) in 1413. (untergeek)
* Fix crashes in restore of "large" number of indices reported by breml in
1360. (anandsinghkunwar)
* Do an empty list check before querying indices for field stats. Fixed by
(CiXiHuo) in 1448.
* Fix "Correctly report task runtime in seconds" while reindexing. Reported
by (jkelastic) in 1335

**Documentation**

* Grammar correction of ilm.asciidoc 1425 (SlavikCA)
* Updates to reflect changes to Elasticsearch 7 documentation 1426 and 1428
(lcawl) and (jrodewig)