Zhmcclient

^^^^^^^^^^^^^^

Released: 2024-06-12

**Incompatible changes:**

* Incompatible changes in the notification support:

- The 'NotificationReceiver.notifications()' method now continues running when
there are no notifications, and only ever returns when
'NotificationReceiver.close()' is called (in some other thread).
Before this change, the method returned when there were no notifications, so
it had to be invoked by the user in a loop. Such user code should be adjusted
to remove the loop and deal with the return indicating a close of the
receiver.

- In addition, the 'NotificationReceiver.notifications()' method can now raise
the new exceptions 'zhmcclient.NotificationConnectionError' and
'zhmcclient.NotificationSubscriptionError'.

- The 'NotificationReceiver.subscribe/unsubscribe()' methods can now raise the
new exception 'zhmcclient.NotificationSubscriptionError'.

- Note that the 'NotificationReceiver.close()' method can raise
'stomp.exception.StompException'. This could already be raised before this
change, but had not been documented before.

Issue: (`1502 <https://github.com/zhmcclient/python-zhmcclient/issues/1502>`_)

**Enhancements:**

* Test: Relaxed the verification of log messages in test_auto_updater.py
to tolerate additional log messages.

* Added a class 'StompRetryTimeoutConfig' for defining retry, timeout and
keepalive/heartbeat parameters for the STOMP connection for HMC
notifications. Added new 'stomp_rt_config' init parameters to the
'NotificationReceiver' and 'AutoUpdater' classes, to specify these config
parameters. Added default values for the configuration in zhmcclient constants. (`1498 <https://github.com/zhmcclient/python-zhmcclient/issues/1498>`_)

* Improved the notification support in several ways:

- Replaced the event-based handover of a single item from the notification
listener thread to the caller's thread with a Python Queue, for better
reliability. It turned out that messages could have been lost in some cases
with the previous design.

- The 'NotificationReceiver.notifications()' method now continues running
when there are no notifications, and only ever returns when
'NotificationReceiver.close()' is called (by some other thread).

- Added methods 'connect()' and 'is_connected()' to the 'NotificationReceiver'
class. The init method of 'NotificationReceiver' no longer connects,
but the 'notifications()' method now calls 'connect()', so overall this is
compatible with the prior behavior.

- Added new exceptions 'NotificationConnectionError' and
'NotificationSubscriptionError' that may be raised by some
'NotificationReceiver' methods.

- Documented the stomp-py exceptions that can be raised from
'NotificationReceiver' methods.

- Added proper detection of STOMP connection loss if STOMP heartbeating is
enabled. The connection loss is surfaced by raising
'NotificationConnectionError' in 'NotificationReceiver.notifications()'.
This allows users to retry 'NotificationReceiver.notifications()' upon
connection loss.

- Added a new public constant 'STOMP_MIN_CONNECTION_CHECK_TIME' that defines
the minimum time between checks for STOMP connection loss. The actual check
time is determined by the heartbeat receive time and is bound by this minimum
time.

- Added the missing event methods to the internal '_NotificationListener' class
in case they are ever invoked (needed due to lazy importing of stomp-py).

- Added more log messages around STOMP connect / disconnect.

Issue: (`1502 <https://github.com/zhmcclient/python-zhmcclient/issues/1502>`_)

* Added support for getting new z16 environmental metrics about CPC and LPAR
or partitions by adding 'get_sustainability_data()' methods to Cpc, Lpar,
and Partition. (`1511 <https://github.com/zhmcclient/python-zhmcclient/issues/1511>`_)

**Cleanup:**

* Removed the pinning of stomp.py to <7.0.0 and increased its minimum version
to 8.1.1 (for Python>=3.7) to pick up fixes, and adjusted to the changed
interface of the stomp event listener methods and the 'stomp.Connection()' call. (`1499 <https://github.com/zhmcclient/python-zhmcclient/issues/1499>`_)

* Test: Upgraded Github Actions plugin actions/setup-python to v5 to no longer
use the deprecated node version 16. (`1503 <https://github.com/zhmcclient/python-zhmcclient/issues/1503>`_)

^^^^^^^^^^^^^^

Released: 2024-06-07

**Incompatible changes:**

* The 'zhmcclient.User' object will no longer be able to store the 'password'
property. The 'password' property is filtered out when creating the User object
in 'UserManager.create()' and when updating the User object in
'User.update_properties()'. (`1490 <https://github.com/zhmcclient/python-zhmcclient/issues/1490>`_)

**Bug fixes:**

* Fixed safety issues up to 2024-06-07

* Addressed dependabot issues up to 2024-06-07

* Dev: In the Github Actions test workflow for Python 3.5, 3.6 and 3.7, changed
macos-latest back to macos-12 because macos-latest got upgraded from macOS 12
to macOS 14 which no longer supports these Python versions.

* Dev: Workaround for cert issue with pip in Python 3.5 in Github Actions.

* Dev: Addressed new issues raised by Pylint 3.1.

* Dev: Fixed new issue 'possibly-used-before-assignment' in Pylint 3.2.0.

* Docs: Fixed broken links to HMC books since IBM changed the links. As part
of that, removed Bibliography entries for the HMC API book versions 2.11/2.12,
and for all versions of the HMC Operations Guide (which changed to become the
HMC Help System PDFs). (`1459 <https://github.com/zhmcclient/python-zhmcclient/issues/1459>`_)

* Docs: Fixed formatting of badges on README page by converting it to
Markdown. (`1473 <https://github.com/zhmcclient/python-zhmcclient/issues/1473>`_)

* Test: Upgraded Github actions plugin actions/github-script to v7 to no longer
use the deprecated Node.js 16. (`1483 <https://github.com/zhmcclient/python-zhmcclient/issues/1483>`_)

**Enhancements:**

* Test: Added the option 'ignore-unpinned-requirements: False' to both
safety policy files because for safety 3.0, the default is to ignore
unpinned requirements (in requirements.txt).

Increased safety minimum version to 3.0 because the new option is not
tolerated by safety 2.x. Safety now runs only on Python >=3.7 because
that is what safety 3.0 requires.

* Changed safety run for install dependencies to use the exact minimum versions
of the dependent packages, by moving them into a separate
minimum-constraints-install.txt file that is included by the existing
minimum-constraints.txt file.

* The safety run for all dependencies now must succeed when the test workflow
is run for a release (i.e. branch name 'release\_...').

* Added support for "Console Delete Retrieved Internal Code" HMC operation
via a new 'zhmcclient.Console.delete_uninstalled_firmware()' method. (`1431 <https://github.com/zhmcclient/python-zhmcclient/issues/1431>`_)

* Added new method Nic.backing_port() to return the backing adapter port
of the NIC. (`1451 <https://github.com/zhmcclient/python-zhmcclient/issues/1451>`_)

* Dev: Migrated from a manually maintained change log file to using change
fragment files with the 'towncrier' package. This simplifies the procedures
for starting and releasing a version, and avoids merge conflicts when there
are multiple Pull Requests at the same time. For details, read the new
'Making a change' section in the documentation. (`1485 <https://github.com/zhmcclient/python-zhmcclient/issues/1485>`_)

^^^^^^^^^^^^^^

This version contains all fixes up to version 1.13.4.

Released: 2024-02-17

**Incompatible changes:**

* The incompatibility caused by the recent change to support regular expression
matching for the resource name in the 'find()' method, which was released in
zhmcclient versions 1.12.3 and 1.13.0, turned out to be too heavy. The change
is now undone to go back to string comparison for the name matching in
'find()'. The 'findall()' method which was also changed in these releases
keeps the regular expression matching for consistency with 'list()'.
(issue 1395)

**Bug fixes:**

* Docs: Increased minimum Sphinx versions to 7.1.0 on Python 3.8 and to 7.2.0 on
Python >=3.9 and adjusted dependent package versions in order to fix a version
incompatibility between sphinxcontrib-applehelp and Sphinx.
Disabled Sphinx runs on Python <=3.7 in order to no longer having to deal
with older Sphinx versions. (issue 1396)

* Changed the recently released support for regular expression matching for the
resource name in 'find()' back to matching by string comparison. The
'findall()' method keeps the regular expression matching for consistency
with 'list()'. (issue 1395)

* Fixed that the resource name in the filter arguments of 'findall()' and
'list()' was not matched case insensitvely with regular expressions for the
resource types that have case insensitive names (user, user pattern, password
rule, LDAP server definition). (related to issue 1395)

* Fixed that 'Console.list_permitted_lpars()' ignored the
'additional_properties' parameter. (issue 1410)

* Test: Fixed that unit tests did not properly check missing properties in
the returned resources. (related to issue 1410)

* Fixed that 'list()' methods returned only a minimal set of properties
for each resource when the resource was found in the name-to-URI cache,
and in that case missed some properties that are documented for the
corresponding HMC list operation. This was fixed by removing the optimization
of using the name-to-URI cache in 'list()' methods. (related to issue 1410)

* In the zhmcclient mock support, fixed the processing of the
'additional-properties' query parameters for the mock support of the following
zhmcclient list methods: 'Console.list_permitted_lpars()',
'Cpc.adapters.list()', 'Cpc.partitions.list()', 'Cpc.virtual_switches.list()',
'Cpc.image_activation_profiles.list()'. (related to issue 1410)

* Development: Fixed dependency issue with safety 3.0.0 by pinning it.

* Performance: In zhmcclient version 1.13.0, an optimization was added where
list() and find_local() were now utilizing the name-to-URI cache when only the
resource name was specified as a filter argument. This caused the 'se-version'
property to no longer be in the local zhmcclient.Cpc objects that were used
as the parent objects of the Lpar/Partition objects returned by
Console.list_permitted_lpars/partitions() and caused a performance
degradation in the zhmc_lpar_list and zhmc_partition_list Ansible modules due
to repeated "Get CPC Properties" operations for retrieving the 'se-version'
property. This was fixed in the Console.list_permitted_lpars/partitions()
methods.

* Fixed the call to pipdeptree in the test workflow to use 'python -m'
because otherwise it does not show the correct packages of the virtual env.

* Fixed the 'Cpc.delete_retrieved_internal_code()' method which passed its
'ec_level' parameter incorrectly to the HMC operation. Added unit tests.
(issue 1432)

**Enhancements:**

* Test: Added Python 3.8 with latest package levels to normal tests because
that is now the minimum version to run Sphinx. (related to issue 1396)

* Added support for a new make target 'authors' that generates an AUTHORS.md
file from the git commit history. Added the invocation of 'make authors' to
the description of how to release a version in the development
documentation. (issue 1393)

* In Console.list_permitted_lpars/partitions(), added CPC-related properties
to the returned resource objects, that are returned by the HMC: 'cpc-name',
'cpc-object-uri', 'se-version'. (issue 1421)

* In Console.list_permitted_lpars(), the additional_properties parameter
is now supported also for HMC versions older than 2.16 GA 1.5. In that
case, the zhmcclient handles adding the properties. (related to issue 1421)

* The pull_full_properties() and pull_properties() methods of zhmcclient
resource objects no longer replace existing properties but now update them,
so that additionally present properties (e.g. the CPC-related properties
returned from Console.list_permitted_lpars/partitions()) are preserved.
(related to issue 1421)

**Cleanup:**

* Increased versions of GitHub Actions plugins to increase node.js runtime
to version 20.

^^^^^^^^^^^^^^

This version contains all fixes up to version 1.12.2.

Released: 2024-01-11

**Incompatible changes:**

* The 'Cpc.single_step_install()' and 'Console.single_step_install()' methods
added in version 1.12.0 got additional optional parameters for FTP server
retrieval added before the existing 'wait_for_completion' parameter. If you
were using these methods and specified 'wait_for_completion' or
'operation_timeout' as positional arguments, these methods will now raise
an AssertionError and you need to change your code to specify them as keyword
arguments, instead.

* When creating a 'zhmcclient.Session' object with a 'session_id' parameter that
is not None, the 'host' parameter with the HMC host for that session now also
needs to be provided. (related to issue 1024)

* The 'base_url' property of the 'zhmcclient.Session' object is now 'None' when
the session is in the logged-off state. (related to issue 1024)

* The 'list()' methods of zhmcclient manager objects when invoked with
full_properties=False and with the resource name as the only filter argument
now return only a minimal set of properties for the returned resource:
'class', 'parent', 'name', 'object/element-id', 'object/element-uri'.
Previously, the full set of properties was returned in such a case.
Code that accesses one of the no longer returned properties via
'resource.properties' will now fail with KeyError. This can be fixed by
changing such code to access the property via 'resource.get_property()',
or by specifying 'full_properties=True' on the 'list()' method.
(part of issue 1070)

* The 'delete()' methods of zhmcclient resource objects now also set the
ceased-existence flag on the resource object. This causes 'get_property()'
and prop()' when called for locally available properties to now raise
CeasedExistence. Previously, the locally available property value was
returned. (part of issue 1070)

**Bug fixes:**

* Addressed safety issues up to 2023-11-26.

* Test: Fixed end2end test function test_hmcdef_cpcs() to no longer stumble over
'loadable_lpars' and 'load_profiles' properties in HMC inventory file.
(issue 1374)

* Test: Fixed end2end testcase 'test_actprof_crud()' to skip the test when the
required 'create-delete-activation-profiles' API feature is not available.
(issue 1375)

* Docs: Clarified that the 'session' and 'session_credential' properties of the
'zhmcclient.Session' object are 'None' when the session is in the logged-off
state. (related to issue 1024)

* Clarified the HMC version requirements for 'Console.list_permitted_adapters()'.

* Docs: Clarified in 'StorageGroup.list_candidate_adapter_ports()' that the
method is only for FCP-type storage groups.

* Fixed that the 'find()' and 'findall()' methods now also support regular
expression matching when the resource name is passed as a filter argument.
(issue 1070)

**Enhancements:**

* Added support for retrievel of firmware from an FTP server to the
Cpc/Console.single_step_install() methods. (issue 1342)

* Additional log entries when HTTP status 403 is received, for easier detection.

* Added support for additional SE firmware upgrade related HMC operations:
(issue 1357)

- "CPC Install and Activate" as 'Cpc.install_and_activate()'
- "CPC Delete Retrieved Internal Code" as 'Cpc.delete_retrieved_internal_code()'

* Added support for tolerating HMC restarts while waiting for a job to complete.
Session.wait_for_completion() now retries in case of ConnectionError instead of
raising the error. (issue 1365)

* Added the session-credential value returned by HMC logon as a new property
'session_credential' to the 'zhmcclient.Session' object. (related to issue
1350)

* Clarified in the description of 'zhmcclient.NotificationReceiver' that
its userid and password init parameters are actually the message broker's
userid and password, and that in case of MFA being configured, they must be
the session ID and session credential returned from the HMC logon.
(issue 1350)

* Added support for targeting multiple redundant HMCs, from which the first
one reachable at session creation time will be used for the duration of the
session. The multiple HMCs are provided via the same 'Session' init parameter
'host' as before, which now can be a list of hosts in addition to being a
single host. Because redundant HMCs can be configured differently regarding
what data they sync between them, there is no automatic failover to another
HMC if the initially determined HMC becomes unavailable during the session.
(issue 1024)

* Added support for specifying multiple redundant HMCs in the 'ansible_host'
property of HMC definition files. The property can now specify a single HMC
like before, or a a list of redundant HMCs. (issue 1024)

* Mock support: Added mock support for the Logon and Logoff HMC operations.
(related to issue 1024)

* Improved the 'list()' methods of zhmcclient manager classes by using the
name-to-URI cache when the resource name is passed as a filter argument.
This improvement avoids retrieving the resource from the HMC when it can be
found in the name-to-URI case, and therefore the resource will have only a
minimal set of properties in that case. See the corresponding entry in the
Incompatibilities section. (part of issue 1070)

* Improved the 'delete()' methods of zhmcclient resource classes by setting
the ceased-existence flag on the resource. This will cause optimized
find-like methods that operate on local data to properly raise
CeasedExistence when used on the deleted resource object.
(part of issue 1070)

^^^^^^^^^^^^^^

This version contains all fixes up to version 1.11.4.

Released: 2023-11-16

**Incompatible changes:**

* The pull_full_properties(), pull_properties(), get_property() and props()
methods on resource objects
now raise zhmcclient.CeasedExistence in all cases where the resource no
longer exists on the HMC. This provides a consistent behavior across different
cases the method can encounter. Previously, that exception was raised only for
resources that had auto-update enabled, and resources with auto-update
disabled raised zhmcclient.HTTPError(404,1) instead when the resource no
longer existed on the HMC.
If you use these methods and check for resource existence using
HTTPError(404,1), you need to change this to check for CeasedExistence
instead.

* The pull_properties() methods on resource objects now retrieves all properties
from the HMC when one or more of the specified properties are not supported
by the resource. This provides a consistent behavior across the different
cases the method can encounter. Previously, that method behaved differently
when the property was not supported by the resource: It has retrieved all
properties when the resource type or HMC version does not support property
filtering, but has raised HTTPError(400,14) in case the resource type and
HMC version did support property filtering.
If you use this method and check for HTTPError(400,14), this check can now be
removed.

**Deprecations:**

* Use of the 'status_timeout' and 'allow_status_exceptions' parameters of the
following methods has been deprecated because the underlying HMC operations
do not actually have deferred status behavior. The waiting for an expected
status has been removed from these methods:
- Lpar.stop()
- Lpar.psw_restart()
- Lpar.reset_normal()
- Lpar.reset_clear()

**Bug fixes:**

* Test: Circumvented a pip-check-reqs issue by excluding its version 2.5.0.

* Addressed safety issues up to 2023-11-05.

* Fixed the maximum number of concurrent threads in bulk operations to be
the documented maximum of 10.

* Test: Added unit tests and end2end tests for list permitted partitions operation

* Docs: Corrected and improved the description of the Lpar.activate() method.

* Test: Added end2end tests for LPAR activation in classic mode.

* Fixed the waiting for LPAR status in Lpar.activate(). Previously, the method
was waiting for 'operating' or 'not-operating', so when an auto-load
happened it already returned when status 'not-operating' was reached, but
the load was still going on in parallel. Now, the method finds out whether
the LPAR is expected to auto-load or not and waits for the corresponding
status.

* Added a debug log entry when Lpar.wait_for_status() is called. This happens
for example when Lpar.activate/deactivate/load() are called with
wait_for_completion.

* Fixed that the Lpar.reset_normal() and Lpar.reset_clear() methods were
waiting for a status "operational", which never happens with these operations.
This was fixed by removing the waiting for an expected status, because the
underlying HMC operations do not actually have deferred status behavior.
(issue 1304)

* Fixed the incorrect empty request body in Lpar.psw_restart().

* Shortened the status timeout from 900 sec to 60 sec. This timeout is used
when waiting for an expected Partition or LPAR status after operations
that change the status and that have deferred status behavior (ie. the
status changes only after the asynchronous HMC job is complete).
This change allows to more reasonably surface the situation where an LPAR
load succeeds but the status of the LPAR does not go to 'operating' due to
issues with the operating system.

* Docs: Fixed the description of the 'status_timeout' parameter of the Partition
and Lpar methods that have deferred status behavior.

* The 'wait_for_completion' and 'operation_timeout' parameters of
Cpc.export_profiles() and Cpc.import_profiles() have never worked, because
the underlying HMC operations are not actually asynchronous. This has been
fixed by removing these parameters from these functions. This does not count
as an incompatible change because using these parameters with non-default
values has failed. (part of issue 1299)

**Enhancements:**

* Added support for Python 3.12. Had to increase the minimum versions of
setuptools to 66.1.0 and pip to 23.1.2 in order to address removal of the
long deprecated pkgutils.ImpImporter in Python 3.12, as well as several
packages used only for development. (issue 1300)

* Mock support: Improved mocked Hipersocket adapters; they now have all their
properties and default values for all except adapter-id and channel-path-id.

* Added support for the "List OS Messages" operation on partitions (in DPM mode)
and LPARs (in classic mode). (issue 1278)

* Examples: Added example script increase_crypto_config.py for increasing the
crypto configuration of a partition on a CPC in DPM mode.

* The pull_properties() method on resource objects was extended so that its
'properties' parameter can now also be a single string (in addition to the
already supported list or tuple of strings).

* Added a get_properties_pulled() method for resource objects, which gets the
current value of a set of properties from the HMC. If the resource has
auto-update enabled, it gets the value from the (automatically updated) local
cache. Otherwise, it retrieves the properties from the HMC in the fastest
possible way, considering property filtering if supported.

* Added support for passing an exception message directly to the
zhmcclient.NotFound exception, instead of creating it from the 'manager' and
'filter_args' parameters, which are now optional.

* Added support for asynchronous job cancellation via a new method Job.cancel().
Documented for all asynchronous methods returning Job objects whether or not
they can be cancelled. (issue 1299)

* Added support for low level management of asynchronous jobs via new methods
Job.query_status() and Job.delete(). Note that higher level methods
Job.check_for_completion() and Job.wait_for_completion() already existed.
(issue 1299)

* Added support for creation and deletion of activation profiles on z16.
This requires the SE to have a code level that has the
'create-delete-activation-profiles' API feature enabled.
(issue 1329)

* Added Lpar.start() to perform the "Start Logical Partition" operation in
classic mode. (issue 1308)

^^^^^^^^^^^^^^

This version contains all fixes up to version 1.10.1.

Released: 2023-09-07

**Incompatible changes:**

* Fixed BaseResource.pull_properties() by returning None when no properties
were specified. Before that, it returned the full set of properties when
the Get Properties operation for the resource does not support the 'properties'
query parameter, and produced 'properties=' as a query parameter when
the resource does support the 'properties' query parameter.

This is incompatible when your code uses pull_properties() on resource objects
and relies on the prior behavior.

* Installation of this package using "setup.py" is no longer supported.
Use "pip" instead.

**Bug fixes:**

* Fixed safety issues from 2023-08-27.

* Fixed zhmcclient_mock support for LDAP Server Definitions.

* Fixed end2end testcases for adapters, auto-updating, and groups.

* Fixed that SubscriptionNotFound exception message did not resolve its
format string.

* Fixed the zhmcclient_mock support by adding support for query parameters,
fixing the the Group operations and the "Query API Version" operation,
and fixing the z16 mock environment definitions.
Auto-update tests are now skipped when testing against mocked environments,
because the mock support does not support notifications.

**Enhancements:**

* Docs: Improved documentation for developing tests.

* Implemented mock support for aggregation service operation "Submit requests"
(bulk operations) (issue 1250).

* Added support for requesting additional properties in list() methods for
Adapter, Certificate, Partition, VirtualSwitch, ImageActivationProfile
resources, and for Console.list_permitted_lpars().

* Improved performance of list() method of all resource types when called
with full_properties=True by using a bulk operation (aggregation service).

* Test: Added a new make target "end2end_mocked" to run the end2end tests against
the mocked environments in the "examples" directory. As part of that, combined
the coverage results of unit tests and end2end tests into a single data file
that each test contributes to.

**Cleanup:**

* Consolidated common code of list() methods into the _utils.py module.