Xnvme

Latest version: v0.7.5

Safety actively analyzes 688600 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 4 of 5

0.0.23

-------

This release contains updates to third-party repositories along with any
changes necessary for xNVMe due to third-party changes.

* Third-party
- SPDK updated to v21.04
- liburing updated to v2.0
- fio, not updated, due to a compiler-warning breaking the xNVMe build

This release contains another major refactoring of the API along with a handful
of fixes and updates. The goal of the refactoring is to further simplify the
"core" of the API.

* The buffer-allocator ``xnvme_buf_alloc()`` automatically selects the type of
memory-allocator to use based on the device. However, it took a 'phys'
argument which is only valid for very specific use-cases. Thus, this argument
is removed and replaced by explicit ``physical`` allocators. This simplifies
the "core" usage, without sacrificing low-level control, it is just provided
via an explitcit interface instead.

* xNVMe now provides an API for file-system file-io
- Plugs into the synchronous as well as the asynchronous xNVMe command API
- I/O provided by ``xnvme_file_pread`` and ``xnvme_file_write``
- Provides support for diirect and non-direct I/O
- Two tools are provided utilizing the API ``xdd`` a simplified version of
``dd`` and ``xnvme_file`` utilizing sync. and sync. code-paths for
load/dump/copy of files

* Examples
- Add minimal examples for command submission and completion

* Backends
- linux:fs: preliminary support for file-system I/O
- linux:io_uring now does batched completion-handling
- linux:io_uring now supports kernel-completion-polling (IOPOLL)
- linux:io_uring fixes for use auto-handling of register-files
- spdk now provides core-mask control via ident-uri-options
- spdk now provides shared-memory group control via ident-uri-options

* A good handful of fixes, see the commit-messages for details

0.0.22

-------

This release contains a major refactoring of the API along with a handful of
minor fixes. The refactoring goals are to align to existing nomenclature and
simplify usage.

* Reduce to five abstractions: devices, queues, commands, and command-contexts
- Devices are base handles to NVMe Namespaces and a list of devices are
retrieved via ``xnvme_enumerate()``, and handles to individual devices
retrieved via ``xnvme_dev_open()`` and released via ``xnvme_dev_close()``.
- The abstraction formerly known as an ``asynchronous context`` is now dubbed
a ``queue``. The ``queue`` now has a ``capacity`` instead of a ``depth``.
- ``queues`` are created on top of ``devices`` and belong to the device.
- The definition, submission, and completion of a command is encapsulated in
a context; the command-context. The command-context replaces the previous
abstraction named the ``request``.
- A command can reach a device via a ``queue``, in a deferred / asynchronous
callback-based manner, or it go via the device in a synchronous / blocking
manner. Regardless, the command needs a context, and the context is
retrieved via ``xnvme_cmd_ctx_from_queue()`` or
``xnvme_cmd_ctx_from_dev()``.
- Commands are passed down via ``xnvme_cmd_pass`` for NVMe IO Commands, and
through ``xnvme_cmd_pass_admin`` for NVMe Admin Commands via the given
command-context.

* Core API reduction
- The core xNVMe API as provided by ``libxnvme.h`` it is reduced to a minimal
interface. Auxilary helpers, convenience functions, and pretty-printers are
no longer part of the core API but provided via individual header-files
- The core of the xNVMe API thus consists of
Device Handling: enumerate, dev_open, dev_close
Memory: alloc, realloc, free, vtophys, virt_alloc, virt_free
Queueing: init, term, poke, wait, get_command_ctx, get_capacity, get_outstanding
Commands: pass, pass_admin
Supporting the four abstractions described above
- The manual allocation of a request-pool / command-context-pool is no longer
needed. xNVMe does not prevent you from creating one if you want to, but it
is no longer required. Each 'queue' now provides a pre-allocated pool of
resources, and the manual request-pool is thus replaced by a call to the
function ``xnvme_cmd_ctx_from_queue()``. If you are familiar with
``io_uring`` then think of this function as the equivalent of
``io_uring_get_sqe()``.

* API re-organization
- Previously each command-set had its own top-level namespace, e.g. functions
and structures for the Zoned Command-Set was using ``znd_*``. This was
slightly quirky since it still relied on core of the xNVMe namespace
``xnvme_*`` for device handles etc. Thus, the command-set specific APIs
providing helper-functions and convenience are now nested in the xNVMe API
Namespace e.g. ``znd_*`` is now ``xnvme_znd_*`` and provided via
``libxnvme_znd.h``.
- The NVM Command-Set API was ``lblk_*`` it is now ``xnvme_nvm_*``, and
provided via ``libxnvme_nvm.h``.

* be:linux: changed error-mapping for non-NVMe errors
- The Linux block based and sync. interfaces does not provide the underlying
NVMe command status code and status code type since this is hidden behind
the block-interface. Previously, the NVMe-completion status-code was just
assigned the ``errno`` provided by the Kernel, which is highly confusing.
This behavior is replaced by assigning the status-code-type of
"vendor-specific" to indicate the status-codes are not defined in the spec.

* be:linux:aio: fixed submission and completion paths
- The submission, via ``cmd_io()``, of a single command would submit all
outstanding command, effectively limiting queue-depth
- The completion via ``poke()``/``wait()`` could potentially complete more
than requested by the user
- The encapsulation of io-control-blocks, array of io-control-block pointers,
were all pointing to the same control-block. Note, this was not causing
issues due to the short-coming in ``cmd_io()``.

0.0.21

-------

* Refactored backend interface

- Changed to support interchangeable ``sync`` and ``async`` implementations

* The Linux backend ``be::linux``
- Merged ``be:lioc``, ``be:laio``, ``be:liou``, and ``nil`` into one backend
``be:linux``, having the async-implementation be an engine parameter
controllable via uri-opt ``?async`` values: ``thr``, ``aio``, ``iou``,
``nil``.
- Added proper support for the Linux Block Device model, replacing the
``?pseudo`` option with ``sync`` interfaces ``nvme_ioctl`` and
``block_ioctl``. Gracefully falling back to the Block Layer when the given
device is not an NVMe device, and thus supporting everything the Linux
Block Supports including the Zoned Block Device model
- Added support for ``XNVME_CMD_ASYNC`` for ``ioctl``-driven commands. This
provides an async.interface to Linux driver-ioctls(), for commands other
than read/write. Next step is to make it run fast by providing a less
costly kernel path. This path is enabled via ``?async=thr``.
- With these changes, the build-configuration of backends has changed and
documentation describes how to enable/disable the different backends, sync,
and async implementations

* Changed command behavior

- api-functions taking command-options, e.g. ``xnvme_cmd_pass``,
``znd_cmd_mgmt_send``, now **require** that either ``XNVME_CMD_SYNC`` or
``XNVME_CMD_ASYNC`` is given as argument. When none is given, negated
``EINVAL`` is returned.

* xNVMe fio io-engine

- Replace ``--be`` option with ``--async``, this makes it a easier to
instrument ``fio`` to use a different async. implementation in the Linux
backend of xNVMe. Previously it relied on schema-prefix, the prefix-prefix
was annoying to use with fio as it required escape-chars.

- ``fio`` scripts and docs have been updated with the new ``--async`` argument

- ``fio`` scripts simplified and aligned such that they all three can be used
in the same manner using the ``--sector=default`` and ``--sector=override``
to override ``rw``, ``iodepth``, and ``bs`` via environment variables.

* Third-party libraries

- Added Linux/UAPI version to ``xnvme library-info``, this can give a good
hint on why certain features aren't behaving as expected, such as the Linux
versions without the Zoned Block headers
- Updated to fio/v3.23

* A general handful of code-cleanups and fixes, both on style as well as
potential issues such local-vars shadowing global-vars, potential arithmetic
overflows

* Continous Integration

- Added testing of Linux paths using Nullblock instances in addition to
emulated NVMe devices

- Added integration of GitHUB/CodeQL, since Semmle got acquired by GitHUB,
this will replace the lgtm.com integration.

0.0.20

-------

* Third-party libraries

- Updated to fio/v3.22
- Made fio available to the third-party SPDK build
- Added build of SPDK fio io-engine
- Fixed missing update of third-party version-strings

* The xNVMe fio io-engine

- Several fixes to locking/serialization and error-handling
- Adjusted to changes in upstream ZBD support
- Changed the zoned fio-example to not be timebased, since it could lead to
the verify-job never getting to the verify-part when running on emulated
devices
- Increased ``ramp_time`` in comparison-script
- Fixed memory issue due to missing ``get_file_size``

* Backends

- Added a backend ``nwrp`` the NULL-Async-IO backend, purpose of which is to
troubleshoot and benchmark the async-io path

* General

- A bunch of fixes including bad format-strings, out-of-bound / array
overflows, non-atomic locks, improper error-path handling

* CI

- Added workflow generating docker-image with latest source, providing
everything needed to build xNVMe and latest qemu to deploy and experiment
with xNVMe on emulated NVMe devices
- Added workflow doing Coverity scan and uploading results for analysis
- Added ``fio`` binary and SPDK fio io-engines as artifacts. During testing,
fio is needed, however, the test-environment might not have the same
version available as the io-engines are built against, usually xNVMe is
built against the latest release which might not have made it into the
package repos.

0.0.19

-------

* Third-party libraries

- Updated to liburing/v0.7, SPDK/v20.07, fio/v3.21
- Updated docs describing new third-party requirements for building
- Adjusted patches and build-system to changes

* Fabrics: SPDK-patches enabling zone-changes over Fabrics

* Added public-domain CI

- Primarily using GitHUB Actions / Workflows
- Aux. analysis via lgtm.com
- Updated docs and scripts for CI via GitHUB Actions

* Updated support for the NVMe Simple-Copy-Command (SCC)

- Targeting TP 2020.05.04 (Ratified)
- Added ``tests/scc.c`` testing for SCC-support, print identify fields, and
exercises the command itself

0.0.18

-------

* Third-party libraries: SPDK

- Updated tracking of SPDK to current master(7dbaf54bf) and adjusted linkage
- Removed patches that are now upstream
- Updated nvmf/IOCS support

* Fixed non-IOCS device identification

Page 4 of 5

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.