Drgn

This release adds helpers all over the place, Linux 6.3 and 6.4 support, Python 3.12 support, full s390x support, bug fixes, and lots of new scripts in `contrib`.

New features:

- `follow_page()`, `follow_pfn()`, and `follow_phys()` helpers were added to `drgn.helpers.linux.mm`. These translate an arbitrary virtual address in an address space.
- `vmalloc_to_page()` and `vmalloc_to_pfn()` helpers were added to `drgn.helpers.linux.mm`. These translate a vmalloc/vmap address.
- The `drgn.helpers.linux.mm.totalram_pages()` helper was added. It returns the number of pages of RAM. Contributed by Martin Liška.
- The `drgn.helpers.linux.sched.loadavg()` helper was added. It returns the load average as a tuple. Contributed by Martin Liška.
- The `drgn.helpers.common.format.number_in_binary_units()` helper was added. It formats a number as a human-readable size (e.g., 2G, 1.5M).
- `drgn.cli.run_interactive()` was added. It can be used to embed drgn's interactive mode in other applications. Contributed by Stephen Brennan.
- The `jiffies` variable in the Linux kernel is now handled specially so that it can be accessed on all kernel versions and architectures.
- Virtual address translation was implemented for s390x. Contributed by Sven Schnelle.
- The `page_to_pfn()`, `page_to_phys()`, `pfn_to_page()`, and `phys_to_page()` helpers in `drgn.helpers.linux.mm` now work on architectures using `CONFIG_FLATMEM` (e.g., Arm and i386).
- Types can now be looked up in C++ namespaces. Contributed by Kevin Svetlitski.
- drgn will now use GNU-style compressed sections (`.zdebug_*`) when available.

Bug fixes:

- A crash when constructing objects on Python 3.12 was fixed. Contributed by Stephen Brennan.
- A bug that caused the ORC stack unwinder to stop prematurely or return the wrong result for IRQ stacks was fixed.
- `drgn.Program.crashed_thread()` was fixed for non-x86 architectures. Previously it always returned the thread on CPU 0.
- `drgn.helpers.linux.fs.for_each_file()` now handles tasks with `NULL` `files` (e.g., zombie tasks). Contributed by Stephen Brennan.
- The `drgn.helpers.linux.cgroup.sock_cgroup_ptr()` helper was fixed to work on Linux 5.15 and newer. Contributed by Martin Liška.
- The `drgn.helpers.linux.slab` helpers were fixed to handle older stable kernels without the patch "slub: improve bit diffusion for freelist ptr obfuscation". Contributed by Stephen Brennan.
- The `slab_object_info()` and `find_containing_slab_cache()` helpers in `drgn.helpers.linux.slab` were fixed to ignore high memory.
- A workaround for weird DWARF generated by GCC for zero-length arrays in C++ was added. Contributed by Jay Kamat.
- A memory leak in an error case when pretty-printing compound (structure/class/union) objects was fixed. Contributed by Kevin Svetlitski.

`contrib` directory:

- `contrib/btrfs_tree.py` and `contrib/btrfs_tree_mod_log.py` were added. They contain work-in-progress helpers for Btrfs data structures. Contributed by Boris Burkov.
- `contrib/dump_btrfs_bgs.py` was added. It prints information about block groups in a Btrfs filesystem. Contributed by Johannes Thumshirn.
- `contrib/kcore_list.py` was added. It prints the list of memory regions registered in `/proc/kcore`.
- `contrib/kernel_sys.py` was added. It prints system information similar to the crash `sys` command. Contributed by Martin Liška.
- `contrib/mount.py` was added. It prints a mount table similar to the crash `mount` command. Contributed by Martin Liška.
- `contrib/platform_drivers.py` was added. It prints all registered platform drivers.
- `contrib/vmmap.py` was added. It prints information about memory mappings in a process, similar to `/proc/$pid/maps`. Note that it only works up to Linux 6.0. Contributed by Martin Liška.
- `contrib/vmstat.py` was added. It prints information about kernel memory usage. Contributed by Martin Liška.
- `contrib/ps.py` was extended to print thread state, whether a thread is a kernel thread, and memory statistics. Contributed by Martin Liška.
- `contrib/fs_inodes.py` was fixed to to handle inodes without a path. Contributed by Martin Liška.
- `contrib/lsmod.py` was fixed to have identical output to `lsmod(8)`. Contributed by Martin Liška.
- `contrib/tcp_sock.py` was fixed to work on Linux 4.9 and newer. Contributed by Martin Liška.

Other improvements:

- Support for Linux 6.3 and 6.4 was added.
- The `compound_order()` and `compound_nr()` helpers in `drgn.helpers.linux.mm` were updated for Linux 6.3.
- ORC unwinder support was updated for Linux 6.3 and 6.4.
- Kernel module detection was updated for Linux 6.4. Contributed by Ido Schimmel.
- The `for_each_disk()` and `for_each_partition()` helpers in `drgn.helpers.linux.block` were updated for Linux 6.4.
- The `drgn.helpers.linux.idr` helpers were extended to work with kernels older than 4.11. Contributed by Imran Khan.
- Documentation was added for special objects that drgn exposes for the Linux kernel. Contributed by Stephen Brennan.
- The example in the documentation for `add_memory_segment()` was fixed. Contributed by Timothée Cocault.
- Immutable attributes were marked with `Final` in type stubs. Contributed by Kevin Svetlitski.

Internals:

- `setup.py` no longer uses distutils (as long as setuptools is new enough).
- Documentation was added for drgn's internal page table iterator interface.
- The virtual machine testing framework now supports AArch64, ppc64, s390x, and Arm. These are not tested automatically yet.
- The virtual machine testing framework now uses compilers from <https://mirrors.kernel.org/pub/tools/crosstool/>.
- The virtual machine testing framework now limits the number of CPUs to 8 to avoid OOMs. Contributed by Martin Liška.
- The pull request CI now only tests the oldest and latest stable Python versions, with the ability to opt into testing all supported versions. Contributed by Stephen Brennan.

This release adds new stack trace features, lots of helpers (especially for memory management), partial s390x support, C++ improvements, important bug fixes, and more. It is also the first release licensed under the LGPLv2.1+.

Miscellaneous:

- drgn is now licensed as LGPLv2.1+ instead of GPLv3+. The goal of the more permissive license is to encourage building tools on top of drgn (like [Object Introspection](https://facebookexperimental.github.io/object-introspection/)).
- The `contrib` directory was added as a place to share scripts with minimal requirements. This also replaces the `examples` directory.
- Support for Linux 4.4 (which has been EOL since February 2022) is no longer being actively tested. Most of drgn will continue to work on Linux 4.4 for the time being, but it is likely to stop working soon. The oldest kernel version officially supported by drgn is now 4.9.

New features:

- The `StackFrame.locals()` method was added. It lists all of the arguments and local variables in the scope of a stack frame. Contributed by Stephen Brennan.
- The `StackFrame.sp` attribute was added. This is a generic way to get the stack pointer of a stack frame on any architecture.
- Helpers for XArrays were added in `drgn.helpers.linux.xarray`: `xa_load()`, `xa_for_each()`, `xa_is_value()`, `xa_to_value()`, and `xa_is_zero()`.
- The `drgn.helpers.linux.slab.get_slab_aliases()` helper was added. It identifies which slab caches are merged. Contributed by Stephen Brennan.
- The `drgn.helpers.linux.slab.slab_object_info()` helper was added. It returns what slab cache a pointer is from, its offset from the beginning of the slab object, and whether it is allocated or free.
- The `drgn.helpers.common.memory.identify_address()` helper now includes additional information for slab addresses: the offset from the beginning of the slab object and whether it is allocated or free.
- The `drgn.helpers.common.stack.print_annotated_stack()` helper was added. It prints the contents of stack memory in a stack trace, annotating each word that can be identified as an address. Contributed by Nhat Pham.
- Support for Linux kernel modules and stack unwinding on s390x was added. Contributed by Sven Schnelle.
- Partial support for looking up types with C++ template arguments was added. For now, the arguments must be spelled exactly as the compiler spells them in the debug info. Contributed by Kevin Svetlitski.
- Parsing debug info for C++ template parameter packs was added. Contributed by Alastair Robertson.

Bug fixes:

- A bug that caused stack unwinding to fail when an executable contained `.eh_frame` but its DWARF information was in a separate file was fixed.
- A bug that caused x86-64 stack unwinding without call frame information to stop on a function with a frame pointer if its caller doesn't use frame pointers was fixed. This affected, for example, BPF programs in a Linux kernel using ORC.
- Linux kernel stack unwinding on ppc64 was fixed for kernel versions newer than 5.10 or older than 4.20.
- The CLI's interactive mode was fixed to allow importing modules from the current directory.
- Missing type annotations required for `len(StackTrace)` and `iter(StackTrace)` were added. Contributed by Nhat Pham.
- A potential segfault (not encountered in practice) when parsing invalid DWARF information for an enum type was fixed.
- Leaks in error cases of `Program.type()` and `Program.object()` were fixed.
- The `drgn.helpers.common.memory.identify_address()` helper was fixed to gracefully handle architectures that we haven't implemented virtual address translation for.
- Parsing of DWARF pointer types without a size was fixed to default to the DWARF unit's address size instead of the program's default size.
- Lookups of type names beginning with `size_t` or `ptrdiff_t` were fixed to look up the full name and not those prefixes.

Other improvements:

- Linux kernel support was tested up to Linux 6.2-rc2.
- Python support was tested up to Python 3.11.
- The CLI warning for missing debug info was made more prominent.
- The CLI now prints nicer error messages for common errors like running without root permissions.
- Documentation for the `drgn.helpers.linux.mm.for_each_page()` helper was improved to mention how `FaultError` should be handled.
- drgn can now be built against a libc without `qsort_r()` (e.g., musl < 1.2.3). Contributed by Boris Burkov.

Internals:

- Various tests for kconfig helpers, radix tree helpers, and stack traces were added or improved.
- Tracking of executable and debug info files was separated from libdwfl more.
- Documentation for new architecture support was expanded.

This release adds lots of new helpers and fixes some important bugs.

New features:

- Helpers for lockless linked lists in the Linux kernel were added: `drgn.helpers.linux.llist`. Contributed by Imran Khan.
- A helper to find the slab cache that a virtual address came from was added: `drgn.helpers.linux.slab.find_containing_slab_cache()`. Contributed by Nhat Pham.
- A `drgn.helpers.common` package was created to contain helpers that can be used with any program (which may have program-specific additional behavior).
- A helper to identify an arbitrary address (e.g., as a symbol or slab object) was added: `drgn.helpers.common.memory.identify_address()`. Contributed by Nhat Pham.
- `PageFoo()` helpers to check various `struct page` flags were added to `drgn.helpers.linux.mm`.
- Helpers for working with compound pages were added to `drgn.helpers.linux.mm`: `compound_head()`, `compound_nr()`, `compound_order()`, and `page_size()`.
- A helper to get the CPU that a task last ran on was added: `drgn.helpers.linux.sched.task_cpu()`.
- Automatic pretty-printing in IPython/Jupyter of `drgn.Object`, `drgn.Type`, `drgn.StackTrace`, and `drgn.StackFrame` was added. Contributed by Shung-Hsi Yu.
- `drgn.StackTrace.prog` was added as a way to get the program that a stack trace came from.

Bug fixes:

- Getting stack traces from a kernel core dump of threads that were running at the time of the crash was fixed for Linux < 4.9 and >= 5.16.
- `drgn.helpers.linux.per_cpu()` and `per_cpu_ptr()` were fixed to work for per-CPU variables defined in kernel modules.
- Reading from pages that were excluded by `makedumpfile` was changed to return zeroes instead of raising a `FaultError`. This is unfortunately necessary because we cannot distinguish between pages that were excluded because their contents were zero and pages that were excluded for other reasons. Contributed by Glen McCready.
- A segfault when looking for a variable in a stack frame caused by strange debug symbols emitted by Clang in certain situations was fixed.
- A reference leak every time a `FaultError` is raised inside of drgn was fixed.
- A use after free when setting an object from a part of itself in libdrgn was fixed. The Python interface is not affected.
- The recommendation for how to get VMCOREINFO in QEMU guest memory dumps was fixed to suggest the correct Linux kernel configuration options.
- A spelling error in a DWARF parsing error message was fixed. Contributed by Michel Alexandre Salim.

API changes:

- `escape_ascii_character()`, `escape_ascii_string()`, `decode_flags()`, and `decode_enum_type_flags()` were moved from `drgn.helpers` to `drgn.helpers.common.format`.
- `enum_type_to_class()` was moved from `drgn.helpers` to `drgn.helpers.common.type`.

Documentation:

- openSUSE installation instructions were added.
- libkdumpfile installation instructions were updated to reflect that it is now packaged on some Linux distributions.
- Python type signatures are now formatted more concisely in the documentation.
- Overloaded helpers are now documented more concisely.
- Various small editorial and formatting issues were fixed.

Other improvements:

- Linux kernel support was tested up to Linux 6.0.
- `drgn.helpers.linux.bpf.cgroup_bpf_prog_for_each()` was updated to work on Linux 6.0.

Internals:

- Some racy stack tracing unit tests were fixed.
- Linux kernel memory management helper unit tests were enabled on AArch64.
- The VM testing setup no longer depends on BusyBox.
- `-Wimplicit-fallthrough` was enabled for builds.
- A syscall number table and normalized machine name were added to `util.py`
for use in test cases and the VM testing setup.
- Some renaming was done to prepare for the upcoming module API.
- The libdrgn-internal `string_builder` API was improved.

This release adds full AArch64 (ARM64) support, a few more helpers, and other small fixes and improvements.

New features:

- Stack traces were implemented for AArch64.
- Virtual address translation was implemented for AArch64.
- Additional registers are now available for the initial stack frame on x86-64: `rflags`, `es`, `cs`, `ss`, `ds`, `fs`, `gs`, `fs.base`, `gs.base`.
- The `lr` register is now available for stack frames on ppc64.
- A helper for looking up cgroups by path was added:`drgn.helpers.linux.cgroup.cgroup_get_from_path()`.
- A helper for walking kernfs paths was added: `drgn.helpers.linux.kernfs.kernfs_walk()`.
- Helpers for translating to and from physical addresses were added to `drgn.helpers.linux.mm`: `PFN_PHYS()`, `PHYS_PFN()`, `page_to_phys()`, `phys_to_page()`, `phys_to_virt()`, `virt_to_phys()`.
- Helpers for iterating BTF objects and BPF links were added to `drgn.helpers.linux.bpf`: `bpf_btf_for_each()` and `bpf_link_for_each()`. Contributed by Quentin Monnet.
- Support for UTF character types (e.g., `char8_t`, `char16_t`, `char32_t`) was added. Contributed by Kevin Svetlitski.
- Support for floating-point reference objects other than 32 or 64 bits was added (values are still not implemented). Contributed by Kevin Svetlitski.

Bug fixes:

- `drgn -s/--symbols` (and the underlying `drgn.Program.load_debug_info()`) now works for kernel module debug info files generated by `objcopy --only-keep-debug` instead of failing with an error.
- `drgn.helpers.linux.slab.slab_cache_for_each_allocated_object()` was fixed when `CONFIG_SLAB_FREELIST_HARDENED` is enabled.
- `drgn.helpers.linux.slab.slab_cache_for_each_allocated_object()` was fixed when using the SLAB allocator on kernels before v5.17.
- The `drgn.helpers.linux.slab.slab_cache_is_merged()` helper is now properly added to `drgn.helpers.linux.slab.__all__` so that it is included in `drgn.helpers.linux` and automatically imported in the CLI.
- The `cgroup_bpf_prog_for_each()` and `cgroup_bpf_prog_for_each_effective()` helpers in `drgn.helpers.linux.bpf` were fixed to support kernels back to v4.10.
- The CLI was fixed to add the current directory to `sys.path` when running in script mode. Contributed by Logan Gunthorpe.
- Missing documentation was added for the `validate_rbtree()` and `validate_rbtree_inorder_for_each_entry()` validators in `drgn.helpers.linux.rbtree`.
- An optimization to coalesce virtual address reads (used for vmcore reads and the `access_process_vm()` and `access_remote_vm()` helpers in `drgn.helpers.linux.mm`) was fixed. Previously, the optimization was skipped when it should be used and in rare situations could be used when it shouldn't have been, causing incorrect reads.
- A bug when a virtual address read straddles the non-canonical address range on x86-64 was fixed.
- Fallback stack unwinding on ppc64 when debugging information is not available was fixed.
- Reading registers from stack traces with a different byte order was fixed to swap the byte order.
- Automatic lookup of incomplete types was fixed when the type is in a C++ namespace. Contributed by Jay Kamat.

Other improvements:

- Linux kernel support was tested up to Linux v5.19-rc8.
- The address range of kernel modules is now determined more efficiently.
- The `bpf_map_for_each()` and `bpf_prog_for_each()` helpers in `drgn.helpers.linux.bpf` were made slightly more efficient by avoiding redundant type lookups.
- drgn will now detect a QEMU memory dump without VMCOREINFO and suggest how to get a dump that drgn can use.

Internal:

- The format for defining architecture registers was changed from a custom language to Python.
- The address/`struct page` translation helpers in `drgn.helpers.linux.mm` were made mostly architecture-agnostic.
- The unit tests for the address/`struct page` translation helpers in `drgn.helpers.linux.mm` were made more thorough.
- Unit tests were added for the BPF helpers.

Feature-wise, this release mainly adds more helpers, including Linux slab helpers and new "validator" helpers. Internally, a lot of work was done to improve testing.

New features:

- Helpers for the Linux kernel slab allocator were added (`slab_cache_is_merged()`, `for_each_slab_cache()`, `find_slab_cache()`, `print_slab_caches()`, and `slab_cache_for_each_allocated_object()` in `drgn.helpers.linux.slab`). Contributed by Alex Zhu.
- The concept of "validators", special helpers which validate a data structure, was introduced.
- Validators for linked lists were added (`validate_list()`, `validate_list_for_each()`, and `validate_list_for_each_entry()` in `drgn.helpers.list`).
- Validators for red-black trees were added (`validate_rbtree()` and `validate_rbtree_inorder_for_each_entry()` in `drgn.helpers.linux.rbtree`).
- The `drgn.helpers.linux.rbtree.RB_EMPTY_ROOT()` helper was added.
- The i386, AArch64, Arm, and RISC-V architectures are now recognized and were added to `Architecture`. (Architecture-specific features have not been implemented for those architectures yet.)
- Support for integer reference objects larger than 64 bits was added (values are still not implemented). Contributed by Jay Kamat.

Bug fixes:

- `Program.crashed_thread()` was fixed when debugging a non-SMP Linux kernel.
- A bug in 32-bit builds that could cause poor performance due to hash collisions was fixed.

Other improvements:

- Linux kernel support was tested up to v5.18-rc7.
- Applying ELF relocations (used for kernel modules) was optimized for ppc64, i386, AArch64, Arm, and RISC-V.
- Documentation for `StackFrame.name` was expanded.

Internal:

- The virtual machine testing setup was expanded to test on kernels using all three Linux kernel slab allocator implementations (SLUB, SLAB, SLOB) and both SMP and !SMP.
- The virtual machine testing setup was also expanded to use a custom kernel module to test helpers that couldn't previously be tested.
- Flake8 was added to the pre-commit configuration.

This is a small release adding a missing test file.