Drgn

This release adds full AArch64 (ARM64) support, a few more helpers, and other small fixes and improvements.

New features:

- Stack traces were implemented for AArch64.
- Virtual address translation was implemented for AArch64.
- Additional registers are now available for the initial stack frame on x86-64: `rflags`, `es`, `cs`, `ss`, `ds`, `fs`, `gs`, `fs.base`, `gs.base`.
- The `lr` register is now available for stack frames on ppc64.
- A helper for looking up cgroups by path was added:`drgn.helpers.linux.cgroup.cgroup_get_from_path()`.
- A helper for walking kernfs paths was added: `drgn.helpers.linux.kernfs.kernfs_walk()`.
- Helpers for translating to and from physical addresses were added to `drgn.helpers.linux.mm`: `PFN_PHYS()`, `PHYS_PFN()`, `page_to_phys()`, `phys_to_page()`, `phys_to_virt()`, `virt_to_phys()`.
- Helpers for iterating BTF objects and BPF links were added to `drgn.helpers.linux.bpf`: `bpf_btf_for_each()` and `bpf_link_for_each()`. Contributed by Quentin Monnet.
- Support for UTF character types (e.g., `char8_t`, `char16_t`, `char32_t`) was added. Contributed by Kevin Svetlitski.
- Support for floating-point reference objects other than 32 or 64 bits was added (values are still not implemented). Contributed by Kevin Svetlitski.

Bug fixes:

- `drgn -s/--symbols` (and the underlying `drgn.Program.load_debug_info()`) now works for kernel module debug info files generated by `objcopy --only-keep-debug` instead of failing with an error.
- `drgn.helpers.linux.slab.slab_cache_for_each_allocated_object()` was fixed when `CONFIG_SLAB_FREELIST_HARDENED` is enabled.
- `drgn.helpers.linux.slab.slab_cache_for_each_allocated_object()` was fixed when using the SLAB allocator on kernels before v5.17.
- The `drgn.helpers.linux.slab.slab_cache_is_merged()` helper is now properly added to `drgn.helpers.linux.slab.__all__` so that it is included in `drgn.helpers.linux` and automatically imported in the CLI.
- The `cgroup_bpf_prog_for_each()` and `cgroup_bpf_prog_for_each_effective()` helpers in `drgn.helpers.linux.bpf` were fixed to support kernels back to v4.10.
- The CLI was fixed to add the current directory to `sys.path` when running in script mode. Contributed by Logan Gunthorpe.
- Missing documentation was added for the `validate_rbtree()` and `validate_rbtree_inorder_for_each_entry()` validators in `drgn.helpers.linux.rbtree`.
- An optimization to coalesce virtual address reads (used for vmcore reads and the `access_process_vm()` and `access_remote_vm()` helpers in `drgn.helpers.linux.mm`) was fixed. Previously, the optimization was skipped when it should be used and in rare situations could be used when it shouldn't have been, causing incorrect reads.
- A bug when a virtual address read straddles the non-canonical address range on x86-64 was fixed.
- Fallback stack unwinding on ppc64 when debugging information is not available was fixed.
- Reading registers from stack traces with a different byte order was fixed to swap the byte order.
- Automatic lookup of incomplete types was fixed when the type is in a C++ namespace. Contributed by Jay Kamat.

Other improvements:

- Linux kernel support was tested up to Linux v5.19-rc8.
- The address range of kernel modules is now determined more efficiently.
- The `bpf_map_for_each()` and `bpf_prog_for_each()` helpers in `drgn.helpers.linux.bpf` were made slightly more efficient by avoiding redundant type lookups.
- drgn will now detect a QEMU memory dump without VMCOREINFO and suggest how to get a dump that drgn can use.

Internal:

- The format for defining architecture registers was changed from a custom language to Python.
- The address/`struct page` translation helpers in `drgn.helpers.linux.mm` were made mostly architecture-agnostic.
- The unit tests for the address/`struct page` translation helpers in `drgn.helpers.linux.mm` were made more thorough.
- Unit tests were added for the BPF helpers.

Feature-wise, this release mainly adds more helpers, including Linux slab helpers and new "validator" helpers. Internally, a lot of work was done to improve testing.

New features:

- Helpers for the Linux kernel slab allocator were added (`slab_cache_is_merged()`, `for_each_slab_cache()`, `find_slab_cache()`, `print_slab_caches()`, and `slab_cache_for_each_allocated_object()` in `drgn.helpers.linux.slab`). Contributed by Alex Zhu.
- The concept of "validators", special helpers which validate a data structure, was introduced.
- Validators for linked lists were added (`validate_list()`, `validate_list_for_each()`, and `validate_list_for_each_entry()` in `drgn.helpers.list`).
- Validators for red-black trees were added (`validate_rbtree()` and `validate_rbtree_inorder_for_each_entry()` in `drgn.helpers.linux.rbtree`).
- The `drgn.helpers.linux.rbtree.RB_EMPTY_ROOT()` helper was added.
- The i386, AArch64, Arm, and RISC-V architectures are now recognized and were added to `Architecture`. (Architecture-specific features have not been implemented for those architectures yet.)
- Support for integer reference objects larger than 64 bits was added (values are still not implemented). Contributed by Jay Kamat.

Bug fixes:

- `Program.crashed_thread()` was fixed when debugging a non-SMP Linux kernel.
- A bug in 32-bit builds that could cause poor performance due to hash collisions was fixed.

Other improvements:

- Linux kernel support was tested up to v5.18-rc7.
- Applying ELF relocations (used for kernel modules) was optimized for ppc64, i386, AArch64, Arm, and RISC-V.
- Documentation for `StackFrame.name` was expanded.

Internal:

- The virtual machine testing setup was expanded to test on kernels using all three Linux kernel slab allocator implementations (SLUB, SLAB, SLOB) and both SMP and !SMP.
- The virtual machine testing setup was also expanded to use a custom kernel module to test helpers that couldn't previously be tested.
- Flake8 was added to the pre-commit configuration.

This is a small release adding a missing test file.

New features:

- A new API representing threads was added. The `Thread` class represents a thread. `Thread.stack_trace()` returns the stack trace for the thread. `Program.threads()` returns an iterator over the threads in the program. `Program.thread()` returns a thread given its thread ID. `Program.main_thread()` returns the main thread of the program. `Program.crashed_thread()` returns the thread that caused the program to crash. Contributed by Kevin Svetlitski and Mykola Lysenko.
- `Program.symbols()` was added. It returns all ELF symbols in the program, optionally filtered by a name or address. Contributed by Stephen Brennan.
- Support for looking up C++ classes with `Program.type()` was added. Contributed by Mykola Lysenko.
- Support for setting the default program language (`Program.language`) was added.
- Helpers for getting the Linux kernel log buffer were added (`drgn.helpers.linux.printk.get_printk_records()` and `drgn.helpers.linux.printk.get_dmesg()`). Contributed by Alex Zhu.
- Helpers for translating between Linux kernel socket objects and inode objects were added (`drgn.helpers.linux.net.SOCKET_I()` and `drgn.helpers.linux.net.SOCK_INODE()`). Contributed by Peilin Ye.
- A helper for getting the Linux kernel's idle tasks was added (`drgn.helpers.linux.sched.idle_task()`).
- A helper for decoding Linux kernel `struct page` flags was added (`drgn.helpers.linux.mm.decode_page_flags()`).
- Generic helpers for decoding flags were added (`drgn.helpers.decode_flags()` and `drgn.helpers.decode_enum_type_flags()`).

Bug fixes:

- A crash when encountering kernel modules processed by `objcopy --only-keep-debug` was fixed. Contributed by Peilin Ye.
- An endianness bug when debugging kernel modules that use a different byte order than the host was fixed.
- A compilation error on older versions of GCC was fixed. Contributed by Alakesh Haloi.
- libdrgn is now linked against libm to fix undefined references to `rint`.
- `drgn.helpers.linux.percpu.per_cpu_ptr()` was fixed to support !SMP kernels.
- A `TypeError` when `drgn.helpers.linux.rbtree.rb_find()` doesn't find a match was fixed.
- A memory leak when using an invalid `TypeTemplateParameter` was fixed.
- A missing type annotation for `drgn.Language.CPP` was added.
- `drgn_stack_trace_destroy()` was fixed to allow a `NULL` argument like other `destroy()` functions.

Other improvements:

- Linux kernel support was tested up to v5.17-rc6.
- The CLI now diagnoses the common mistake of passing a core dump file where a script is expected.
- `drgn.helpers.linux.list.list_for_each_entry()`, `drgn.helpers.linux.list.list_for_each_entry_reverse()`, `drgn.helpers.linux.list.hlist_for_each_entry()`, `drgn.helpers.linux.list_nulls.hlist_nulls_for_each_entry()`, and `drgn.helpers.linux.rbtree.rbtree_inorder_for_each_entry()` were optimized to only look up the entry type once per call.
- Parsing of type names in `Program.type()` was slightly optimized.
- Lookups of built-in kernel variables (e.g., `UTS_RELEASE`, `PAGE_SIZE`) were slightly optimized.
- `Program.type()` now accepts a `Type`. This makes it easier to write helpers that accept a type name or a `Type`.

Build system:

- Python is now required to build libdrgn even if the Python bindings are disabled.
- GNU Awk is no longer required to build drgn.
- Additional compiler warnings are now enabled by default. Developers should enable them as errors.
- Configuration options for developers to enable ASan (contributed by Kevin Svetlitski) and UBSan were added.

Internal:

- Developer documentation was improved.
- [`pre-commit`](https://pre-commit.com/) was configured to enforce coding standards. Contributed by Stephen Brennan.
- s390x and EPEL 9 builds were enabled on Packit. Contributed by Davide Cavalca.
- Test cases using `/proc/vmcore` in a kdump environment were added. Contributed by Kevin Svetlitski.
- Test cases for the Linux kernel red-black tree and linked list helpers were added.

This is a small release to fix a couple of issues with debuginfod.

Bug fixes:

- A crash or hang when downloading debugging information from debuginfod was fixed.

Other improvements:

- The CLI now prints debuginfod download progress.
- Some missing files were added to the source distribution.

New features:

- `Symbol.binding` and `Symbol.kind` were added to provide more information about symbols (e.g., whether a symbol is global or local, whether it refers to a function or variable).
- `Program.symbol(name)` now falls back to returning a local symbol if a matching global symbol was not found.
- Helpers for looking up network namespaces were added (`drgn.helpers.linux.net.for_each_net()` and `drgn.helpers.linux.net.get_ns_by_{inode,fd}()`).
- More networking helpers were added (`drgn.helpers.linux.net.netdev_for_each_tx_queue()` and `drgn.helpers.linux.tc.qdisc_lookup()`).
- Partial support for alternate DWARF files (`.gnu_debugaltlink`) was added. These are created by DWZ and used by Linux distros in some debugging symbol packages. Types and objects from the alternate file can not yet be looked up by name.

Bug fixes:

- A workaround was added for a bug in elfutils 0.183-0.185 that causes crashes when targeting userspace core dumps.
- Reading memory from a core dump was fixed to return a fault error instead of zeros for regions that were not saved in the core dump.
- Type name formatting was fixed to not include the entire definition of anonymous types.
- Parsing of DWARF representing an enum type which is backed by a typedef was fixed (now the underlying type is used).
- ELF core dump note matching was fixed to use an exact name match instead of a prefix match.
- Support for variable locations in DWARF 3 files was fixed.
- Support for structure member locations in DWARF 2 files was fixed.
- A memory leak when targeting a makedumpfile compressed kernel core dump was fixed.
- Indexing debugging symbols for a C++ namespace was fixed to be parallelized as intended.
- Incorrect memory allocation error checks were fixed for a couple of hash table insertions.
- Incorrect cleanup after failing to index DWARF information was fixed.
- An incorrect error when failing to get the build ID of an ELF file was fixed.
- A dropped error when applying ELF relocations was fixed.
- A missing error check in the C implementation of `drgn.helpers.linux.pid.pid_task()` was added.
- A missing error check when parsing C type names was fixed.
- An incorrect error return when memory allocation failed while parsing an array type from DWARF was fixed.

Other improvements:

- Memory usage after loading debugging symbols was decreased (by more aggressively freeing unneeded memory and by lazily allocating memory for C++ namespaces).
- Stack trace variable resolution in inline functions was optimized (by optimizing how we look up the context of an inline function).
- Type and object lookups were optimized (by avoiding OpenMP overhead after the DWARF information was already indexed).
- Installation documentation was improved.
- Documentation for getting debugging symbols was added.
- Linux 5.15 and 5.16 were tested and did not require any changes.

Internal:

- Debugging information handling was reorganized to prepare for upcoming changes.
- Tests were added for the cgroup helpers.
- [Packit](https://packit.dev/) configuration was added.