Pybind11

Changes:

- `python3 -m pybind11` gained a `--version` option (prints the version and exits). 4526

Bug Fixes:

- Fix a warning when pydebug is enabled on Python 3.11. 4461
- Ensure `gil_scoped_release` RAII is non-copyable. 4490
- Ensure the tests dir does not show up with new versions of setuptools. 4510
- Better stacklevel for a warning in setuptools helpers. 4516

Changes:

- Temporarily made our GIL status assertions (added in 2.10.2) disabled by default (re-enable manually by defining `PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF`, will be enabled in 2.11). 4432
- Improved error messages when `inc_ref`/`dec_ref` are called with an invalid GIL state. 4427 4436

Bug Fixes:

- Some minor touchups found by static analyzers. 4440

Changes:

- `scoped_interpreter` constructor taking `PyConfig`. 4372
- `pybind11/eigen/tensor.h` adds converters to and from `Eigen::Tensor` and `Eigen::TensorMap` 4201
- `PyGILState_Check()`'s were integrated to `pybind11::handle` `inc_ref()` & `dec_ref()`. The added GIL checks are guarded by `PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF`, which is the default only if `NDEBUG` is not defined. 4246
- Add option for enable/disable enum members in docstring. 2768
- Fixed typing of `KeysView`, `ValuesView` and `ItemsView` in `bind_map`. 4353

Bug fixes:

- Bug fix affecting only Python 3.6 under very specific, uncommon conditions: move `PyEval_InitThreads()` call to the correct location. 4350
- Fix segfault bug when passing foreign native functions to functional.h. 4254

Build system improvements:

- Support setting PYTHON_LIBRARIES manually for Windows ARM cross-compilation (classic mode). 4406
- Extend IPO/LTO detection for ICX (a.k.a IntelLLVM) compiler. 4402
- Allow calling `find_package(pybind11 CONFIG)` multiple times from separate directories in the same CMake project and properly link Python (new mode). 4401
- `multiprocessing_set_spawn` in pytest fixture for added safety. 4377
- Fixed a bug in two pybind11/tools cmake scripts causing "Unknown arguments specified" errors. 4327

This is the first version to fully support embedding the newly released Python 3.11.

Changes:

- Allow `pybind11::capsule` constructor to take null destructor pointers. 4221
- `embed.h` was changed so that `PYTHONPATH` is used also with Python 3.11 (established behavior). 4119
- A `PYBIND11_SIMPLE_GIL_MANAGEMENT` option was added (cmake, C++ define), along with many additional tests in `test_gil_scoped.py`. The option may be useful to try when debugging GIL-related issues, to determine if the more complex default implementation is or is not to blame. See \4216 for background. WARNING: Please be careful to not create ODR violations when using the option: everything that is linked together with mutual symbol visibility needs to be rebuilt. 4216
- `PYBIND11_EXPORT_EXCEPTION` was made non-empty only under macOS. This makes Linux builds safer, and enables the removal of warning suppression pragmas for Windows. 4298

Bug fixes:

- Fixed a bug where `UnicodeDecodeError` was not propagated from various `py::str` ctors when decoding surrogate utf characters. 4294
- Revert perfect forwarding for `make_iterator`. This broke at least one valid use case. May revisit later. 4234
- Fix support for safe casts to `void*` (regression in 2.10.0). 4275
- Fix `char8_t` support (regression in 2.9). 4278
- Unicode surrogate character in Python exception message leads to process termination in `error_already_set::what()`. 4297
- Fix MSVC 2019 v.1924 & C++14 mode error for `overload_cast`. 4188
- Make augmented assignment operators non-const for the object-api. Behavior was previously broken for augmented assignment operators. 4065
- Add proper error checking to C++ bindings for Python list append and insert. 4208
- Work-around for Nvidia's CUDA nvcc compiler in versions 11.4.0 - 11.8.0. 4220
- A workaround for PyPy was added in the `py::error_already_set` implementation, related to PR 1895 released with v2.10.0. 4079
- Fixed compiler errors when C++23 `std::forward_like` is available. 4136
- Properly raise exceptions in contains methods (like when an object in unhashable). 4209
- Further improve another error in exception handling. 4232
- `get_local_internals()` was made compatible with `finalize_interpreter()`, fixing potential freezes during interpreter finalization. 4192

Performance and style:

- Reserve space in set and STL map casters if possible. This will prevent unnecessary rehashing / resizing by knowing the number of keys ahead of time for Python to C++ casting. This improvement will greatly speed up the casting of large unordered maps and sets. 4194
- GIL RAII scopes are non-copyable to avoid potential bugs. 4183
- Explicitly default all relevant ctors for pytypes in the `PYBIND11_OBJECT` macros and enforce the clang-tidy checks `modernize-use-equals-default` in macros as well. 4017
- Optimize iterator advancement in C++ bindings. 4237
- Use the modern `PyObject_GenericGetDict` and `PyObject_GenericSetDict` for handling dynamic attribute dictionaries. 4106
- Document that users should use `PYBIND11_NAMESPACE` instead of using `pybind11` when opening namespaces. Using namespace declarations and namespace qualification remain the same as `pybind11`. This is done to ensure consistent symbol visibility. 4098
- Mark `detail::forward_like` as constexpr. 4147
- Optimize unpacking_collector when processing `arg_v` arguments. 4219
- Optimize casting C++ object to `None`. 4269

Build system improvements:

- CMake: revert overwrite behavior, now opt-in with `PYBIND11_PYTHONLIBS_OVERRWRITE OFF`. 4195
- Include a pkg-config file when installing pybind11, such as in the Python package. 4077
- Avoid stripping debug symbols when `CMAKE_BUILD_TYPE` is set to `DEBUG` instead of `Debug`. 4078
- Followup to 3948, fixing vcpkg again. 4123

Removed support for Python 2.7, Python 3.5, and MSVC 2015. Support for MSVC 2017 is limited due to availability of CI runners; we highly recommend MSVC 2019 or 2022 be used. Initial support added for Python 3.11.

New features:

- `py::anyset` & `py::frozenset` were added, with copying (cast) to `std::set` (similar to `set`). [3901](https://github.com/pybind/pybind11/pull/3901)
- Support bytearray casting to string. [3707](https://github.com/pybind/pybind11/pull/3707)
- `type_caster<std::monostate>` was added. `std::monostate` is a tag type that allows `std::variant` to act as an optional, or allows default construction of a `std::variant` holding a non-default constructible type. [3818](https://github.com/pybind/pybind11/pull/3818)
- `pybind11::capsule::set_name` added to mutate the name of the capsule instance. [3866](https://github.com/pybind/pybind11/pull/3866)
- NumPy: dtype constructor from type number added, accessors corresponding to Python API `dtype.num`, `dtype.byteorder`, `dtype.flags` and `dtype.alignment` added. [3868](https://github.com/pybind/pybind11/pull/3868)

Changes:

- Python 3.6 is now the minimum supported version. [3688](https://github.com/pybind/pybind11/pull/3688) [#3719](https://github.com/pybind/pybind11/pull/3719)
- The minimum version for MSVC is now 2017. [3722](https://github.com/pybind/pybind11/pull/3722)
- Fix issues with CPython 3.11 betas and add to supported test matrix. [3923](https://github.com/pybind/pybind11/pull/3923)
- `error_already_set` is now safer and more performant, especially for exceptions with long tracebacks, by delaying computation. [1895](https://github.com/pybind/pybind11/pull/1895)
- Improve exception handling in python `str` bindings. [3826](https://github.com/pybind/pybind11/pull/3826)
- The bindings for capsules now have more consistent exception handling. [3825](https://github.com/pybind/pybind11/pull/3825)
- `PYBIND11_OBJECT_CVT` and `PYBIND11_OBJECT_CVT_DEFAULT` macro can now be used to define classes in namespaces other than pybind11. [3797](https://github.com/pybind/pybind11/pull/3797)
- Error printing code now uses `PYBIND11_DETAILED_ERROR_MESSAGES` instead of requiring `NDEBUG`, allowing use with release builds if desired. [3913](https://github.com/pybind/pybind11/pull/3913)
- Implicit conversion of the literal `0` to `pybind11::handle` is now disabled. [4008](https://github.com/pybind/pybind11/pull/4008)

Bug fixes:

- Fix exception handling when `pybind11::weakref()` fails. [3739](https://github.com/pybind/pybind11/pull/3739)
- `module_::def_submodule` was missing proper error handling. This is fixed now. [3973](https://github.com/pybind/pybind11/pull/3973)
- The behavior or `error_already_set` was made safer and the highly opaque "Unknown internal error occurred" message was replaced with a more helpful message. [3982](https://github.com/pybind/pybind11/pull/3982)
- `error_already_set::what()` now handles non-normalized exceptions correctly. [3971](https://github.com/pybind/pybind11/pull/3971)
- Support older C++ compilers where filesystem is not yet part of the standard library and is instead included in `std::experimental::filesystem`. [3840](https://github.com/pybind/pybind11/pull/3840)
- Fix `-Wfree-nonheap-object` warnings produced by GCC by avoiding returning pointers to static objects with `return_value_policy::take_ownership`. [3946](https://github.com/pybind/pybind11/pull/3946)
- Fix cast from pytype rvalue to another pytype. [3949](https://github.com/pybind/pybind11/pull/3949)
- Ensure proper behavior when garbage collecting classes with dynamic attributes in Python \>=3.9. [4051](https://github.com/pybind/pybind11/pull/4051)
- A couple long-standing `PYBIND11_NAMESPACE` `__attribute__((visibility("hidden")))` inconsistencies are now fixed (affects only unusual environments). [4043](https://github.com/pybind/pybind11/pull/4043)
- `pybind11::detail::get_internals()` is now resilient to in-flight Python exceptions. [3981](https://github.com/pybind/pybind11/pull/3981)
- Arrays with a dimension of size 0 are now properly converted to dynamic Eigen matrices (more common in NumPy 1.23). [4038](https://github.com/pybind/pybind11/pull/4038)
- Avoid catching unrelated errors when importing NumPy. [3974](https://github.com/pybind/pybind11/pull/3974)

Performance and style:

- Added an accessor overload of `(object &&key)` to reference steal the object when using python types as keys. This prevents unnecessary reference count overhead for attr, dictionary, tuple, and sequence look ups. Added additional regression tests. Fixed a performance bug the caused accessor assignments to potentially perform unnecessary copies. [3970](https://github.com/pybind/pybind11/pull/3970)
- Perfect forward all args of `make_iterator`. [3980](https://github.com/pybind/pybind11/pull/3980)
- Avoid potential bug in pycapsule destructor by adding an `error_guard` to one of the dtors. [3958](https://github.com/pybind/pybind11/pull/3958)
- Optimize dictionary access in `strip_padding` for numpy. [3994](https://github.com/pybind/pybind11/pull/3994)
- `stl_bind.h` bindings now take slice args as a const-ref. [3852](https://github.com/pybind/pybind11/pull/3852)
- Made slice constructor more consistent, and improve performance of some casters by allowing reference stealing. [3845](https://github.com/pybind/pybind11/pull/3845)
- Change numpy dtype from_args method to use const ref. [3878](https://github.com/pybind/pybind11/pull/3878)
- Follow rule of three to ensure `PyErr_Restore` is called only once. [3872](https://github.com/pybind/pybind11/pull/3872)
- Added missing perfect forwarding for `make_iterator` functions. [3860](https://github.com/pybind/pybind11/pull/3860)
- Optimize c++ to python function casting by using the rvalue caster. [3966](https://github.com/pybind/pybind11/pull/3966)
- Optimize Eigen sparse matrix casting by removing unnecessary temporary. [4064](https://github.com/pybind/pybind11/pull/4064)
- Avoid potential implicit copy/assignment constructors causing double free in `strdup_gaurd`. [3905](https://github.com/pybind/pybind11/pull/3905)
- Enable clang-tidy checks `misc-definitions-in-headers`, `modernize-loop-convert`, and `modernize-use-nullptr`. [3881](https://github.com/pybind/pybind11/pull/3881) [#3988](https://github.com/pybind/pybind11/pull/3988)

Build system improvements:

- CMake: Fix file extension on Windows with cp36 and cp37 using FindPython. [3919](https://github.com/pybind/pybind11/pull/3919)
- CMake: Support multiple Python targets (such as on vcpkg). [3948](https://github.com/pybind/pybind11/pull/3948)
- CMake: Fix issue with NVCC on Windows. [3947](https://github.com/pybind/pybind11/pull/3947)
- CMake: Drop the bitness check on cross compiles (like targeting WebAssembly via Emscripten). [3959](https://github.com/pybind/pybind11/pull/3959)
- Add MSVC builds in debug mode to CI. [3784](https://github.com/pybind/pybind11/pull/3784)
- MSVC 2022 C++20 coverage was added to GitHub Actions, including Eigen. [3732](https://github.com/pybind/pybind11/pull/3732), [#3741](https://github.com/pybind/pybind11/pull/3741)

Backend and tidying up:

- New theme for the documentation. [3109](https://github.com/pybind/pybind11/pull/3109)
- Remove idioms in code comments. Use more inclusive language. [3809](https://github.com/pybind/pybind11/pull/3809)
- `include <iostream>` was removed from the `pybind11/stl.h` header. Your project may break if it has a transitive dependency on this include. The fix is to "Include What You Use". [3928](https://github.com/pybind/pybind11/pull/3928)
- Avoid `setup.py <command>` usage in internal tests. [3734](https://github.com/pybind/pybind11/pull/3734)

Changes:

- Enum now has an `__index__` method on Python \<3.8 too. [\3700](https://github.com/pybind/pybind11/pull/3700)
- Local internals are now cleared after finalizing the interpreter. [\3744](https://github.com/pybind/pybind11/pull/3744)

Bug fixes:

- Better support for Python 3.11 alphas. [\3694](https://github.com/pybind/pybind11/pull/3694)
- `PYBIND11_TYPE_CASTER` now uses fully qualified symbols, so it can be used outside of `pybind11::detail`. [\3758](https://github.com/pybind/pybind11/pull/3758)
- Some fixes for PyPy 3.9. [\3768](https://github.com/pybind/pybind11/pull/3768)
- Fixed a potential memleak in PyPy in `get_type_override`. [\3774](https://github.com/pybind/pybind11/pull/3774)
- Fix usage of `VISIBILITY_INLINES_HIDDEN`. [\3721](https://github.com/pybind/pybind11/pull/3721)

Build system improvements:

- Uses `sysconfig` module to determine installation locations on Python \>= 3.10, instead of `distutils` which has been deprecated. [\3764](https://github.com/pybind/pybind11/pull/3764)
- Support Catch 2.13.5+ (supporting GLIBC 2.34+). [\3679](https://github.com/pybind/pybind11/pull/3679)
- Fix test failures with numpy 1.22 by ignoring whitespace when comparing `str()` of dtypes. [\3682](https://github.com/pybind/pybind11/pull/3682)

Backend and tidying up:

- clang-tidy: added `readability-qualified-auto`, `readability-braces-around-statements`, `cppcoreguidelines-prefer-member-initializer`, `clang-analyzer-optin.performance.Padding`, `cppcoreguidelines-pro-type-static-cast-downcast`, and `readability-inconsistent-declaration-parameter-name`. [\3702](https://github.com/pybind/pybind11/pull/3702), [\#3699](https://github.com/pybind/pybind11/pull/3699), [\#3716](https://github.com/pybind/pybind11/pull/3716), [\#3709](https://github.com/pybind/pybind11/pull/3709)
- clang-format was added to the pre-commit actions, and the entire code base automatically reformatted (after several iterations preparing for this leap). [\3713](https://github.com/pybind/pybind11/pull/3713)