Pyinstaller

-------------------

Features
~~~~~~~~

* Add a work-around for pure-python modules that do not specify encoding via
:pep:`263` encoding header but contain non-ASCII characters in local
(non-UTF8) encoding. When such characters are present only in code comments,
python still loads and runs the module, but attempting to retrieve its source
code via the loader's ``get_source()`` method results in a
:class:`UnicodeDecodeError`, which interrupts the analysis process. The error
is now caught and a fall-back codepath attempts to retrieve the source code as
raw data to avoid encoding issues. (:issue:`7622`)


Bugfix
~~~~~~

* (Windows) Avoid writing collected binaries to binary cache unless
they need to be processed (i.e., only if binary stripping or ``upx``
processing is enabled). (:issue:`7595`)
* Fix a regression in bootloader that caused crash in onefile executables
when encountering a duplicated entry in the PKG/CArchive and the
``PYINSTALLER_STRICT_UNPACK_MODE`` environment variable not being set.
(:issue:`7613`)


Deprecations
~~~~~~~~~~~~

* The ``TOC`` class is now deprecated; use a plain ``list`` with the same
three-element tuples instead. PyInstaller now performs explicit
normalization (i.e., entry de-duplication) of the TOC lists passed
to the build targets (e.g., ``PYZ``, ``EXE``, ``COLLECT``) during their
instantiation. (:issue:`7615`)


Bootloader
~~~~~~~~~~

* Fix bootloader building with old versions of ``gcc`` that do not
support the ``-Wno-error=unused-but-set-variable`` compiler flag
(e.g., ``gcc`` v4.4.3). (:issue:`7592`)


Documentation
~~~~~~~~~~~~~

* Update the documentation on TOC lists and ``Tree`` class to reflect the
deprecation of the ``TOC`` class. (:issue:`7615`)


PyInstaller Core
~~~~~~~~~~~~~~~~

* Remove the use of the ``TOC`` class in the analysis / build process,
and use plain ``list`` instances instead. The implicit normalization
(de-duplication) of TOC entries performed by the ``TOC`` class has been
replaced with explicit normalization. The TOC lists produced by ``Analysis``
are explicitly normalized at the end of Analysis instantiation, before
they are stored in the Analysis properties (e.g., ``Analysis.pure``,
``Analysis.binaries``, ``Analysis.datas``). Similarly, the TOC lists
passed to the build targets (e.g., ``PYZ``, ``EXE``, ``COLLECT``) are
explicitly normalized as part of the targets' instantiation process.
(:issue:`7615`)

-------------------

Bugfix
~~~~~~

* Fix regression on platforms with strict data alignment requirements (such as
linux on ``armhf``/``armv7``), caused by bug in PKG/CArchive generation that
was introduced during the archive writer code cleanup. The regression caused
executable to terminate with ``Bus error`` on the affected platforms, such as
32-bit Debian Buster on Raspberry Pi 4. (:issue:`7566`)

-------------------

Bugfix
~~~~~~

* (Linux) Ignore the executable name resolution based on ``/proc/self/exe``
when the PyInstaller-frozen executable is launched via the ``ld.so``
dynamic loader executable. In such cases, the resolved name points to
the ``ld.so`` executable, causing the PyInstaller-frozen executable to
fail with *Cannot open PyInstaller archive from executable...* error.
(:issue:`7551`)
* Ensure that binaries that are manually specified in the .spec file (or via
corresponding :option:`--add-binary` or :option:`--collect-binaries`
command-line switches) undergo the binary dependency analysis, so their
dependencies are automatically collected. (:issue:`7522`)
* Extend the ``excludedimports`` mechanism rework from :issue:`7066`
to properly handle relative imports within the package. For example,
ensure that ``excludedimports = ['a.b']`` within the hook for package
``a`` takes effect when package ``a`` does ``from . import b`` (in
addition to ``from a import b``). (:issue:`7495`)
* Extend the ``excludedimports`` mechanism rework from :issue:`7066`
to properly handle the case of multiple submodules being imported in a
single ``from ... import ...`` statement (using absolute or relative import).
For example, when package ``c`` does ``from d import e, f``, we need to
consider potential ``excludedimports`` rules matching package ``d`` and,
if ``d`` itself is not excluded, potential rules individually matching
``d.e`` and ``d.f``. (:issue:`7495`)
* Fix marshal error in binary dependency search stage, caused by the list of
collected packages containing a ``modulegraph.Alias`` instance instead of only
plain :class:`str` instances. (:issue:`7515`)
* Reorganize the ``multiprocessing`` run-time hook to override ``Popen``
implementations only for ``spawn`` and ``forkserver`` start methods,
but not for the ``fork`` start method. This avoids a dead-lock when
attempting to perform nested multiprocessing using the ``fork`` start
method, which occurred due to override-provided lock (introduced in
:issue:`7411`) being copied in its locked state into the forked
sub-process. (:issue:`7494`)


Incompatible Changes
~~~~~~~~~~~~~~~~~~~~

* The ``archive_viewer`` utility has been rewritten with modified
command-line interface (``--log`` has been renamed to ``--list``) and
with changed output formatting. (:issue:`7518`)


Hooks
~~~~~

* (Windows) Improve support for ``matplotlib >= 3.7.0`` by collecting all
``delvewheel``-generated files from the ``matplotlib.libs`` directory,
including the load-order file. This is required when PyPI ``matplotlib``
wheels are used in combination with Anaconda python 3.8 and 3.9.
(:issue:`7503`)
* Add hook for ``PyQt6.QtSpatialAudio`` module, which was added in
``PyQt6`` 6.5.0. (:issue:`7549`)
* Add hook for ``PyQt6.QtTextToSpeech`` module, which was added in
``PyQt6`` 6.4 series. (:issue:`7549`)
* Extend ``PySide6`` hooks for ``PySide6`` 6.5.0 compatibility: add hooks
for ``QtLocation``, ``QtTextToSpeech``, and ``QtSerialBus`` modules
that were introduced in ``PySide`` 6.5.0. (:issue:`7549`)


Documentation
~~~~~~~~~~~~~

* Clarify the supported color specification formats and apply consistent
formatting of default parameter values in the splash screen documentation.
(:issue:`7529`)

------------------

Features
~~~~~~~~

* Choose :ref:`hooks provided by packages <provide hooks with package>` over
hooks from
`pyinstaller-hooks-contrib <https://github.com/pyinstaller/pyinstaller-hooks-contrib/>`_
if both provide the same hook. (:issue:`7456`)


Bugfix
~~~~~~

* Fix changes to :data:`sys.path` made in the spec file being ignored by hook
utility functions (e.g. :func:`~PyInstaller.utils.hooks.collect_submodules`).
(:issue:`7456`)

------------------

Features
~~~~~~~~

* Compile the collected GLib schema files using ``glib-schema-compiler``
instead of collecting the pre-compiled ``gschemas.compiled`` file, in
order to properly support collection of schema files from multiple
locations. Do not collect the source schema files anymore, as only
``gschemas.compiled`` file should be required at run time. (:issue:`7394`)


Bugfix
~~~~~~

* (Cygwin) Avoid using Windows-specific codepaths that require
``pywin32-ctypes`` functionality that is not available in Cygwin
environment. (:issue:`7382`)
* (non-Windows) Fix race condition in environment modification done by
``multiprocessing`` runtime hook when multiple threads concurrently
spawn processes using the ``spawn`` method. (:issue:`7410`)
* (Windows) Changes in the version info file now trigger rebuild of the
executable file. (:issue:`7338`)
* Disallow empty source path in the ``binaries`` and ``datas`` tuples
that are returned from the hooks and sanitized in the
``PyInstaller.building.utils.format_binaries_and_datas``. The empty
source path is usually result of an error in the hook's path retrieval
code, and causes implicit collection of the whole current working
directory. This is never the intended behavior, so raise a ``SystemExit``.
(:issue:`7384`)
* Fix *unknown log level* error raised with ``--log-level=DEPRECATION``.
(:issue:`7413`)


Incompatible Changes
~~~~~~~~~~~~~~~~~~~~

* The deprecated ``PEP-302`` ``find_module()`` and ``load_module()``
methods have been removed from PyInstaller's ``FrozenImporter``. These
methods have not been used by python's import machinery since
python 3.4 and ``PEP-451``, and were effectively left untested and
unmaintained. The removal affects 3rd party code that still relies
on ``PEP-302`` finder/loader methods instead of the ``PEP-451`` ones.
(:issue:`7344`)


Hooks
~~~~~

* Collect ``multimedia`` plugins that are required by ``QtMultimedia``
module starting with Qt6 v6.4.0. (:issue:`7352`)
* Do not collect ``designer`` plugins as part of ``QtUiTools`` module in
``PySide2`` and ``PySide6`` bindings. Instead, tie the collection of
plugins only to the ``QtDesigner`` module. (:issue:`7322`)


Module Loader
~~~~~~~~~~~~~

* Remove deprecated ``PEP-302`` functionality from ``FrozenImporter``.
The ``find_module()`` and ``load_module()`` methods are deprecated
since python 3.4 in favor of ``PEP-451`` loader. (:issue:`7344`)

------------------

Features
~~~~~~~~

* Add the package's location and exact interpreter path to the error message
for
the check for obsolete and PyInstaller-incompatible standard library
back-port
packages (``enum34`` and ``typing``). (:issue:`7221`)
* Allow controlling the build log level (:option:`--log-level`) via a
``PYI_LOG_LEVEL`` environment variable. (:issue:`7235`)
* Support building native ARM applications for Windows. If PyInstaller is ran
on
an ARM machine with an ARM build of Python, it will prodice an ARM
application. (:issue:`7257`)


Bugfix
~~~~~~

* (Anaconda) Fix the ``PyInstaller.utils.hooks.conda.collect_dynamic_libs``
hook utility function to collect only dynamic libraries, by introducing
an additional type check (to exclude directories and symbolic links to
directories) and additional suffix check (to include only files whose
name matches the following patterns: ``*.dll``, ``*.dylib``, ``*.so``,
and ``*.so.*``). (:issue:`7248`)
* (Anaconda) Fix the problem with Anaconda python 3.10 on linux and macOS,
where all content of the environment's ``lib`` directory would end up
collected as data due to additional symbolic link pointing from
``python3.1``
to ``python3.10``. (:issue:`7248`)
* (GNU/Linux) Fixes an issue with gi shared libraries not being packaged if
they don't
have version suffix and are in a special location set by ``LD_LIBRARY_PATH``
instead of
a typical library path. (:issue:`7278`)
* (Windows) Fix the problem with ``windowed`` frozen application being unable
to spawn interactive command prompt console via ``subprocess`` module due
to interference of the ``subprocess`` runtime hook with stream handles.
(:issue:`7118`)
* (Windows) In ``windowed``/``noconsole`` mode, stop setting ``sys.stdout``
and ``sys.stderr`` to custom ``NullWriter`` object, and instead leave
them at ``None``. This matches the behavior of windowed python interpreter
(``pythonw.exe``) and prevents interoperability issues with code that
(rightfully) expects the streams to be either ``None`` or objects that
are fully compatible with ``io.IOBase``. (:issue:`3503`)
* Ensure that ``PySide6.support.deprecated`` module is collected for
``PySide6`` 6.4.0 and later in order to enable continued support for
``|`` and ``&`` operators between Qt key and key modifier enum values
(e.g., ``QtCore.Qt.Key_D`` and ``QtCore.Qt.AltModifier``). (:issue:`7249`)
* Fix potential duplication of python extension modules in ``onefile``
builds, which happened when an extension was collected both as an
``EXTENSION`` and as a ``DATA`` (or a ``BINARY``) TOC type. This
resulted in run-time warnings about files already existing; the
most notorious example being ``WARNING: file already exists but
should not:
C:\Users\user\AppData\Local\Temp\MEI1234567\torch\_C.cp39-win_amd64.pyd``
when building ``onefile`` applications that use ``torch``. (:issue:`7273`)
* Fix spurious attempt at reading the ``top_level.txt`` metadata from
packages installed in egg form. (:issue:`7086`)
* Fix the log level (provided via :option:`--log-level`) being ignored by some
build steps. (:issue:`7235`)
* Fix the problem with ``MERGE`` not properly cleaning up passed
``Analysis.binaries`` and ``Analysis.datas`` TOCs due to changes made to
``TOC`` class in PyInstaller 5.0. This effectively broke the supposed
de-duplication functionality of ``MERGE`` and multi-package bundles,
which should be restored now. (:issue:`7273`)
* Prevent ``$pythonprefix/bin`` from being added to :data:`sys.path` when
PyInstaller is invoked using ``pyinstaller your-code.py`` but not using
``python -m PyInstaller your-code.py``. This prevents collection mismatch
when
a library has the same name as console script. (:issue:`7120`)
* Prevent isolated-subprocess calls from indefinitely blocking in their
clean-up codepath when the subprocess fails to exit. After the grace
period of 5 seconds, we now attempt to terminate such subprocess in
order to prevent hanging of the build process. (:issue:`7290`)


Incompatible Changes
~~~~~~~~~~~~~~~~~~~~

* (Windows) In ``windowed``/``noconsole`` mode, PyInstaller does not set
``sys.stdout`` and ``sys.stderr`` to custom ``NullWriter`` object anymore,
but leaves them at ``None``. The new behavior matches that of the windowed
python interpreter (``pythonw.exe``), but may break the code that uses
``sys.stdout`` or ``sys.stderr`` without first checking that they are
available. The code intended to be run frozen in ``windowed``/``noconsole``
mode should be therefore be validated using the windowed python interpreter
to catch errors related to console being unavailable. (:issue:`7216`)


Deprecations
~~~~~~~~~~~~

* Deprecate bytecode encryption (the ``--key`` option), to be removed in
PyInstaller v6.0. (:issue:`6999`)


Hooks
~~~~~

* (Windows) Remove the ``subprocess`` runtime hook. The problem with invalid
standard stream handles, which caused the ``subprocess`` module raise an
``OSError: [WinError 6] The handle is invalid`` error in a ``windowed``
``onefile`` frozen application when trying to spawn a subprocess without
redirecting all standard streams, has been fixed in the bootloader.
(:issue:`7182`)
* Ensure that each ``Qt*`` submodule of the ``PySide2``, ``PyQt5``,
``PySide6``,
and ``PyQt6`` bindings has a corresponding hook, and can therefore been
imported in a frozen application on its own. Applicable to the latest
versions of packages at the time of writing: ``PySide2 == 5.15.2.1``,
``PyQt5 == 5.15.7``, ``PySide6 == 6.4.0``, and ``PyQt6 == 6.4.0``.
(:issue:`7284`)
* Improve compatibility with contemporary ``Django`` 4.x version by removing
the override of ``django.core.management.get_commands`` from the ``Django``
run-time hook. The static command list override is both outdated (based on
``Django`` 1.8) and unnecessary due to dynamic command list being properly
populated under contemporary versions of ``PyInstaller`` and ``Django``.
(:issue:`7259`)
* Introduce additional log messages to ``matplotlib.backend`` hook to
provide better insight into what backends are selected and why when the
detection of ``matplotlib.use`` calls comes into effect. (:issue:`7300`)


Bootloader
~~~~~~~~~~

* (Windows) In a ``onefile`` application, avoid passing invalid stream handles
(the ``INVALID_HANDLE_VALUE`` constant with value ``-1``) to the launched
application child process when the standard streams are unavailable (for
example, in a windowed/no-console application). (:issue:`7182`)


Bootloader build
~~~~~~~~~~~~~~~~

* Support building ARM native binaries using MSVC using the command
``python waf --target-arch=64bit-arm all``. If built on an ARM machine,
``--target-arch=64bit-arm`` is the default. (:issue:`7257`)
* Windows ARM64 bootloaders may now be built using an ARM build of clang with
``python waf --target-arch=64bit-arm --clang all``. (:issue:`7257`)