This new release of archinstall addresses the many issues raised by the community.
We would like to thank every single one who submitted issues, gave feedback and most importantly suggested or contributed to solutions to all these issues.
This is by no means a perfect release, a lot of work remains ahead. But we believe this is a step in the right direction with both accessibility improvements as well as stability and some security enhancements for the guided template that we ship. Below is a list of all the changes since the previous version, and we will begin with the breaking changes to highlight those.
Breaking changes
* `Filesystem()`'s context manager *(using `with Filesystem()`)* no longer wipes the disk on entry, it no longer detects or handles differences in partition table vs expected partition table format (gpt vs mbr), instead those operations have moved to individual functions, see below in the `Filesystem` header as well as the `Installer` header, where mount logic has moved to.
* Going from `*` imports to targeted imports. This should not cause issues, but if there's any API call not available do let us know.
* `BlockDevice().json()` now only returns the string representation of path to the blockdevice *(`"/dev/sda"` for instance)*
* `BlockDevice().__dump__()` has one additional dimension to the JSON dictionary, where it represent itself as `"/dev/sda" : {"info": "value"}` rather than a flat structure.
* `BlockDevice().device_or_backfile` now return the backfile when the device is a loopdevice, `BlockDevice().device` still returns limited information of the type *(raid, crypt and others)*.
* `Partition().size` is now a read-only value and the parameter `Partition(size=X)` has hence been removed
* `Partition().allow_formatting` has been removed/deprecated, this is handled via direct function calls instead of automatically wiping a partition.
* `Partition().safe_to_format()` has been deprecated and removed since no automatic formatting or changes are done. Any destructive changes has to be explicitly called, and there for that is the safeguard instead.
* `Partition().format()` no longer has the argument `allow_formatting` as it's implicitly set whenever the function is called. This means that "testing" which file systems are supported is no longer supported, but instead we have a curated list of supported file systems that can be used.
* `Partition().format()` no longer uses the nomenclature `vfat` for the `filesystem` argument, instead `fat32` is used. This to better reflect the `mount` call later where `fat32` is specified and better reflects `man parted`'s expectations.
* `archinstall.JSON` serializer now correctly removes any key in a dictionary that starts with `!`, for instance `"!password" : "moo"` will get filtered out. To serialize a structure without loosing values, use `archinstall.UNSAFE_JSON` which is unsafe to use unless you know what you're doing.
* The `gfx_driver` argument in profiles etc no longer has supported for `Nvidia`, and instead of using sub-menu's under the `Nvidia` category there is now two top level choices of either `Nvidia (open-source)` and `Nvidia (proprietary)`. They both do the same as before, just on a top level of choices and as a single argument in JSON configurations/profiles.
* The default installation mountpoint is no longer `/mnt` but instead `/mnt/archinstall`
* Disk layout and configuration has moved from `user_configuration.json` into `user_disk_layouts.json`
*(It's still supported for as long as possible to define these in `user_configuration.json` for backwards compatibility reasons)*
* User credentials have moved from `user_configuration.json` into `user_credentials.json`
*(It's still supported for as long as possible to define these in `user_configuration.json` for backwards compatibility reasons)*
* Keywords in the JSON configuration has changed in the guided installer:
* `keyboard-language` has been re-named to `keyboard-layout`
* `harddrive` has been re-named to `harddrives` *(and changed structure)*
* `disk_layouts` have been added and replaces `harddrives` partially
* `users` have changed to `!users`
* `superusers` have changed to `!superusers`
Highlighted new features
* GRUB and Disk Encryption now properly works.
* BTRFS subvolumes *(very early beta feature, hence some limitations for initial tests. Supports only a [nested structure](https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Nested) and some [issues are known](https://github.com/archlinux/archinstall/issues/718) with this approach)*
* Archinstall now detects if `espeakup.service` is active on the installation ISO and copies the setup over to the installed medium automatically.
* Now supports multiple encrypted volumes *(slightly limited, it's either on or off, but multiple partitions will be encrypted)*. All encrypted partitions are protected by the passphrase as usual and stored in key-slot 0, bu every partition that is not the root partition will also have a computer generated passphrase stored on the root device under `/dev/cryptsetup-keys.d/` and that passphrase is stored in key-slot 1 on the partition. It's a unique passphrase per volume.
* More reliable disk operations *(partitioning, encryption and mounting)*
* A rudimentary [plugin-support](https://github.com/archlinux/archinstall/blob/master/archinstall/lib/plugins.py), allowing users to create their own plugins. There are four ways of activating the plugin support, either via `--plugin=url|location`, via en entry in `--config` as `{"plugin": "url|location"}` or via the API call `archinstall.load_plugin()` or finally via `pip install yourplugin` assuming that the plugin registers itself via Pythons builtin entry-point system as `archinstall.plugin`. These plugins are triggered/called at various points in the installation and more documentation on this specific will come and be continuously updated. One such step is `archinstall.pacstrap()` which calls `on_pacstrap(package_list)` in the plugin and the return value should be a curated list of packages, this enables for instance custom plugins to be written and deal with for instance AUR packages and deal with the custom operations needed and filter out those packages from the rest of the install process, leaving only official packages to be dealt with by the official installation process. *(again, just an example)*.
* [swap](https://wiki.archlinux.org/title/Swap) has been added into the mix using [zram](https://www.kernel.org/doc/html/latest/admin-guide/blockdev/zram.html) and [zram-generator](https://wiki.archlinux.org/title/Swap#zram-generator).
* Manual partitioning has been re-worked *(but released partially unfinished, the reason being it was impossible to release this version where this specific change was removed - as it's so intertwined with everything else that's been re-worked)*
Detailed changelog
Parameters
* More static arguments have been added to the initialization making for a more traditional argument parsing of known arguments.
* `--creds` is a new argument that take the previous `"users"`, `"superusers"` and `"root-password"` keys & values, this allows for separation of JSON machine configuration and user credentials which might want to be stored separately for security reasons.
* `--disk_layouts` is a new argument that takes the previous structure of `"harddrive"` which later was renamed `"harddrives"`. The old JSON structure of `"harddrives"` still remains in the `--config` input as an indicator of which harddrives to use and find from `--disk_layouts`.
* `--dry-run` has been added, which will abort before any changes are made on hardware. This allows for generating a configuration file which can then be passed to either of `--config`, `--creds` and `--disk_layouts`.
* `--mount-point` has been added to enable choosing where the installed system gets mounted during installation/configuration.
* `--plugin` has been added for loading external archinstall plugins via parameter.
Library changes
* `archinstall/lib/disk.py` has been split up into `archinstall/lib/disk/*.py` since it grew out of hand. The exposure of API calls should remain the same as they're exposed via `__init__.py`, but for contributors this change is worth noting when finding functions and files.
* Going from `*` imports to targeted imports. This should not cause issues, but if there's any API call not available do let us know.
General changes
* A dummy `epoll()` has been added in order to be able to run certain tests on Windows, for instance "reach mirror"-tests etc without causing import issues.
* `archinstall.generate_password(length)` has been added to generate a rudimentary password for disk encryption and other things. This using the `secrets` library and `string.printable` as a haystack.
* `archinstall.json_dumps(*args, **kwargs)` performs a safe `json.dumps()` call using the `archinstall.JSON` serializer
* `archinstall.UNSAFE_JSON()` has been added to serialize JSON structures without consideration of private information.
* `archinstall.SysCommandWorker()` now outputs the stdout as-is, without any attempt to make it pretty.
* `archinstall.SysCommandWorker()` *(and in extension, even `SysCommand()`)* now logs all commands executed under `/var/log/archinstall/cmd_history.txt` for easier debugging and learning which commands we executed. Currently we don't filter anything, so there will be a lot of spam from `parted()` calls and `lsblk` multiple times *"for no reason"*.
* `archinstall.SysCommand()` has support for retrieving slices using `SysCommand("ls")[:10]` for instance, where the output generated by the internal `SysCommandWorker()` call will be simplified into retrieving the final output and only the first 10 bytes in this example.
* `archinstall.cpuinfo()` has been created to return CPU information from `/proc/cpuinfo` in a more general sense
* `archinstall.has_amd_cpu()` and `archinstall.has_intel_cpu()` now uses the above `cpuinfo()` call.
* `archinstall.meminfo()` has been created to return memory information from `/proc/meminfo` as a whole or a specific key if specified.
* `archinstall.graphics_devices()` now correctly catches `" 3D "` entries from `lspci` correctly.
* `archinstall.cpu_vendor()` now uses `cpuinfo()` instead.
* `archinstall.cpu_model()` now uses `cpuinfo()` instead
* `archinstall.sys_vendor()` has been added which returns the vendor as specified by `/sys/devices/virtual/dmi/id/sys_vendor`
* `archinstall.product_name()` has been added which returns the product name from `/sys/devices/virtual/dmi/id/product_name`
* `archinstall.mem_available()` returns the available memory using `meminfo()`
* `archinstall.mem_free()` returns the available free memory using `meminfo()`
* `archinstall.mem_total()` returns the total amount of memory using `meminfo()`
* `archinstall.virtualization()` returns the used virtualization platform using `systemd-detect-virt`
* `archinstall.sort_mirrorlist()` has been added to sort the `/etc/pacman.d/mirrorlist` according to preference of URL prefix. Defaults to HTTPS before HTTP, while preserving origin of the servers in groups.
* `archinstall.filter_mirrors_by_region()` now takes the `sort_order` parameter, which uses the `sort_mirrorlist()` call to arrange mirrors in a preferred way.
* `archinstall.use_mirrors()` now correctly iterates over multiple regions to support multi-region support for mirror selection *(menu system will be updated in the next release to offer this)*
* `archinstall.list_mirrors()` also supports the `sort_order` parameter as previously mentioned.
* `archinstall.storage` has gotten three new keys, `ENC_IDENTIFIER` which is the prefix for all loopdevices upon opening encrypted volumes, `DISK_TIMEOUTS` is the time in seconds in which archinstall should sleep pending a retry of the same operation, `DISK_RETRY_ATTEMPTS` which is the amount of retries of the same disk operation before quitting *(and usually exception-error out)*
* `archinstall.Boot()` which can temporarily boot the installed system without having to reboot the running machine now properly avoids using output-pagers when outputting the terminal output which is read by archinstall. In the future, we will use `--console=pipe` instead of reading the stdout via *"brute force"*
* `archinstall.Boot()` no longer uses `machinectl` to run commands in the temporary boot environment, instead it uses `systemd-run`
* `archinstall.select_encrypted_partitions()` has been added to mark partitions as encrypted. In the future this function will allow for selecting and manipulating which partitions are encrypted or not. For now, when called it will mark all partitions *(except `/boot`)* as encrypted. There is no toggle for off in this release and no user-interaction.
* A bunch of language improvements throughout the output
Profiles and it's API
* `archinstall.is_desktop_profile()` has been added to determine whether or not a specific profile is of the category *"desktop"*.
* `archinstall.Profile().get_profile_description()` has been added to return a profiles `__description__` variable if one exists.
* [cutefish](https://github.com/cutefishos) has been [added](https://github.com/archlinux/archinstall/blob/2058e61dc7d0662400a99157671d905e8c7c55f7/profiles/cutefish.py) as a desktop environment.
* `lightdm-gtk-greeter` have changed to `lightdm-deepin-greeter`
* `gnome-software-packagekit-plugin` have been added to `gnome` in order to get the Gnome Software app to work properly
* `ark` has been added to `KDE` to allow for archive management
* `xarchiver` have been added to `xfce4` to allow for archive management
* `gvfs` have been added to `xfce4` to fix *"trash"*
* `network-manager-applet` added to `xfce4` to allow for network management using NetworkManager
Guided installation changes
* `archinstall.ask_for_swap()` has been added as a API call, which is used in the guided installer to ask if the users wants swap or not.
* `archinstall.ask_for_bootloader()` have been modified to allow listing *"advanced"* boot loaders via the `advanced_options=` parameter.
* `archinstall.ask_for_audio_selection()` now defaults to pipewire instead of pulseaudio, but still supports choosing either or none.
* `archinstall.ask_for_main_filesystem_format()` now supports the `advanced_options=` parameter, which will unlock `ntfs` as the root filesystem *(hidden and triggered by `--advanced`)*
* `archinstall.get_default_partition_layout()` returns the *"ideal"* disk layout for one or more disk setups.
* In turn, for single disk `archinstall.suggest_single_disk_layout()` has been added to suggest a single disk layout setup
* For multi-disk setups `archinstall.suggest_multi_disk_layout()` has been added to suggest a multi-disk layout
* `archinstall.manage_new_and_existing_partitions()` has been added to handle the menu-operations and alternatives for manipulating and managing partitions during manual configuration. This feature is in an early stage and will contain bugs, dead code paths and other edge cases that we haven't picked up yet. We release it partially unfinished because this release is getting too big as it is and we need more user input. And finding the optimal layouts and suggestions are tricky. **we strongly suggest manually partitioning the disk and using it *"as is"* if you want to get full control of partition layouts** but we provide this tool as a test to see if it would be viable.
* General warnings have been added surrounding AMD, Intel and Nvidia hardware.
* Graphics driver defaults to `All open-source (default)` if no other option is given and a desktop profile has been selected.
* The guided installation will now log and output the relevant hardware specs, which we can use in support arrends assuming the user is willing to include the log when asking for support.
* `load_config()` has been added in guided to facilitate the setup of all the required arguments prior to running the installation steps.
* `perform_filesystem_operations()` have been added to deal with disk operations. A lot of the encryption and formatting have moved away from the guided core and in to the respective classes *(see other parts of the release notes)* to simplify the guided profile a bit and separate it's duties.
* `perform_installation_steps()` have been re-named to `perform_installation()` which now deals with mounting disk layouts and is cleaned up to only contain installation steps and no disk operations *(other than mounting)*
`archinstall.Installer` *(and the installer.py file in general)*
* `archinstall.InstallationFile()` has been added, which is a per-installation helper to create files using the `with open()` context with correct file permissions as seen by the installation. `owner=` will look up the user-id within the installation and set ownership upon closing.
* `archinstall.accessibility_tools_in_use()` returns if `espeakup.service` is active in the live medium, can be used to carry over functionality into the installed system.
* `archinstall.Installer()` now installs `archinstall.__accessibility_packages__` when detected in use on the live medium.
* `archinstall.Installer().partitions()` returns any `Partition()` object in use under the installers mountpoint.
* `archinstall.Installer().mount_ordered_layout()` is a new function to mount a defined JSON structure of partitions and mountpoints, as well as meta-info such as encryption and passwords. This replaces the old way of setting up block devices, partitions and dealing with encryption.
* `archinstall.Installer()`
* `archinstall.Installer().mount_ordered_layout()` will create a encryption key on `/` *(root)* if multiple volumes are detected as encrypted. It will then add that encryption key in key-slot 1 on that encrypted volume. Where key-slot 0 will remain the overall encryption key used for the root device. This ensures a secure automatic-unlock key that the system can use during boot, but allows a fallback manual passphrase for manual mounting from say other operating system instances or recovery modes.
* `archinstall.Installer().activate_ntp()` has been replaced by `archinstall.Installation().activate_time_syncronization()` but the old function will live on for a version or two with a warning.
* `archinstall.Installation().enable_espeakup()` enables text to speech services in the installed system.
* `archinstall.Installation().run_as()` has been improved to deal with quote issues a bit better *(still not perfect)*
* `archinstall.Installation().mkinitcpio()` will now remove `fsck` in the HOOKS if `ntfs` is detected as the root filesystem.
* `archinstall.Installation().setup_swap()` is a new helper function to configure swap, currently only `zram` is supported.
* `archinstall.Installation().add_bootloader()` now correctly configurs and creates boot entries based on the selected kernels
* `archinstall.Installation().add_bootloader()` now has support for [efistub](https://wiki.archlinux.org/title/EFISTUB) when used with your own custom installation scripts or via passing the `--advanced` flag to `archinstall`.
* `archinstall.Installation().chown()` has been added as a convenience function to chown files inside the installation using the installation user-map.
* `archinstall.Installation().create_file()` is a wrapper for `archinstall.InstallationFile()`.
`archinstall.luks`
* `archinstall.luks2()` has two new functions:
* `.add_key()` which adds a keyfile to a slot on a existing encrypting volume
* `.crypttab()` which creates a `/etc/crypttab` entry for a volume in the installation *(the installation has to be passed to the function)*
`archinstall.BlockDevice`
* More detailed output in `__repr__` to represent the class object instance
* Now supports `len()` to get the amount of partitions on the block device.
* Now supports `sorted()` by comparing `self.path` against other objects `.path`.
* `BlockDevice().partition_type` returns the partition table type *(GPT or MBR)*
* `BlockDevice().size` has been added to return the size of the device in GB *(not GiB)*
* `BlockDevice().bus_type` has been added to return the type of underlying bus *(nvme etc)*
* `BlockDevice().spinning` has been added, returns `True` if the underlying physical device is a spinning device
* `BlockDevice().free_space` has been added to report how much available space is free on the device *(still a bit buggy when no partitions exist)*
* `BlockDevice().largest_free_space` has been added and reports the largest continuous free space on the block device.
* `BlockDevice().first_free_sector` has been added and reports the first free sector and is reported in MB, GB or other similar unit.
* `BlockDevice().first_end_sector` has been added and reports the first free sector closest to the first available end, also in units of MB etc *(if partitions exist, it will report the last sector right before that partition)*
* `BlockDevice().get_partition(uuid)` returns the `Partition()` object from the block device partitions table based on the `uuid` *(PARTUUID)` filter.
`archinstall.Filesystem`
* `Filesystem().partuuid_to_index` has been added to give the positional index *(order)* of a specific partition in relation to other partitions based on it's `uuid` *(PARTUUID)*
* `Filesystem().load_layout` has been added to load a specific partition layout, this function will be moved in the future to create better separation of duties, but in the large split/move/change this is where it ended up. The function takes a dictionary structure *(see examples in the examples catalogue or generate with `--dry-run`)* and converts it into disk and partition operations, such as `mklabel`, `mkpart` and encryption calls.
* `Filesystem().add_partition()` has been re-worked in a way to allow for more gracefully letting the kernel come up to speed. it will re-try grabbing the newly formed partitions PARTUUID for a set period of time before giving up.
`archinstall.Partition`
* A new `__dump__` function has been added for use with `archinstall.JSON` serializer
* `Partition().sector_size` has been added to return the logical sector size
* `Partition().start` has been added to return the start sector from `(sfdisk /path).start`
* `Partition().end` has been added to return the end sector from `(sfdisk /path).size`
* `Partition().size` has been added as a read only property which returns the size of the partition in GB
* `Partition().boot` has been added and returns if the partition is set/marked as bootable
* `Partition().partition_type` has been added and returns the partition table type that this partition lives under
* `Partition().uuid` now features retrying the `PARTUUID` retrieval, as the kernel some times hands over to the user before certain disk operations are actually complete. These retry-attempts are configurable both in terms of delay-time between and amount of attempts.
* `Partition()._safe_uuid()` has been added as a compliment to `.uuid` but does not raise any exceptions and does not attempt to do any re-tries. This feature is meant to be a safe way to "get" the PARTUUID without halting other operations, such as printing information or representation of the `Partition()` object.
* `Partition().format()` has support for additional file systems
* `ext2`
* `ntfs` *(as of writing, only works with the latest Linux kernel and not `-lts` kernels. And it's highly experimental and lacks fscheck, but its supported)*
* `Partition().format()` now supports passing options such as compression or setting [case-insensitivity](https://github.com/archlinux/archinstall/issues/380) via the JSON config or API calls.
btrfs and subvolumes
*This feature is released as a BETA feature, do expect issues but feel free to try out the limited support for btrfs subvolumes and report feedback*
* `archinstall.mount_subvolume` has been added as a helper function to mount a btrfs subvolume to a specific location
* `archinstall.create_subvolume` has been added as a helper function to create a subvolume on a existing btrfs mounted volume location
`archinstall.lib.disk.helpers*`, `archinstall.lib.disk.user_guides*` & `archinstall.lib.disk.validators*`
* `archinstall.convert_size_to_gb()` has been added and takes a size in `bytes` and converts it to `gigabytes` *(I sure hope I get the convention right here, bytes to GB and not GiB)*.
* `archinstall.sort_block_devices_based_on_performance()` performs a rudimentary sort of block devices using an *"algorithm"* based on *"performance"*.
* `archinstall.filter_disks_below_size_in_gb()` generates an iterator which filters out blockdevices based on a lowest GB number
* `archinstall.select_largest_device()` selects the largest disk from a collection of disks
* `archinstall.convert_to_gigabytes(string)` is a limited helper function to convert MB and TB to GB
* `archinstall.get_mount_info()` now supports more features
* Traversing the targeted structure backwards when `traverse=True`.
* Added `return_real_path=True` which when True returns two return values rather than the mount information alone, this is so that in junction with `traverse` it will return the path in which it found the mount information. `/mnt/archinstall/opt/` for instance will not return any info, but `/mnt/archinstall` will so that's the path that will be returned together with the mount information at that location.
* `archinstall.encrypted_partitions()` generates an iterator containing all partition blobs in a JSON structure that are encrypted
* `archinstall.find_partition_by_mountpoint()` iterates a JSON structure and find the relative mountpoint, `/boot` for instance can be located *(note: not `/mnt/boot`)*
* `archinstall.partprobe()` A new helper function which simply calls `partprobe`
* `archinstall.convert_device_to_uuid()` A new helper function to convert `/dev/sda` or `/dev/sda2` to a UUID *(not PARTUUID)*
* `archinstall.suggest_single_disk_layout()` Has been added and performs rudimentary guesswork to set up the best partition layout and/or subvolumes for the selected block device.
* `archinstall.suggest_multi_disk_layout()` Has been added and performs rudimentary guesswork to set up the best partition layout and/or subvolumes for multiple selected block devices.
* `archinstall.valid_parted_position()` Has been added as a validation function before sending start/end positions to `parted`
* `archinstall.valid_fs_type()` Has been added and replaces the previous `Partition().format()` of `/dev/null` to detect if a file system is valid or not. Instead, this function will perform a static lookup of hardcoded and supported file systems.
Building and contributing
* Updated [flake8](https://github.com/archlinux/archinstall/blob/e729457b6c12a00b17207254ee72e98b78912f8d/.flake8#L4-L9) to be a bit more strict but stay in line with our contribution guidelines allowing some flexibility to the code.
* Added automatic [ISO builds](https://github.com/archlinux/archinstall/blob/e729457b6c12a00b17207254ee72e98b78912f8d/.gitlab-ci.yml) based on the official Arch Linux ISO configuration [releng](https://wiki.archlinux.org/title/Archiso#Prepare_a_custom_profile) for GitLab, this in perparation for the potential/eventual move to GitLab for all Arch Linux related projects.
* Automatic build of man pages in the [PKGBUILD](https://github.com/archlinux/archinstall/blob/e729457b6c12a00b17207254ee72e98b78912f8d/PKGBUILD#L26)