Skip to contentSkip to conversation

Fedora 36, grub2, UEFI: booting after adding 2nd disk to root?

DISCUSSION Viktor Dukhovni 2 participants 3 messages Sep 4, 2022 – Sep 5, 2022 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Context:

Distro: Fedora 36
Arch: x86_64
Boot: grub2 + UEFI, GPT
Kernel: Linux 5.19.6-200.fc36.x86_64
Btrfs: btrfs-progs-5.18-1.fc36.x86_64

Installed on: nvme0n1 (~2TB SSD)
- GPT first 2048 sectors of disk
- p1: /boot/efi, vfat 511MB
- p2: 128GB btrfs /
- p3: initially free space (ultimately for Postgres)

The system has a second identically partitioned disk: nvme1n1

After installation I naively added nvme1n1p2 to the root btrfs hoping to extend the root fsit over both disks (not RAID1, just concat or stripe).

    # btrfs device nvme1n1p2 /

expecting this to be handled trasparently. After a bit more housekeeping I attempted to reboot the system, and it failed to boot.

I don't have console access (system racked and installed for me by someone else, will have to wait for hands-on help), but I guessed (it seems correctly) that splitting the [rb]oot btrfs FS across multiple devices is not handled automatically at boot time by grub2.

The system is freshly built, and it is perhaps simplest to just reinstall, but is there a not too painful way to recover it?

More importantly, I am having trouble finding a canonical description of what steps are needed to correctly configure a multi-device btrfs boot filesystem.

* Is it possible/best to create such a multi-device btrfs rootfs at Fedora 36 install time? Will the installer manage to set up grub2 correctly in that case?

* If not, what is the correct way to keep the system bootable after adding another device to the root btrfs?

* A secondary question: To split off /home, /var/log and /var/spool into separate subvoluments, should these be nested or top-level? The initial state created by the installer was:

      # btrfs subvolume list /
      ID 256 gen 362 top level 5 path root
      ID 257 gen 68 top level 256 path var/lib/portables
      # btrfs subvolume get-default /
      ID 5 (FS_TREE)
      # grep btrfs /etc/fstab
      UUID=... / btrfs   subvol=root,compress=zstd:1 0 0

Naive attempts to create the subvolumes makes them nested, perhaps because I don't have an empty unmounted top-level volume?

* And finally, the free space is intended for a Postgres database using most of the space on the two disks. Would a suitably tuned subvolme be a good choice, or would ext4 or xfs be more prudent (and performant?).

It is perhaps simplest to make "p2" span also the free space that is currently reserved under p3, and tell btrfs to stripe the disks?

Alternatively, I could build a separate btrfs over the two "p3" partitions.

My Unix experience spans some decades starting with SunOS 3.x, and I have been using ZFS on FreeBSD for 5+ years, but I am a "btrfs" noob, so your answers do not need to be "dumbed down", but can't assume knowledge of "btrfs", and particularly its integration with "grub2", beyond a few days of reading manpages.

Signature · 1 line
    Viktor.
#1 lore raw
Quoted 14 lines · Viktor Dukhovni

Context:

Distro: Fedora 36
Arch: x86_64
Boot: grub2 + UEFI, GPT
Kernel: Linux 5.19.6-200.fc36.x86_64
Btrfs: btrfs-progs-5.18-1.fc36.x86_64

Installed on: nvme0n1 (~2TB SSD)
- GPT first 2048 sectors of disk
- p1: /boot/efi, vfat 511MB
- p2: 128GB btrfs /
- p3: initially free space (ultimately for Postgres)

Quoted 6 lines

The system has a second identically partitioned disk: nvme1n1

After installation I naively added nvme1n1p2 to the root btrfs hoping to extend the root fsit over both disks (not RAID1, just concat or stripe).

    # btrfs device nvme1n1p2 /
# btrfs device add /dev/nvme1n1p2 /

?

Quoted 3 lines

expecting this to be handled trasparently. After a bit more housekeeping I attempted to reboot the system, and it failed to boot.

What does happen? What's the last moment of normal boot and the first unexpected messages of the failure?

i.e. do you see a GRUB menu? What happens if you edit the boot entry, remove "rhgb quiet" and add "rd.shell" then booth (F12 or Control-X)?

If that fails, try the same steps with the GRUB entry that has the word "rescue" in the name, i.e. remove rhgb quiet and add rd.shell.

Quoted 4 lines

I don't have console access (system racked and installed for me by someone else, will have to wait for hands-on help), but I guessed (it seems correctly) that splitting the [rb]oot btrfs FS across multiple devices is not handled automatically at boot time by grub2.

GRUB can read btrfs multiple device pools fine. I'm a little suspicious that maybe the NVMe drives use different drivers (?) and one of the kernel drivers isn't in the (older) initramfs, therefore it doesn't show up at the same time as the original NVMe drive. Since this 2nd drive is now required to mount the file system, udev is waiting indefinitely for the drive to appear. That module might be in the "rescue" boot entry, which is not really a special kernel but it's a "no host-only" initramfs created by dracut -N, so it might have the kernel module.

If it boots, then all you need to do is recreate the initramfs for all the kernel entries. e.g.

dracut -f /boot/initramfs-5.19.4-200.fc36.x86_64.img 5.19.4-200.fc36.x86_64

For each kernel version you have installed.

Quoted 7 lines

More importantly, I am having trouble finding a canonical description of what steps are needed to correctly configure a multi-device btrfs boot filesystem.

* Is it possible/best to create such a multi-device btrfs rootfs at Fedora 36 install time? Will the installer manage to set up grub2 correctly in that case?

You can do it at installation time if you choose both drives for installation. Using the default "automatic" partitioning option, the installer will do a default mkfs.btrfs on the two NVMe partitions, resulting in single profile data, and raid1 profile for metadata. You'd need to use Custom partitioning to choose raid1 for data, but it sounds like you don't want that anyway so you can just do an automatic installation with both drives selected.

The GRUB configuration should be the same as with a single drive, all GRUB does is look for an fs UUID. And both drives have the same fs UUID. The GRUB btrfs module knows how to find additional devices for the same file system.

Quoted 2 lines

* If not, what is the correct way to keep the system bootable after adding another device to the root btrfs?

It's a guess that this is related to initramfs. I'm not sure what else it could be.

Quoted 3 lines

* A secondary question: To split off /home, /var/log and /var/spool into separate subvoluments, should these be nested or top-level?

Anaconda creates subvolumes in the top-level of the file system anytime you create explicit mount-points in the Custom or Advanced-Custom partitioning UI. And they follow a simple / becomes _ naming system so it'll be var_log and var_spool for the subvolume names.

Quoted 6 lines

The
initial state created by the installer was:

      # btrfs subvolume list /
      ID 256 gen 362 top level 5 path root
      ID 257 gen 68 top level 256 path var/lib/portables

/var/lib/portables and /var/lib/machines are created by systemd, not the installer

Quoted 2 lines

Naive attempts to create the subvolumes makes them nested, perhaps because I don't have an empty unmounted top-level volume?

To do this manually, you need to mount the btrfs file system, e.g. at /mnt, without any mount options. It'll have its top-level mounted, and you'll see the subvolumes in /mnt, looking like they're directories.

Not that all Btrfs mounts are effectively bind mounts. You can mount a Btrfs file system multiple times *with the same kernel* (just to disambiguate from the rare and fatal case of mounting a Btrfs file system with a host and guest VM kernel).

Quoted 4 lines

* And finally, the free space is intended for a Postgres database using most of the space on the two disks. Would a suitably tuned subvolme be a good choice, or would ext4 or xfs be more prudent (and performant?).

It really depends on the workload of the database. How big is it, how busy is it, is it compressible? Etc. You might find the performance is OK if you enable WAL, and the workload isn't significant. Also, this is on NVMe drives, which tend to have such low latency that you may not notice any sub-optimal performance issues. Also, other databases work OK or even quite well on Btrfs, e.g. sqlite is pretty OK, and RocksDB is great. Postgresql just has a write pattern that I guess isn't great on Btrfs. The only way to know is to bench mark your workload with various configurations and see what works best or at least what works well enough you don't notice a problem.

Quoted 3 lines

It is perhaps simplest to make "p2" span also the free space that is currently reserved under p3, and tell btrfs to stripe the disks?

Btrfs will let you do that but there's no performance advantage of striping across partitions on the same physical drive.

Signature · 1 line
Chris Murphy
#2 lore raw
Quoted 7 lines · Chris Murphy
Quoted 4 lines

After installation I naively added nvme1n1p2 to the root btrfs hoping to extend the root fsit over both disks (not RAID1, just concat or stripe).

    # btrfs device nvme1n1p2 /
# btrfs device add /dev/nvme1n1p2 /

Yes, sorry, of course "add".

Quoted 5 lines
Quoted 2 lines

expecting this to be handled trasparently. After a bit more housekeeping I attempted to reboot the system, and it failed to boot.

What does happen? What's the last moment of normal boot and the first unexpected messages of the failure?

It turns out (once hands-on help arrived) that the machine did boot, but for some mysterious reason failed to detect the network cards. Which is not a btrfs issue, so as you note the "UUID" is sufficient to find all the devices.

Quoted 11 lines
Quoted 3 lines

* Is it possible/best to create such a multi-device btrfs rootfs at Fedora 36 install time? Will the installer manage to set up grub2 correctly in that case?

You can do it at installation time if you choose both drives for installation. Using the default "automatic" partitioning option, the installer will do a default mkfs.btrfs on the two NVMe partitions, resulting in single profile data, and raid1 profile for metadata. You'd need to use Custom partitioning to choose raid1 for data, but it sounds like you don't want that anyway so you can just do an automatic installation with both drives selected.

I went with "raid0" for data, manually after the install.

Quoted 8 lines
Quoted 2 lines

* A secondary question: To split off /home, /var/log and /var/spool into separate subvoluments, should these be nested or top-level?

Anaconda creates subvolumes in the top-level of the file system anytime you create explicit mount-points in the Custom or Advanced-Custom partitioning UI. And they follow a simple / becomes _ naming system so it'll be var_log and var_spool for the subvolume names.

Thanks.

Quoted 6 lines
Quoted 2 lines

Naive attempts to create the subvolumes makes them nested, perhaps because I don't have an empty unmounted top-level volume?

To do this manually, you need to mount the btrfs file system, e.g. at /mnt, without any mount options. It'll have its top-level mounted, and you'll see the subvolumes in /mnt, looking like they're directories.

Thanks for the confirmation, eventually also figured it out.

Quoted 8 lines
Quoted 4 lines

* And finally, the free space is intended for a Postgres database using most of the space on the two disks. Would a suitably tuned subvolme be a good choice, or would ext4 or xfs be more prudent (and performant?).

It really depends on the workload of the database. How big is it, how busy is it, is it compressible? Etc. You might find the performance is OK if you enable WAL, and the workload isn't significant.

The database is presently ~200GB and growing gradually, but could grow roughly 5x over the next year if anticipated external events take place. I'll see how btrfs works out. I did disable datacow and compression.

Signature · 1 line
    Viktor.