Skip to contentSkip to conversation

RAID mount fails after upgrading to kernel 6.2.0

DISCUSSION Eric Levy 3 participants 9 messages Jul 20, 2023 – Jul 22, 2023 Btrfs file system

lore
Share & tools
lore

Eric Levy

author
#0 lore raw

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
kernel: BTRFS error (device sde): failed to read the system array: -2
kernel: BTRFS error (device sde): open_ctree failed
mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error. systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

#1 lore raw
Quoted 10 lines · Eric Levy

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

Quoted 10 lines

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
kernel: BTRFS error (device sde): failed to read the system array: -2
kernel: BTRFS error (device sde): open_ctree failed
mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error. systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

Looks like the fsid is already mounted. Could you please help check?

cat /proc/self/mounts | grep btrfs

You could try a fresh scan and mount.

umount ..
btrfs device scan
mount ...

If this doesn't help. Can you share the output of:

btrfs filesystem dump-super /dev/sd[a-g] <-- basically all devices

Thanks.

author
#2 lore raw
Quoted 33 lines · Anand Jain
Quoted 10 lines · Eric Levy

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

Quoted 10 lines

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
kernel: BTRFS error (device sde): failed to read the system array: -2
kernel: BTRFS error (device sde): open_ctree failed
mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error. systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

Looks like the fsid is already mounted. Could you please help check?

cat /proc/self/mounts | grep btrfs

You could try a fresh scan and mount.

umount ..
btrfs device scan
mount ...

Quoted 6 lines

If this doesn't help. Can you share the output of:

btrfs filesystem dump-super /dev/sd[a-g] <-- basically all
devices

Thanks.

The unmount command followed by rescan does enable a successful mount, but the suggestion that the volume was mounted already had not been validated by the dump of the mount table. Based on the mount table, the volume appeared as unmounted even before the command.

Do you have any suggestions for how to resolve why the volume would be registered as having been mounted?

#3 lore raw
Quoted 38 lines · Eric Levy
Quoted 35 lines · Anand Jain
Quoted 10 lines · Eric Levy

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

Quoted 10 lines

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
kernel: BTRFS error (device sde): failed to read the system array: -2
kernel: BTRFS error (device sde): open_ctree failed
mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error. systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

Looks like the fsid is already mounted. Could you please help check?

    cat /proc/self/mounts | grep btrfs

You could try a fresh scan and mount.

    umount  ..
    btrfs device scan
    mount ...

If this doesn't help. Can you share the output of:

Quoted 8 lines
Quoted 3 lines

    btrfs filesystem dump-super /dev/sd[a-g]  <-- basically all devices

Thanks.

The unmount command followed by rescan does enable a successful mount, but the suggestion that the volume was mounted already had not been validated by the dump of the mount table. Based on the mount table, the volume appeared as unmounted even before the command.

Do you have any suggestions for how to resolve why the volume would be registered as having been mounted?

As mentioned, dump-super might help.

author
#4 lore raw
Quoted 45 lines · Anand Jain
Quoted 42 lines · Eric Levy
Quoted 39 lines · Anand Jain 
Quoted 10 lines · Eric Levy

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

Quoted 11 lines

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
kernel: BTRFS error (device sde): failed to read the system array:
-2
kernel: BTRFS error (device sde): open_ctree failed
mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error. systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

Looks like the fsid is already mounted. Could you please help check?

cat /proc/self/mounts | grep btrfs

You could try a fresh scan and mount.

umount ..
btrfs device scan
mount ...

If this doesn't help. Can you share the output of:

btrfs filesystem dump-super /dev/sd[a-g] <-- basically all
devices

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Thanks.

The unmount command followed by rescan does enable a successful mount, but the suggestion that the volume was mounted already had not been validated by the dump of the mount table. Based on the mount table, the volume appeared as unmounted even before the command.

Do you have any suggestions for how to resolve why the volume would be registered as having been mounted?

As mentioned, dump-super might help.

After further investigation, I believe the issue is not particularly related to the kernel or the filesystem.

I believe that systemd is attempting to mount a volume before all of the devices are attached through the iSCSI login process.

The issue may be outside the scope of Btrfs, but I certainly would appreciate any suggestions.

How can systemd be forced to wait, before attempting to mount, until all units in the volume, identified by an UUID, have been successfully attached?

Quoted 1 line
#5 lore raw
Quoted 56 lines · Eric Levy
Quoted 43 lines · Anand Jain
Quoted 40 lines · Eric Levy
Quoted 37 lines · Anand Jain 
Quoted 10 lines · Eric Levy

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

Quoted 10 lines

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
kernel: BTRFS error (device sde): failed to read the system array: -2
kernel: BTRFS error (device sde): open_ctree failed
mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error. systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

Looks like the fsid is already mounted. Could you please help check?

    cat /proc/self/mounts | grep btrfs

You could try a fresh scan and mount.

    umount  ..
    btrfs device scan
    mount ...

If this doesn't help. Can you share the output of:

    btrfs filesystem dump-super /dev/sd[a-g]  <-- basically all devices

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Thanks.

The unmount command followed by rescan does enable a successful mount, but the suggestion that the volume was mounted already had not been validated by the dump of the mount table. Based on the mount table, the volume appeared as unmounted even before the command.

Do you have any suggestions for how to resolve why the volume would be registered as having been mounted?

 As mentioned, dump-super might help.

Quoted 7 lines

After further investigation, I believe the issue is not particularly related to the kernel or the filesystem.

I believe that systemd is attempting to mount a volume before all of the devices are attached through the iSCSI login process.

The issue may be outside the scope of Btrfs, but I certainly would appreciate any suggestions.

How can systemd be forced to wait, before attempting to mount, until all units in the volume, identified by an UUID, have been successfully attached?

Thanks for narrowing it down. Please check if you have the btrfs
kernel commit:

50d281fc434c btrfs: scan device in non-exclusive mode

This commit addresses a bug that was caused by interference from systemd during the device scan/mount process.

author
#6 lore raw
Quoted 68 lines · Anand Jain
Quoted 57 lines · Eric Levy
Quoted 44 lines · Anand Jain 
Quoted 41 lines · Eric Levy
Quoted 38 lines · Anand Jain 
Quoted 10 lines · Eric Levy

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

Quoted 10 lines

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
kernel: BTRFS error (device sde): failed to read the system array: -2
kernel: BTRFS error (device sde): open_ctree failed
mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error. systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

Looks like the fsid is already mounted. Could you please help check?

cat /proc/self/mounts | grep btrfs

You could try a fresh scan and mount.

umount ..
btrfs device scan
mount ...

If this doesn't help. Can you share the output of:

btrfs filesystem dump-super /dev/sd[a-g] <-- basically all
devices

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Thanks.

The unmount command followed by rescan does enable a successful mount, but the suggestion that the volume was mounted already had not been validated by the dump of the mount table. Based on the mount table, the volume appeared as unmounted even before the command.

Do you have any suggestions for how to resolve why the volume would be registered as having been mounted?

As mentioned, dump-super might help.

Quoted 7 lines

After further investigation, I believe the issue is not particularly related to the kernel or the filesystem.

I believe that systemd is attempting to mount a volume before all of the devices are attached through the iSCSI login process.

The issue may be outside the scope of Btrfs, but I certainly would appreciate any suggestions.

How can systemd be forced to wait, before attempting to mount, until all units in the volume, identified by an UUID, have been successfully attached?

Quoted 7 lines

Thanks for narrowing it down. Please check if you have the btrfs
kernel commit:

50d281fc434c btrfs: scan device in non-exclusive mode

This commit addresses a bug that was caused by interference from systemd during the device scan/mount process.

I don't know how to check for a specific commit.

I am running Linux Mint, which is based on regular updates for Ubuntu LTS.

The current kernel is Ubuntu mainline 6.2.0-25-generic for x86_64.

#8 lore raw
In reply to Eric Levy

---- From: Eric Levy <[email protected]> -- Sent: 2023-07-20 - 22:10 ----

Quoted 79 lines
Quoted 75 lines · Anand Jain
Quoted 71 lines · Eric Levy
Quoted 51 lines · Anand Jain 
Quoted 30 lines · Eric Levy

I recently performed a routine update on a Linux Mint system, version 21.2 (Victoria). The update moved the kernel from 5.19.0 to 6.2.0. The system includes a non-root mount that is Btrfs with RAID, which no longer mounts. Error reporting is rather limited and opaque.

I am assuming the file system is healthy from the standpoint of the old kernel, but I may need help understanding how to make it viable for the new one.

Mounting from the command line prints the following:

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdg, missing codepage or helper program, or other error.

The following is extracted from the boot sequence recorded in the kernel ring:

kernel: BTRFS error: device /dev/sdd belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS error: device /dev/sdf belongs to fsid c6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is already mounted
kernel: BTRFS info (device sde): using crc32c (crc32c-intel) checksum algorithm
kernel: BTRFS info (device sde): turning on async discard
kernel: BTRFS info (device sde): disk space caching is enabled
kernel: BTRFS error (device sde): devid 7 uuid 2f62547b-067f-433c-bec1-b90e0c8cb75e is missing

Quoted 8 lines

kernel: BTRFS error (device sde): failed to read the system array:
-2
kernel: BTRFS error (device sde): open_ctree failed mount[969]: mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error.
systemd[1]: mnt.mount: Mount process exited, code=exited, status=32/n/a

Looks like the fsid is already mounted. Could you please help check?

cat /proc/self/mounts | grep btrfs

You could try a fresh scan and mount.

umount ..
btrfs device scan
mount ...

Quoted 6 lines

If this doesn't help. Can you share the output of:

btrfs filesystem dump-super /dev/sd[a-g] <-- basically all
devices

Thanks.

The unmount command followed by rescan does enable a successful mount, but the suggestion that the volume was mounted already had not been validated by the dump of the mount table. Based on the mount table, the volume appeared as unmounted even before the command.

Do you have any suggestions for how to resolve why the volume would be registered as having been mounted?

Quoted 9 lines
Quoted 1 line

As mentioned, dump-super might help.

After further investigation, I believe the issue is not particularly related to the kernel or the filesystem.

I believe that systemd is attempting to mount a volume before all of the devices are attached through the iSCSI login process.

Btrfs should normally fail to mount a filesystem with missing devices.

Quoted 2 lines

The issue may be outside the scope of Btrfs, but I certainly would appreciate any suggestions.

Do you happen to have the 'degraded' mount option enabled?

Quoted 9 lines

How can systemd be forced to wait, before attempting to mount, until all units in the volume, identified by an UUID, have been successfully attached?

Quoted 1 line
#7 lore raw
In reply to Eric Levy
Quoted 5 lines
Quoted 5 lines

 Thanks for narrowing it down. Please check if you have the btrfs kernel commit:

   50d281fc434c btrfs: scan device in non-exclusive mode

Quoted 8 lines
Quoted 2 lines

This commit addresses a bug that was caused by interference from systemd during the device scan/mount process.

I don't know how to check for a specific commit.

I am running Linux Mint, which is based on regular updates for Ubuntu LTS.

The current kernel is Ubuntu mainline 6.2.0-25-generic for x86_64.

Commit 50d281fc434c went into kernel v6.3.
You could try upgrading if available.