Skip to contentSkip to conversation

unable to mount btrfs after upgrading from 4.16.1 to 4.19.1

DISCUSSION Tomasz Chmielewski 2 participants 5 messages Nov 9, 2018 Btrfs file system

lore
Share & tools
lore

Tomasz Chmielewski

author
#0 lore raw

btrfs sits on md RAID-5:

/dev/md2 /data btrfs noatime,compress-force=zstd,space_cache=v2,noauto 0 0

After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the fs no longer mounts:

# mount /data
mount: wrong fs type, bad option, bad superblock on /dev/md2,
       missing codepage or helper program, or other error

In some cases useful info is found in syslog - try dmesg | tail or so.

# dmesg
[  322.877321] BTRFS info (device md2): force zstd compression, level 0
[  322.877326] BTRFS info (device md2): enabling free space tree
[  322.877329] BTRFS info (device md2): using free space tree
[  322.877330] BTRFS info (device md2): has skinny extents
[  367.832058] BTRFS error (device md2): bad tree block start, want 21120019922944 have 620757027
[  367.832116] BTRFS error (device md2): failed to read block groups: -5
[  367.891339] BTRFS error (device md2): open_ctree failed

The error (bad tree block start) in dmesg changes every time as I re-run "mount /data":

[  589.425362] BTRFS error (device md2): bad tree block start, want 22008094932992 have 3378573292323635748
[  589.425423] BTRFS error (device md2): failed to read block groups: -5
[  589.469979] BTRFS error (device md2): open_ctree failed
[  680.585908] BTRFS error (device md2): bad tree block start, want 21058406105088 have 18446616720224032488
[  680.585991] BTRFS error (device md2): failed to read block groups: -5
[  680.625419] BTRFS error (device md2): open_ctree failed

Any advice how to recover?

Tomasz Chmielewski
https://lxadm.com

#1 lore raw
Quoted 2 lines · On Sat, 10 Nov 2018 03:08:01 +0900 Tomasz Chmielewski <[email protected]> wrote:

After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the fs no longer mounts:

Did you try rebooting back to 4.16.1 to see if it still mounts there?

Signature · 2 lines
With respect,
Roman
author
#2 lore raw
Quoted 7 lines · Roman Mamedov
Quoted 2 lines · On Sat, 10 Nov 2018 03:08:01 +0900 Tomasz Chmielewski <[email protected]> wrote:

After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the fs no longer mounts:

Did you try rebooting back to 4.16.1 to see if it still mounts there?

Yes, just did.

Interestingly, it does mount when I boot back to 4.16.1 - side note - it takes some 50 (!) minutes and ~8 GB of reads (according to iostat -m) to mount... device size is 16 TB, on HDD.

Tomasz Chmielewski
https://lxadm.com

author
#3 lore raw
Quoted 7 lines · Tomasz Chmielewski
Quoted 6 lines · Roman Mamedov
Quoted 2 lines · On Sat, 10 Nov 2018 03:08:01 +0900 Tomasz Chmielewski <[email protected]> wrote:

After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the fs no longer mounts:

Quoted 7 lines
Quoted 1 line

Did you try rebooting back to 4.16.1 to see if it still mounts there?

Yes, just did.

Interestingly, it does mount when I boot back to 4.16.1 - side note - it takes some 50 (!) minutes and ~8 GB of reads (according to iostat -m) to mount... device size is 16 TB, on HDD.

Also - it did mount with 4.18.17.

Way faster, in some 2 min.

Tomasz Chmielewski
https://lxadm.com

author
#4 lore raw
Quoted 12 lines · Tomasz Chmielewski
Quoted 11 lines · Tomasz Chmielewski
Quoted 7 lines · Roman Mamedov
Quoted 2 lines · On Sat, 10 Nov 2018 03:08:01 +0900 Tomasz Chmielewski <[email protected]> wrote:

After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the fs no longer mounts:

Did you try rebooting back to 4.16.1 to see if it still mounts there?

Yes, just did.

Quoted 7 lines
Quoted 3 lines

Interestingly, it does mount when I boot back to 4.16.1 - side note - it takes some 50 (!) minutes and ~8 GB of reads (according to iostat -m) to mount... device size is 16 TB, on HDD.

Also - it did mount with 4.18.17.

Way faster, in some 2 min.

A few more clean reboot cycles with 4.18.17 and got:

[  113.677829] BTRFS error (device md2): open_ctree failed
[  113.692298] BTRFS info (device md2): force zstd compression, level 0
[  113.692302] BTRFS info (device md2): using free space tree
[  113.692304] BTRFS info (device md2): has skinny extents
[  113.897681] BTRFS error (device md2): super_total_bytes 17920974913536 mismatch with fs_devices total_rw_bytes 35841949827072
[  113.897751] BTRFS error (device md2): failed to read chunk tree: -22
[  113.935149] BTRFS error (device md2): open_ctree failed

Another "mount /data" (without rebooting) mounted it fine.

Why are btrfs mounts so irregular here?

# btrfs device stats /data
[/dev/md2].write_io_errs    0
[/dev/md2].read_io_errs     0
[/dev/md2].flush_io_errs    0
[/dev/md2].corruption_errs  0
[/dev/md2].generation_errs  0
# btrfs fi usage /data
Overall:
    Device size:                  16.30TiB
    Device allocated:             14.26TiB
    Device unallocated:            2.04TiB
    Device missing:                  0.00B
    Used:                          7.99TiB
    Free (estimated):              8.27TiB      (min: 8.27TiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:14.15TiB, Used:7.92TiB
/dev/md2 14.15TiB

Metadata,single: Size:111.00GiB, Used:74.78GiB
/dev/md2 111.00GiB

System,single: Size:32.00MiB, Used:1.81MiB
/dev/md2 32.00MiB

Unallocated:
/dev/md2 2.04TiB

Tomasz Chmielewski
https://lxadm.com