Skip to contentSkip to conversation

btrfs check --repair is clean, but mount fails

DISCUSSION Marc MERLIN 4 participants 13 messages Feb 27, 2016 – Feb 28, 2016 Btrfs file system

lore
Share & tools
lore

Marc MERLIN

author
#0 lore raw

btrfs-tools 4.4-1

gargamel:~# uname -r
4.4.2-amd64-i915-volpreempt-20160214bc2

2 drive array stopped working after a crash/reboot. Check --repair finds nothing wrong with it:

gargamel:~# btrfs check --repair /dev/mapper/raid0d1
enabling repair mode
Checking filesystem on /dev/mapper/raid0d1
UUID: 01334b81-c0db-4e80-92e4-cac4da867651
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1201345524312 bytes used err is 0
total csum bytes: 1165124220
total tree bytes: 8258322432
total fs tree bytes: 5574197248
total extent tree bytes: 1020428288
btree space waste bytes: 1902023247
file data blocks allocated: 1193398628352
 referenced 1209324777472
gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
[ 8200.511021] BTRFS info (device dm-6): disk space caching is enabled
[ 8200.533030] BTRFS: failed to read the system array on dm-6
[ 8200.582097] BTRFS: open_ctree failed
gargamel:~# btrfs check --repair /dev/mapper/raid0d2
enabling repair mode
Checking filesystem on /dev/mapper/raid0d2
UUID: 01334b81-c0db-4e80-92e4-cac4da867651
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1201345540696 bytes used err is 0
total csum bytes: 1165124220
total tree bytes: 8258338816
total fs tree bytes: 5574197248
total extent tree bytes: 1020444672
btree space waste bytes: 1902039421
file data blocks allocated: 1193398628352
 referenced 1209324777472
gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
[13373.606737] BTRFS info (device dm-6): disk space caching is enabled
[13373.628682] BTRFS: failed to read the system array on dm-6
[13373.672607] BTRFS: open_ctree failed
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
#1 lore raw
Quoted 25 lines · Marc MERLIN

btrfs-tools 4.4-1

gargamel:~# uname -r
4.4.2-amd64-i915-volpreempt-20160214bc2

2 drive array stopped working after a crash/reboot. Check --repair finds nothing wrong with it:

gargamel:~# btrfs check --repair /dev/mapper/raid0d1
enabling repair mode
Checking filesystem on /dev/mapper/raid0d1
UUID: 01334b81-c0db-4e80-92e4-cac4da867651
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1201345524312 bytes used err is 0
total csum bytes: 1165124220
total tree bytes: 8258322432
total fs tree bytes: 5574197248
total extent tree bytes: 1020428288
btree space waste bytes: 1902023247
file data blocks allocated: 1193398628352
 referenced 1209324777472
Quoted 8 lines
gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
[ 8200.511021] BTRFS info (device dm-6): disk space caching is enabled
[ 8200.533030] BTRFS: failed to read the system array on dm-6
[ 8200.582097] BTRFS: open_ctree failed

Does 'btrfs dev scan' help?

Thanks,

-liubo

Quoted 36 lines

gargamel:~# btrfs check --repair /dev/mapper/raid0d2
enabling repair mode
Checking filesystem on /dev/mapper/raid0d2
UUID: 01334b81-c0db-4e80-92e4-cac4da867651
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1201345540696 bytes used err is 0
total csum bytes: 1165124220
total tree bytes: 8258338816
total fs tree bytes: 5574197248
total extent tree bytes: 1020444672
btree space waste bytes: 1902039421
file data blocks allocated: 1193398628352
 referenced 1209324777472
gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
[13373.606737] BTRFS info (device dm-6): disk space caching is enabled
[13373.628682] BTRFS: failed to read the system array on dm-6
[13373.672607] BTRFS: open_ctree failed
Signature · 8 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#2 lore raw
Quoted 35 lines · Liu Bo
Quoted 25 lines · Marc MERLIN

btrfs-tools 4.4-1

gargamel:~# uname -r
4.4.2-amd64-i915-volpreempt-20160214bc2

2 drive array stopped working after a crash/reboot. Check --repair finds nothing wrong with it:

gargamel:~# btrfs check --repair /dev/mapper/raid0d1
enabling repair mode
Checking filesystem on /dev/mapper/raid0d1
UUID: 01334b81-c0db-4e80-92e4-cac4da867651
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1201345524312 bytes used err is 0
total csum bytes: 1165124220
total tree bytes: 8258322432
total fs tree bytes: 5574197248
total extent tree bytes: 1020428288
btree space waste bytes: 1902023247
file data blocks allocated: 1193398628352
 referenced 1209324777472
Quoted 8 lines
gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
[ 8200.511021] BTRFS info (device dm-6): disk space caching is enabled
[ 8200.533030] BTRFS: failed to read the system array on dm-6
[ 8200.582097] BTRFS: open_ctree failed
Quoted 1 line

Does 'btrfs dev scan' help?

Oh my, it does...

gargamel:~# btrfs dev scan
Scanning for Btrfs filesystems
gargamel:~# mount /var/local/space
[14477.028083] BTRFS: device label btrfs_space devid 2 transid 776784 /dev/mapper/raid0d2
[14500.262307] BTRFS info (device dm-7): disk space caching is enabled
[14504.042485] BTRFS: checking UUID tree

Err, I'm very perplexed now. I already have a scan in my boot process after device decrypts. Somehow it saw one of my 2 devices, but not the other one?

I'm looking at my boot:

[  112.063677] BTRFS: device label btrfs_space devid 1 transid 776782 /dev/mapper/raid0d1
[  112.090192] BTRFS info (device dm-6): disk space caching is enabled
[  112.111740] BTRFS: failed to read the system array on dm-6
[  112.160047] BTRFS: open_ctree failed
[  112.269710] BTRFS info (device dm-6): disk space caching is enabled
[  112.291430] BTRFS: failed to read the system array on dm-6
[  112.320104] BTRFS: open_ctree failed

So dm-6 is: raid0d1 -> ../dm-6

So, raid0d1 had an issue, btrfs check didn't really report any, for some unknown reason this caused raid0d2 not to be scanned, and in turn this caused mounting that filesystem to fail?

Or did btrfs check actually fix something that I missed? Any why would scan have missed raid0d2 the first time around?

Is it complaining that it can't read btrfs structures from raid0d1 because raid0d2 wasn't known yet? I'm not too sure what open_ctree means in that context.

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
#3 lore raw
Quoted 59 lines · Marc MERLIN
Quoted 35 lines · Liu Bo
Quoted 25 lines · Marc MERLIN

btrfs-tools 4.4-1

gargamel:~# uname -r
4.4.2-amd64-i915-volpreempt-20160214bc2

2 drive array stopped working after a crash/reboot. Check --repair finds nothing wrong with it:

gargamel:~# btrfs check --repair /dev/mapper/raid0d1
enabling repair mode
Checking filesystem on /dev/mapper/raid0d1
UUID: 01334b81-c0db-4e80-92e4-cac4da867651
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1201345524312 bytes used err is 0
total csum bytes: 1165124220
total tree bytes: 8258322432
total fs tree bytes: 5574197248
total extent tree bytes: 1020428288
btree space waste bytes: 1902023247
file data blocks allocated: 1193398628352
 referenced 1209324777472
Quoted 8 lines
gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
[ 8200.511021] BTRFS info (device dm-6): disk space caching is enabled
[ 8200.533030] BTRFS: failed to read the system array on dm-6
[ 8200.582097] BTRFS: open_ctree failed
Quoted 1 line

Does 'btrfs dev scan' help?

Oh my, it does...

gargamel:~# btrfs dev scan
Scanning for Btrfs filesystems
gargamel:~# mount /var/local/space
[14477.028083] BTRFS: device label btrfs_space devid 2 transid 776784 /dev/mapper/raid0d2
[14500.262307] BTRFS info (device dm-7): disk space caching is enabled
[14504.042485] BTRFS: checking UUID tree

Err, I'm very perplexed now. I already have a scan in my boot process after device decrypts. Somehow it saw one of my 2 devices, but not the other one?

I'm looking at my boot:

[  112.063677] BTRFS: device label btrfs_space devid 1 transid 776782 /dev/mapper/raid0d1
[  112.090192] BTRFS info (device dm-6): disk space caching is enabled
[  112.111740] BTRFS: failed to read the system array on dm-6
[  112.160047] BTRFS: open_ctree failed
[  112.269710] BTRFS info (device dm-6): disk space caching is enabled
[  112.291430] BTRFS: failed to read the system array on dm-6
[  112.320104] BTRFS: open_ctree failed
Quoted 8 lines

So dm-6 is: raid0d1 -> ../dm-6

So, raid0d1 had an issue, btrfs check didn't really report any, for some unknown reason this caused raid0d2 not to be scanned, and in turn this caused mounting that filesystem to fail?

Or did btrfs check actually fix something that I missed? Any why would scan have missed raid0d2 the first time around?

I feel confused, too.

Can you grep this message since btrfs dev scan has a "printf("Scanning for Btrfs filesystems\n");"? And if a scan failed somehow, it may print an error after this message
and we then know what was happening..

Quoted 3 lines

Is it complaining that it can't read btrfs structures from raid0d1 because raid0d2 wasn't known yet? I'm not too sure what open_ctree means in that context.

No idea about this, not sure if it's due to any dependency?

Thanks,

-liubo

author
#4 lore raw
Quoted 3 lines · Liu Bo

Can you grep this message since btrfs dev scan has a "printf("Scanning for Btrfs filesystems\n");"? And if a scan failed somehow, it may print an error after this message
and we then know what was happening..

I'm not seeing anything in my console boot log outside of this posted in my previous message:

Fri Feb 26 14:55:17 2016: raid0d1 (started)...raid0d2 (starting)... Fri Feb 26 14:55:18 2016: raid0d2 (started)...done.

Quoted 10 lines
Quoted 10 lines

I'm looking at my boot:

[  112.063677] BTRFS: device label btrfs_space devid 1 transid 776782 /dev/mapper/raid0d1
[  112.090192] BTRFS info (device dm-6): disk space caching is enabled
[  112.111740] BTRFS: failed to read the system array on dm-6
[  112.160047] BTRFS: open_ctree failed
[  112.269710] BTRFS info (device dm-6): disk space caching is enabled
[  112.291430] BTRFS: failed to read the system array on dm-6
[  112.320104] BTRFS: open_ctree failed

So dm-6 is: raid0d1 -> ../dm-6

btrfs dev scan didn't output any messages from what I can tell, outside of the dmesg stuff here.

The total messages around that time:

[  110.182362] bcache: bch_journal_replay() journal replay done, 92 keys in 6 entries, seq 8855
[  110.208992] bcache: register_cache() registered cache device dm-4
[  110.236345] bcache: register_bdev() registered backing device dm-1
[  110.316129] bcache: bch_cached_dev_attach() Caching dm-1 as bcache1 on set 0226553a-37cf-41d5-b3ce-8b1e944543a8
[  110.348350] bcache: register_bcache() error opening /dev/md5: device already registered
[  110.445200] Adding 15616764k swap on /dev/mapper/eswap1.  Priority:-1 extents:1 across:15616764k FS
[  111.585812] EXT4-fs (md11): mounted filesystem with ordered data mode. Opts: (null)
[  112.063677] BTRFS: device label btrfs_space devid 1 transid 776782 /dev/mapper/raid0d1
[  112.090192] BTRFS info (device dm-6): disk space caching is enabled
[  112.111740] BTRFS: failed to read the system array on dm-6
[  112.160047] BTRFS: open_ctree failed
[  112.269710] BTRFS info (device dm-6): disk space caching is enabled
[  112.291430] BTRFS: failed to read the system array on dm-6
[  112.320104] BTRFS: open_ctree failed
[  112.332567] BTRFS: device label dshelf1 devid 1 transid 25903 /dev/bcache1
[  112.333076] BTRFS info (device bcache1): disk space caching is enabled
[  112.333077] BTRFS: has skinny extents
[  115.185582] BTRFS: detected SSD devices, enabling SSD mode
[  115.216381] BTRFS: device label dshelf2 devid 1 transid 444834 /dev/mapper/dshelf2
[  115.241607] BTRFS info (device dm-2): disk space caching is enabled
[  115.591923] BTRFS info (device dm-2): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 65, gen 0

Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#5 lore raw
In reply to Marc MERLIN
Quoted 43 lines · Marc MERLIN
Quoted 35 lines · Liu Bo
Quoted 25 lines · Marc MERLIN

btrfs-tools 4.4-1

gargamel:~# uname -r
4.4.2-amd64-i915-volpreempt-20160214bc2

2 drive array stopped working after a crash/reboot. Check --repair finds nothing wrong with it:

gargamel:~# btrfs check --repair /dev/mapper/raid0d1
enabling repair mode
Checking filesystem on /dev/mapper/raid0d1
UUID: 01334b81-c0db-4e80-92e4-cac4da867651
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1201345524312 bytes used err is 0
total csum bytes: 1165124220
total tree bytes: 8258322432
total fs tree bytes: 5574197248
total extent tree bytes: 1020428288
btree space waste bytes: 1902023247
file data blocks allocated: 1193398628352
  referenced 1209324777472
Quoted 8 lines
gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
        missing codepage or helper program, or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so
[ 8200.511021] BTRFS info (device dm-6): disk space caching is enabled
[ 8200.533030] BTRFS: failed to read the system array on dm-6
[ 8200.582097] BTRFS: open_ctree failed
Quoted 1 line

Does 'btrfs dev scan' help?

Oh my, it does...

gargamel:~# btrfs dev scan
Scanning for Btrfs filesystems
gargamel:~# mount /var/local/space
Quoted 5 lines
[14477.028083] BTRFS: device label btrfs_space devid 2 transid 776784 /dev/mapper/raid0d2
[14500.262307] BTRFS info (device dm-7): disk space caching is enabled
[14504.042485] BTRFS: checking UUID tree

Err, I'm very perplexed now. I already have a scan in my boot process after device decrypts.

Can you log the output of blkid here. If blkid output does not report all the btrfs devs correctly the kernel won't know as well.

Thanks, Anand

Quoted 26 lines

Somehow it saw one of my 2 devices, but not the other one?

I'm looking at my boot:

[  112.063677] BTRFS: device label btrfs_space devid 1 transid 776782 /dev/mapper/raid0d1
[  112.090192] BTRFS info (device dm-6): disk space caching is enabled
[  112.111740] BTRFS: failed to read the system array on dm-6
[  112.160047] BTRFS: open_ctree failed
[  112.269710] BTRFS info (device dm-6): disk space caching is enabled
[  112.291430] BTRFS: failed to read the system array on dm-6
[  112.320104] BTRFS: open_ctree failed

So dm-6 is: raid0d1 -> ../dm-6

So, raid0d1 had an issue, btrfs check didn't really report any, for some unknown reason this caused raid0d2 not to be scanned, and in turn this caused mounting that filesystem to fail?

Or did btrfs check actually fix something that I missed? Any why would scan have missed raid0d2 the first time around?

Is it complaining that it can't read btrfs structures from raid0d1 because raid0d2 wasn't known yet? I'm not too sure what open_ctree means in that context.

Thanks,
Marc

author
#6 lore raw
Quoted 3 lines · Anand Jain

Can you log the output of blkid here. If blkid output does not report all the btrfs devs correctly the kernel won't know as well.

Sure. The relevant entries are at the bottom (last 3 lines).
It looks good now of course, but I how it would have been before I ran a btrfs dev scan

Is there anything helpful in there?

/dev/sdc2: UUID="d9a11ed8-536b-e2d7-2b63-6e45318e24d6" UUID_SUB="c8b717b3-d1bb-9211-f5fd-88b4cad3891a" LABEL="gargamel.svh.merlins.org:12" TYPE="linux_raid_member" PARTUUID="fc061e12-02" /dev/sdc3: UUID="d98cdbd0-4879-0342-1cb9-3a199281d756" UUID_SUB="c33a26df-abca-4bb2-8064-741d05334129" LABEL="gargamel.svh.merlins.org:13" TYPE="linux_raid_member" PARTUUID="fc061e12-03" /dev/sdc4: UUID="1a134ab6-0e02-43fa-8704-deed85998155" TYPE="crypto_LUKS" PARTUUID="fc061e12-04"
/dev/sde1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="d10f45f6-52e0-d793-f3e3-b687251a83f5" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="5bd72031-01" /dev/sdf1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="aee9d2d4-bb18-579a-28f4-f6b15f89a40a" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="23f4b9f6-01" /dev/sdd1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="db2df41e-1daf-02af-89ed-6e77813e29da" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="90f15dcd-01" /dev/sda1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="00d115df-f27d-9293-6048-7af099602670" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="06a7af4a-01" /dev/sdg1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="ca4598ba-de58-5baa-b993-5222e06ac97d" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="bd9db056-9d37-4191-9bd1-353b6a47b487" /dev/sdh1: UUID="284ccdd7-5516-4ff2-b798-3bc1829c9a04" TYPE="crypto_LUKS" PARTUUID="ca3bb5a5-01"
/dev/sdh2: UUID="7be7e175-8f4c-4f99-94b2-9c904d227045" TYPE="bcache" PARTUUID="ca3bb5a5-02"
/dev/sdj1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="684b477c-81a8-6ce9-222b-1d385aab485c" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="6834173c-d6a9-4da1-8de0-ddf2411557e3" /dev/sdk1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="7d311e9f-6747-24a5-c962-9fb63739a6e8" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="16d76002-b9f1-4016-8119-a200ec16d4a2" /dev/sdl1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="cdef3e24-371c-c5d4-a289-16dd0939eff6" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="8b79953b-9f6b-4649-a526-2c61fe07a6f1" /dev/sdm1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="075571ff-4115-17e9-027f-8c2fcef0457a" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="dd673133-e1a9-46dc-a42b-1554a44a9b28" /dev/sdi1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="02eae0f6-af27-0529-1d77-09483114b540" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="03d0f2a3-45a4-4172-81e2-1afad7e06d10" /dev/sdo1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="e8bf196a-9a58-f8de-9472-6f16ab4599d9" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="97573ae8-878b-41bc-b838-e0ea97a54a1b" /dev/sdn1: UUID="a9bdc59b-51e9-c58e-58d7-48c43de9f367" TYPE="linux_raid_member" PARTUUID="911f14a9-01"
/dev/sdn2: UUID="d9a11ed8-536b-e2d7-2b63-6e45318e24d6" UUID_SUB="d2d23610-d966-dfcb-92c5-b49a3a161964" LABEL="gargamel.svh.merlins.org:12" TYPE="linux_raid_member" PARTUUID="911f14a9-02" /dev/sdn3: UUID="d98cdbd0-4879-0342-1cb9-3a199281d756" UUID_SUB="9c50a539-3923-840a-e720-66b23d823cc2" LABEL="gargamel.svh.merlins.org:13" TYPE="linux_raid_member" PARTUUID="911f14a9-03" /dev/sdn4: UUID="01ed40a3-a9f2-4d7d-af14-d57035a5ed55" TYPE="crypto_LUKS" PARTUUID="911f14a9-04"
/dev/sdp1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="4910400c-8e3e-0d32-fdb9-16f8f7ca5f87" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="1bd91aa4-7f0e-4574-9c0e-630c2de9295f" /dev/sdq1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="497a9a28-4059-842b-25e2-ab31c6d48ded" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="84740d8d-7689-499f-9171-7d0acf8154b7" /dev/sdr1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="1e3ab905-ba67-7efe-2aaf-cf0d99276e26" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="47aa6022-6a58-4856-87d8-5c4f54212b37" /dev/md11: UUID="6e06a918-80dc-4f00-8cd3-234939d7172d" TYPE="ext4" PTTYPE="dos"
/dev/md13: UUID="e90613ef-e719-4396-97d2-93ca62c17736" TYPE="crypto_LUKS"
/dev/md5: UUID="2b8cf5be-d74f-43fa-aebd-fed79ba5cb50" TYPE="bcache"
/dev/md6: UUID="ae32d57e-12c3-457a-bc17-4166aa6dde75" TYPE="crypto_LUKS"
/dev/md8: UUID="33b49001-7253-40b3-b617-ea91850a69d9" TYPE="crypto_LUKS"
/dev/bcache0: UUID="c02da1dc-f62b-4232-a0ed-264dae65f16a" TYPE="crypto_LUKS"
/dev/mapper/dshelf1b: UUID="47cf2391-1942-46d1-9690-d597e860a5c6" TYPE="bcache"
/dev/mapper/dshelf2: LABEL="dshelf2" UUID="d4a51178-c1e6-4219-95ab-5c5864695bfd" UUID_SUB="4f46aa7b-23a5-4ad8-8b74-d735d7caf156" TYPE="btrfs"
/dev/mapper/eswap1: UUID="b061e3e6-e775-4cdf-ab2b-1a27ac985888" TYPE="swap"
/dev/mapper/cache1: UUID="16dbdda5-46da-43ef-ac6a-6b81eaed3bb2" TYPE="bcache"
/dev/mapper/oldds1: UUID="79974649-f771-498c-8477-10d076ec3627" UUID_SUB="462789c1-b339-44ba-a53a-a96680e94ac8" TYPE="btrfs"
/dev/mapper/raid0d1: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="a87c0954-e0d5-4cdb-acb9-05ed290c16df" TYPE="btrfs"
/dev/mapper/raid0d2: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="b4ea90fa-4262-489a-830d-3bd4e7518ff2" TYPE="btrfs"
/dev/bcache1: LABEL="dshelf1" UUID="5d0847f8-b243-494a-9e31-0a7c9adbd764" UUID_SUB="f65c681d-f1a1-45f1-a2b5-290358fd9b39" TYPE="btrfs"

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#7 lore raw

Marc,

Quoted 3 lines

Err, I'm very perplexed now. I already have a scan in my boot process after device decrypts.
Somehow it saw one of my 2 devices, but not the other one?

If blkid shows both the devices and if you are running 'btrfs dev scan' during boot, then yes kernel should see both the devices.

Just to be sure, Can you reconfirm if the below blkid output is taken at the time of the bootup where you are running btrfs dev scan in the script ?

If you have a choice pls add, 'btrfs fi show -d' as well for the outputs to be taken at the time of boot just before system's 'btrfs dev scan',

Pls do share your /etc/fstab output for /var/local/space (I guess you don't have -o degrade option) which is good for this debugging, which means mount fails if any one of the device is missing.

Thanks, Anand

Quoted 48 lines · Marc MERLIN
Quoted 3 lines · Anand Jain

Can you log the output of blkid here. If blkid output does not report all the btrfs devs correctly the kernel won't know as well.

Sure. The relevant entries are at the bottom (last 3 lines).
It looks good now of course, but I how it would have been before I ran a btrfs dev scan

Is there anything helpful in there?

/dev/sdc2: UUID="d9a11ed8-536b-e2d7-2b63-6e45318e24d6" UUID_SUB="c8b717b3-d1bb-9211-f5fd-88b4cad3891a" LABEL="gargamel.svh.merlins.org:12" TYPE="linux_raid_member" PARTUUID="fc061e12-02" /dev/sdc3: UUID="d98cdbd0-4879-0342-1cb9-3a199281d756" UUID_SUB="c33a26df-abca-4bb2-8064-741d05334129" LABEL="gargamel.svh.merlins.org:13" TYPE="linux_raid_member" PARTUUID="fc061e12-03" /dev/sdc4: UUID="1a134ab6-0e02-43fa-8704-deed85998155" TYPE="crypto_LUKS" PARTUUID="fc061e12-04"
/dev/sde1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="d10f45f6-52e0-d793-f3e3-b687251a83f5" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="5bd72031-01" /dev/sdf1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="aee9d2d4-bb18-579a-28f4-f6b15f89a40a" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="23f4b9f6-01" /dev/sdd1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="db2df41e-1daf-02af-89ed-6e77813e29da" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="90f15dcd-01" /dev/sda1: UUID="0d2b91aa-3c93-54d1-5bca-28c1f96f9def" UUID_SUB="00d115df-f27d-9293-6048-7af099602670" LABEL="gargamel.svh.merlins.org:8" TYPE="linux_raid_member" PARTUUID="06a7af4a-01" /dev/sdg1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="ca4598ba-de58-5baa-b993-5222e06ac97d" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="bd9db056-9d37-4191-9bd1-353b6a47b487" /dev/sdh1: UUID="284ccdd7-5516-4ff2-b798-3bc1829c9a04" TYPE="crypto_LUKS" PARTUUID="ca3bb5a5-01"
/dev/sdh2: UUID="7be7e175-8f4c-4f99-94b2-9c904d227045" TYPE="bcache" PARTUUID="ca3bb5a5-02"
/dev/sdj1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="684b477c-81a8-6ce9-222b-1d385aab485c" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="6834173c-d6a9-4da1-8de0-ddf2411557e3" /dev/sdk1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="7d311e9f-6747-24a5-c962-9fb63739a6e8" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="16d76002-b9f1-4016-8119-a200ec16d4a2" /dev/sdl1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="cdef3e24-371c-c5d4-a289-16dd0939eff6" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="8b79953b-9f6b-4649-a526-2c61fe07a6f1" /dev/sdm1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="075571ff-4115-17e9-027f-8c2fcef0457a" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="dd673133-e1a9-46dc-a42b-1554a44a9b28" /dev/sdi1: UUID="ec672af7-a66d-9557-2f00-d76c38c9f705" UUID_SUB="02eae0f6-af27-0529-1d77-09483114b540" LABEL="gargamel.svh.merlins.org:5" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="03d0f2a3-45a4-4172-81e2-1afad7e06d10" /dev/sdo1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="e8bf196a-9a58-f8de-9472-6f16ab4599d9" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="97573ae8-878b-41bc-b838-e0ea97a54a1b" /dev/sdn1: UUID="a9bdc59b-51e9-c58e-58d7-48c43de9f367" TYPE="linux_raid_member" PARTUUID="911f14a9-01"
/dev/sdn2: UUID="d9a11ed8-536b-e2d7-2b63-6e45318e24d6" UUID_SUB="d2d23610-d966-dfcb-92c5-b49a3a161964" LABEL="gargamel.svh.merlins.org:12" TYPE="linux_raid_member" PARTUUID="911f14a9-02" /dev/sdn3: UUID="d98cdbd0-4879-0342-1cb9-3a199281d756" UUID_SUB="9c50a539-3923-840a-e720-66b23d823cc2" LABEL="gargamel.svh.merlins.org:13" TYPE="linux_raid_member" PARTUUID="911f14a9-03" /dev/sdn4: UUID="01ed40a3-a9f2-4d7d-af14-d57035a5ed55" TYPE="crypto_LUKS" PARTUUID="911f14a9-04"
/dev/sdp1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="4910400c-8e3e-0d32-fdb9-16f8f7ca5f87" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="1bd91aa4-7f0e-4574-9c0e-630c2de9295f" /dev/sdq1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="497a9a28-4059-842b-25e2-ab31c6d48ded" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="84740d8d-7689-499f-9171-7d0acf8154b7" /dev/sdr1: UUID="66bccdfb-afbf-9683-fcf1-f12ef2af2dcb" UUID_SUB="1e3ab905-ba67-7efe-2aaf-cf0d99276e26" LABEL="gargamel.svh.merlins.org:6" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="47aa6022-6a58-4856-87d8-5c4f54212b37" /dev/md11: UUID="6e06a918-80dc-4f00-8cd3-234939d7172d" TYPE="ext4" PTTYPE="dos"
/dev/md13: UUID="e90613ef-e719-4396-97d2-93ca62c17736" TYPE="crypto_LUKS"
/dev/md5: UUID="2b8cf5be-d74f-43fa-aebd-fed79ba5cb50" TYPE="bcache"
/dev/md6: UUID="ae32d57e-12c3-457a-bc17-4166aa6dde75" TYPE="crypto_LUKS"
/dev/md8: UUID="33b49001-7253-40b3-b617-ea91850a69d9" TYPE="crypto_LUKS"
/dev/bcache0: UUID="c02da1dc-f62b-4232-a0ed-264dae65f16a" TYPE="crypto_LUKS"
/dev/mapper/dshelf1b: UUID="47cf2391-1942-46d1-9690-d597e860a5c6" TYPE="bcache"
/dev/mapper/dshelf2: LABEL="dshelf2" UUID="d4a51178-c1e6-4219-95ab-5c5864695bfd" UUID_SUB="4f46aa7b-23a5-4ad8-8b74-d735d7caf156" TYPE="btrfs"
/dev/mapper/eswap1: UUID="b061e3e6-e775-4cdf-ab2b-1a27ac985888" TYPE="swap"
/dev/mapper/cache1: UUID="16dbdda5-46da-43ef-ac6a-6b81eaed3bb2" TYPE="bcache"
/dev/mapper/oldds1: UUID="79974649-f771-498c-8477-10d076ec3627" UUID_SUB="462789c1-b339-44ba-a53a-a96680e94ac8" TYPE="btrfs"
/dev/mapper/raid0d1: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="a87c0954-e0d5-4cdb-acb9-05ed290c16df" TYPE="btrfs"
/dev/mapper/raid0d2: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="b4ea90fa-4262-489a-830d-3bd4e7518ff2" TYPE="btrfs"
/dev/bcache1: LABEL="dshelf1" UUID="5d0847f8-b243-494a-9e31-0a7c9adbd764" UUID_SUB="f65c681d-f1a1-45f1-a2b5-290358fd9b39" TYPE="btrfs"

author
#8 lore raw
Quoted 9 lines · Anand Jain

Marc,

Quoted 3 lines

Err, I'm very perplexed now. I already have a scan in my boot process after device decrypts.
Somehow it saw one of my 2 devices, but not the other one?

If blkid shows both the devices and if you are running 'btrfs dev scan' during boot, then yes kernel should see both the devices.

Agreed. It has every single time except that one time.

Quoted 3 lines

Just to be sure, Can you reconfirm if the below blkid output is taken at the time of the bootup where you are running btrfs dev scan in the script ?

It's taken after the problem was fixed with a 2nd btrfs scan. It's a server doing a 3 day long copy. Can't reboot it right now :)

Quoted 2 lines

If you have a choice pls add, 'btrfs fi show -d' as well for the outputs to be taken at the time of boot just before system's 'btrfs dev scan',

I'll do that next time, thanks.

Quoted 3 lines

Pls do share your /etc/fstab output for /var/local/space (I guess you don't have -o degrade option) which is good for this debugging, which means mount fails if any one of the device is missing.

Nothing fancy:
LABEL=btrfs_space /var/local/space btrfs subvol=varlocalspace,defaults,compress=lzo,skip_balance,noatime,noexec 0 0

I'll do more debugging if it happens agian, but I'm pretty sure it was a one time thing.
Next boot will tell :)

Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#9 lore raw

Marc MERLIN posted on Sat, 27 Feb 2016 19:09:30 -0800 as excerpted:

[line in fstab]

Quoted 1 line

LABEL=btrfs_space /var/local/space btrfs subvol=varlocalspace,defaults,compress=lzo,skip_balance,noatime,noexec 0 0

Nothing to do with your issue at hand, but some potentially useful info, FWIW...

The "defaults" mount option in fstab is just that, the defaults, and doesn't really apply as an option because they /are/ the defaults. The only reason for "defaults" as an explicit option to exist is to hold the options field in the case of no non-default option being there to hold it, so once there's _any_ non-default option added, "defaults" can be removed, as the non-default option is now holding the field and the defaults apply anyway unless overruled by some other option.

Additionally, with any reasonably modern util-linux, the last two fields, dump and fsck-passno, both default to 0 if not present, so don't need to be explicitly included if zero, unless of course you want to as a matter of style, possibly for consistency with other fstab entries where the fields are needed, perhaps because fsck-passno isn't zero and you need the dump field of zero as a field-holder to have the non-zero fsck-passno in the correct field.

Of course neither the two trailing zero fields nor the defaults mount option do any harm, and you may simply be including them as a matter of style. But because the options field in particular can be rather long, as yours certainly is, knowing that you can skip "defaults" as a mount option can be quite a useful hint, if you weren't aware of it before.

So FWIW, YMMV, etc, but I thought it might be worth mentioning, for other readers for whom it might be new and useful info, even if it's "the way you like it, ****it!" for you. =:^)

FWIW the fstab (5) manpage is the official documentation for it, but it says effectively what I said above. =:^)

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
author
#10 lore raw
Quoted 2 lines · Duncan

The "defaults" mount option in fstab is just that, the defaults, and doesn't really apply as an option because they /are/ the defaults. The

Yeah, been having this for some 20 years, cut and paste, and I should get used to just removing it :)

Quoted 3 lines

Additionally, with any reasonably modern util-linux, the last two fields, dump and fsck-passno, both default to 0 if not present, so don't need to be explicitly included if zero, unless of course you want to as a matter of

Correct. That one is mostly style/cut&paste and an fstab that was created in 1999 or so :)

Cheers,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
author
#11 lore raw
In reply to Anand Jain
Quoted 2 lines · Anand Jain

If you have a choice pls add, 'btrfs fi show -d' as well for the outputs to be taken at the time of boot just before system's 'btrfs dev scan',

Ok, I had to reboot anyway, so here's the state when it's bad:

gargamel:~# btrfs fi show -d
Label: 'btrfs_boot'  uuid: e4c1daa8-9c39-4a59-b0a9-86297d397f3b
        Total devices 1 FS bytes used 43.52GiB
        devid    1 size 79.93GiB used 62.13GiB path /dev/mapper/cryptroot

Label: 'dshelf2' uuid: d4a51178-c1e6-4219-95ab-5c5864695bfd
Total devices 1 FS bytes used 4.27TiB
devid 1 size 7.28TiB used 4.44TiB path /dev/mapper/dshelf2

Label: 'btrfs_space' uuid: 01334b81-c0db-4e80-92e4-cac4da867651

        Total devices 2 FS bytes used 1.09TiB
        devid    1 size 836.13GiB used 641.03GiB path /dev/mapper/raid0d1
        devid    2 size 836.13GiB used 641.03GiB path /dev/mapper/raid0d2

Label: 'dshelf1' uuid: 5d0847f8-b243-494a-9e31-0a7c9adbd764
Total devices 1 FS bytes used 8.88TiB
devid 1 size 21.83TiB used 8.92TiB path /dev/bcache1

gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
[ 3209.752862] BTRFS info (device dm-6): disk space caching is enabled
[ 3209.775418] BTRFS: failed to read the system array on dm-6
[ 3209.807673] BTRFS: open_ctree failed

blkid shows:
/dev/mapper/raid0d1: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="a87c0954-e0d5-4cdb-acb9-05ed290c16df" TYPE="btrfs" /dev/mapper/raid0d2: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="b4ea90fa-4262-489a-830d-3bd4e7518ff2" TYPE="btrfs"

but mount fails.

And then:

gargamel:~# btrfs dev scan
Scanning for Btrfs filesystems
gargamel:~# mount /var/local/space
gargamel:~#

Does this make any sense to you?

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#12 lore raw
Quoted 39 lines · Marc MERLIN
Quoted 2 lines · Anand Jain

If you have a choice pls add, 'btrfs fi show -d' as well for the outputs to be taken at the time of boot just before system's 'btrfs dev scan',

Ok, I had to reboot anyway, so here's the state when it's bad:

gargamel:~# btrfs fi show -d
Label: 'btrfs_boot'  uuid: e4c1daa8-9c39-4a59-b0a9-86297d397f3b
         Total devices 1 FS bytes used 43.52GiB
         devid    1 size 79.93GiB used 62.13GiB path /dev/mapper/cryptroot

Label: 'dshelf2' uuid: d4a51178-c1e6-4219-95ab-5c5864695bfd
Total devices 1 FS bytes used 4.27TiB
devid 1 size 7.28TiB used 4.44TiB path /dev/mapper/dshelf2

Label: 'btrfs_space' uuid: 01334b81-c0db-4e80-92e4-cac4da867651

         Total devices 2 FS bytes used 1.09TiB
         devid    1 size 836.13GiB used 641.03GiB path /dev/mapper/raid0d1
         devid    2 size 836.13GiB used 641.03GiB path /dev/mapper/raid0d2

Label: 'dshelf1' uuid: 5d0847f8-b243-494a-9e31-0a7c9adbd764
Total devices 1 FS bytes used 8.88TiB
devid 1 size 21.83TiB used 8.92TiB path /dev/bcache1

gargamel:~# mount /var/local/space
mount: wrong fs type, bad option, bad superblock on /dev/mapper/raid0d1,
        missing codepage or helper program, or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so
[ 3209.752862] BTRFS info (device dm-6): disk space caching is enabled
[ 3209.775418] BTRFS: failed to read the system array on dm-6
[ 3209.807673] BTRFS: open_ctree failed

blkid shows:
/dev/mapper/raid0d1: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="a87c0954-e0d5-4cdb-acb9-05ed290c16df" TYPE="btrfs" /dev/mapper/raid0d2: LABEL="btrfs_space" UUID="01334b81-c0db-4e80-92e4-cac4da867651" UUID_SUB="b4ea90fa-4262-489a-830d-3bd4e7518ff2" TYPE="btrfs"

but mount fails.

Quoted 7 lines

And then:

gargamel:~# btrfs dev scan
Scanning for Btrfs filesystems
gargamel:~# mount /var/local/space
gargamel:~#

Does this make any sense to you?

Not really. Here we are trying to understand why the first 'btrfs dev scan' doesn't help. And, as I understand the first 'btrfs dev scan' is during boot up by some system'd script. But the above output seems to be taken from the terminal which is after the system has booted up. That does not help. We need those commands output during the bootup at the time and just before when systemd issues 'btrfs dev scan'. Hope I sound clearer now, sorry if it wasn't before.

Thanks, Anand

Quoted 3 lines

Thanks,
Marc