Skip to contentSkip to conversation

bad btree header at bucket xxx.

DISCUSSION Thierry 3 participants 6 messages Nov 3, 2013 – Nov 10, 2013 4 messages not in the archive bcache (block layer cache)

lore
Share & tools
lore

Thierry

author
#0 lore raw

Hi,

After a reboot, the bcache devices didn't appeared. Looking at dmesg I foud out the following entries:

[ 12.182288] bcache: register_bdev() registered backing device dm-0
[ 12.217379] bcache: register_bdev() registered backing device dm-1
[ 12.253040] bcache: error on d8e791cc-47c2-449c-909a-1e4433a5de11: bad
btree header at bucket 32018, block 0, 0 keys, disabling caching
[ 12.253050] bcache: register_cache() registered cache device sdb1
[ 12.253488] bcache: cache_set_free() Cache set

d8e791cc-47c2-449c-909a-1e4433a5de11 unregistered

Here is my device map
NAME MAJ:MIN RM SIZE TYPE FSTYPE MOUNTPOINT UUID PARTUUID
sda 8:0 0 931.5G disk
├─sda1 8:1 0 150M part ext2 0ced49ec-a9ec-4711-95b6-8ce01153ee3c 2a20f8be-8162-48db-9e51-d97e8fde21ae
├─sda2 8:2 0 49.9G part ext4 / 79f36847-8365-46da-85a1-cbab5ea9e30a 59d37a51-5f05-4356-857c-1937b793e9b1
├─sda3 8:3 0 881.5G part LVM2_member Vzf2b6-NmgQ-5Zff-QKRS-puR8-Dlhz-XIkY3t e337497b-b1e3-479e-ba29-e1b203434e97 │ ├─vgDataNew-lvHome (dm-0) 254:0 0 50G lvm bcache ff11059e-0edf-428d-9657-37f255bac07e
│ │ └─bcache0 252:0 0 50G disk ext4 /home 66bf41ef-174a-4770-b9ef-d37ab06d3f23
│ ├─vgDataNew-lvVM (dm-1) 254:1 0 100G lvm bcache 9497600e-839f-4f59-862a-a7512cb59f0a
│ │ └─bcache1 252:1 0 100G disk ext4 /data/vm 5ed8126a-2bda-465e-bedb-61272d878fde

│ ├─vgDataNew-lvDownloads (dm-2) 254:2 0 50G lvm /home/xxx/xxx
│ └─vgDataNew-lvSwap (dm-3) 254:3 0 12G lvm [SWAP]
└─sda4 8:4 0 1007K part f6826e28-5dc0-4122-ba84-8ccaffd900fd

sdb 8:16 0 223.6G disk
└─sdb1 8:17 0 223.6G part bcache dff8e23e-7f9a-4a14-94a0-ec5b618a35c4 ba9624c2-7fb7-4cfb-8d8a-1da6ca429820
sr0

Here is the output of bacache-super-show

bcache-super-show /dev/sdb1
sb.magic ok
sb.first_sector 8 [match]
sb.csum C161466F71949DD0 [match]
sb.version 3 [cache device]

dev.uuid dff8e23e-7f9a-4a14-94a0-ec5b618a35c4
dev.sectors_per_block 8
dev.sectors_per_bucket 1024
dev.cache.first_sector 1024
dev.cache.cache_sectors 468858880
dev.cache.total_sectors 468859904
dev.cache.discard yes
dev.cache.pos 0

cset.uuid d8e791cc-47c2-449c-909a-1e4433a5de11

bcache-super-show /dev/vgDataNew/lvHome
sb.magic ok
sb.first_sector 8 [match]
sb.csum ADBDC19FD7011CAB [match]
sb.version 1 [backing device]

dev.uuid ff11059e-0edf-428d-9657-37f255bac07e
dev.sectors_per_block 8
dev.sectors_per_bucket 1024
dev.data.first_sector 16
dev.data.cache_mode 3 [no caching]
dev.data.cache_state 3 [inconsistent]

cset.uuid d8e791cc-47c2-449c-909a-1e4433a5de11

bcache-super-show /dev/vgDataNew/lvVM
sb.magic ok
sb.first_sector 8 [match]
sb.csum BFD4613F155DD0D7 [match]
sb.version 1 [backing device]

dev.uuid 9497600e-839f-4f59-862a-a7512cb59f0a
dev.sectors_per_block 8
dev.sectors_per_bucket 1024
dev.data.first_sector 16
dev.data.cache_mode 3 [no caching]
dev.data.cache_state 3 [inconsistent]

cset.uuid d8e791cc-47c2-449c-909a-1e4433a5de11

As you see I restarted the devices after having changed the cache mode to "none" followed by a echo 1 > running (The state of the 2 devices were dirty)
e2fsck do not report any problem on any of the bcache device

I am using gentoo kernel 3.11.6.

Not sure it is related but the last operations before reboot were:
* extend the lvDownloads (without FS resize)
* erronously executed resize2fs /dev/bcache1 which resulted in a message that indicated that the device already had the maximum length.

Are there other "emegency actions" I should take (cache is till reported as inconsistent)?

Any idea what can be the cause of this corruption? Do you need more information to investigate the issue?

How can I fix it now?

Thanks!

Thierry

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hi,

Any idea? I would like to come back to a sane situation as soon as possible.

Thanks!

Thierry

Quoted 111 lines · Thierry

Hi,

After a reboot, the bcache devices didn't appeared. Looking at dmesg I foud out the following entries:

[ 12.182288] bcache: register_bdev() registered backing device dm-0
[ 12.217379] bcache: register_bdev() registered backing device dm-1
[ 12.253040] bcache: error on d8e791cc-47c2-449c-909a-1e4433a5de11: bad
btree header at bucket 32018, block 0, 0 keys, disabling caching
[ 12.253050] bcache: register_cache() registered cache device sdb1
[ 12.253488] bcache: cache_set_free() Cache set

d8e791cc-47c2-449c-909a-1e4433a5de11 unregistered

Here is my device map
NAME MAJ:MIN RM SIZE TYPE FSTYPE MOUNTPOINT UUID PARTUUID
sda 8:0 0 931.5G disk
├─sda1 8:1 0 150M part ext2 0ced49ec-a9ec-4711-95b6-8ce01153ee3c 2a20f8be-8162-48db-9e51-d97e8fde21ae
├─sda2 8:2 0 49.9G part ext4 / 79f36847-8365-46da-85a1-cbab5ea9e30a 59d37a51-5f05-4356-857c-1937b793e9b1
├─sda3 8:3 0 881.5G part LVM2_member Vzf2b6-NmgQ-5Zff-QKRS-puR8-Dlhz-XIkY3t e337497b-b1e3-479e-ba29-e1b203434e97 │ ├─vgDataNew-lvHome (dm-0) 254:0 0 50G lvm bcache ff11059e-0edf-428d-9657-37f255bac07e
│ │ └─bcache0 252:0 0 50G disk ext4 /home 66bf41ef-174a-4770-b9ef-d37ab06d3f23
│ ├─vgDataNew-lvVM (dm-1) 254:1 0 100G lvm bcache 9497600e-839f-4f59-862a-a7512cb59f0a
│ │ └─bcache1 252:1 0 100G disk ext4 /data/vm 5ed8126a-2bda-465e-bedb-61272d878fde

│ ├─vgDataNew-lvDownloads (dm-2) 254:2 0 50G lvm /home/xxx/xxx
│ └─vgDataNew-lvSwap (dm-3) 254:3 0 12G lvm [SWAP]
└─sda4 8:4 0 1007K part f6826e28-5dc0-4122-ba84-8ccaffd900fd

sdb 8:16 0 223.6G disk
└─sdb1 8:17 0 223.6G part bcache dff8e23e-7f9a-4a14-94a0-ec5b618a35c4 ba9624c2-7fb7-4cfb-8d8a-1da6ca429820
sr0

Here is the output of bacache-super-show

bcache-super-show /dev/sdb1
sb.magic ok
sb.first_sector 8 [match]
sb.csum C161466F71949DD0 [match]
sb.version 3 [cache device]

dev.uuid dff8e23e-7f9a-4a14-94a0-ec5b618a35c4
dev.sectors_per_block 8
dev.sectors_per_bucket 1024
dev.cache.first_sector 1024
dev.cache.cache_sectors 468858880
dev.cache.total_sectors 468859904
dev.cache.discard yes
dev.cache.pos 0

cset.uuid d8e791cc-47c2-449c-909a-1e4433a5de11

bcache-super-show /dev/vgDataNew/lvHome
sb.magic ok
sb.first_sector 8 [match]
sb.csum ADBDC19FD7011CAB [match]
sb.version 1 [backing device]

dev.uuid ff11059e-0edf-428d-9657-37f255bac07e
dev.sectors_per_block 8
dev.sectors_per_bucket 1024
dev.data.first_sector 16
dev.data.cache_mode 3 [no caching]
dev.data.cache_state 3 [inconsistent]

cset.uuid d8e791cc-47c2-449c-909a-1e4433a5de11

bcache-super-show /dev/vgDataNew/lvVM
sb.magic ok
sb.first_sector 8 [match]
sb.csum BFD4613F155DD0D7 [match]
sb.version 1 [backing device]

dev.uuid 9497600e-839f-4f59-862a-a7512cb59f0a
dev.sectors_per_block 8
dev.sectors_per_bucket 1024
dev.data.first_sector 16
dev.data.cache_mode 3 [no caching]
dev.data.cache_state 3 [inconsistent]

cset.uuid d8e791cc-47c2-449c-909a-1e4433a5de11

As you see I restarted the devices after having changed the cache mode to "none" followed by a echo 1 > running (The state of the 2 devices were dirty)
e2fsck do not report any problem on any of the bcache device

I am using gentoo kernel 3.11.6.

Not sure it is related but the last operations before reboot were:
* extend the lvDownloads (without FS resize)
* erronously executed resize2fs /dev/bcache1 which resulted in a message that indicated that the device already had the maximum length.

Are there other "emegency actions" I should take (cache is till reported as inconsistent)?

Any idea what can be the cause of this corruption? Do you need more information to investigate the issue?

How can I fix it now?

Thanks!

Thierry

#3 lore raw
In reply to a message not in the archive
Quoted 10 lines · Thierry

After a reboot, the bcache devices didn't appeared. Looking at dmesg I foud out the following entries:

[ 12.182288] bcache: register_bdev() registered backing device dm-0
[ 12.217379] bcache: register_bdev() registered backing device dm-1
[ 12.253040] bcache: error on d8e791cc-47c2-449c-909a-1e4433a5de11: bad
btree header at bucket 32018, block 0, 0 keys, disabling caching
[ 12.253050] bcache: register_cache() registered cache device sdb1
[ 12.253488] bcache: cache_set_free() Cache set

d8e791cc-47c2-449c-909a-1e4433a5de11 unregistered

Quoted 7 lines

As you see I restarted the devices after having changed the cache mode to "none" followed by a echo 1 > running (The state of the 2 devices were dirty)
e2fsck do not report any problem on any of the bcache device

Are there other "emegency actions" I should take (cache is till reported as inconsistent)?

I've seen the same problems in earlier kernels, but not in kernels > 3.11.5.

After also forcing the bcache device to running (echo 1 > running) I also recreated the caching device (make-bcache -C ...) and reattached it to the bcache device. Have you tried to do that?

Rolf

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

Hi Rolf,

Thanks for your answer. Do you know of a specific commit that was made to fix that issue or it just did not appear anymore? How frequently did you encoutner this issue?

I also thought of just reformatting the cache, but I would remove all evidences/traces that could maybe be useful to investigate the issue. So if I have no-one interested in some post morten on the cache device in the coming days, I will proceed with that approach.

There is also little tooling to trace, investigate or fix problems apparently. Is it something planned in the future?

What is also odd to me is that the GUID listed in dmesg does not seem to match any device (eventhough it is clear that the message is about the cache device sdb1)

Thanks!

Thierry

Quoted 32 lines · Rolf Fokkens
Quoted 10 lines · Thierry

After a reboot, the bcache devices didn't appeared. Looking at dmesg I foud out the following entries:

[ 12.182288] bcache: register_bdev() registered backing device dm-0
[ 12.217379] bcache: register_bdev() registered backing device dm-1
[ 12.253040] bcache: error on d8e791cc-47c2-449c-909a-1e4433a5de11: bad
btree header at bucket 32018, block 0, 0 keys, disabling caching
[ 12.253050] bcache: register_cache() registered cache device sdb1
[ 12.253488] bcache: cache_set_free() Cache set

d8e791cc-47c2-449c-909a-1e4433a5de11 unregistered

Quoted 7 lines

As you see I restarted the devices after having changed the cache mode to "none" followed by a echo 1 > running (The state of the 2 devices were dirty)
e2fsck do not report any problem on any of the bcache device

Are there other "emegency actions" I should take (cache is till reported as inconsistent)?

I've seen the same problems in earlier kernels, but not in kernels > 3.11.5.

After also forcing the bcache device to running (echo 1 > running) I also recreated the caching device (make-bcache -C ...) and reattached it to the bcache device. Have you tried to do that?

Rolf

Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-bcache" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw
Quoted 5 lines · Thierry De Leeuw

Hi Rolf,

Thanks for your answer. Do you know of a specific commit that was made to fix that issue or it just did not appear anymore? How frequently did you encoutner this issue?

I ran into this once, but haven't had this issue since. I can't recall a specific commit, but I think it was kernel 3.11.4.

Quoted 10 lines

I also thought of just reformatting the cache, but I would remove all evidences/traces that could maybe be useful to investigate the issue. So if I have no-one interested in some post morten on the cache device in the coming days, I will proceed with that approach.
There is also little tooling to trace, investigate or fix problems apparently. Is it something planned in the future?

What is also odd to me is that the GUID listed in dmesg does not seem to match any device (eventhough it is clear that the message is about the cache device sdb1)

When I ran into this issue I was mosty interested in recovering from a "broken caching device" which showed not to be that hard. I assumed I could gather more evidence if the issue would show up again. I can imagine Kent or Gabriel could be interested in some post-mortem analysis, but that's up to them.

Rolf

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw

Hi Rolf

Thanks for the info. I will still wait a bit for Kent or Gabriel to tell if and what info they are interested in (or maybe say that I can drop and recreate the device)

Is there any chance that some data is still in the cache that could be recovered?

Maybe an emergency tool that tries to read the data and apply it before restarting the devices could be useful.

Thanks again for your reply much appreciated!

Thierry

Quoted 26 lines · Rolf Fokkens
Quoted 7 lines · Thierry De Leeuw

Hi Rolf,

Thanks for your answer. Do you know of a specific commit that was made to fix that issue or it just did not appear anymore? How frequently did you encoutner this issue?

I ran into this once, but haven't had this issue since. I can't recall a specific commit, but I think it was kernel 3.11.4.

Quoted 10 lines

I also thought of just reformatting the cache, but I would remove all evidences/traces that could maybe be useful to investigate the issue. So if I have no-one interested in some post morten on the cache device in the coming days, I will proceed with that approach.
There is also little tooling to trace, investigate or fix problems apparently. Is it something planned in the future?

What is also odd to me is that the GUID listed in dmesg does not seem to match any device (eventhough it is clear that the message is about the cache device sdb1)

When I ran into this issue I was mosty interested in recovering from a "broken caching device" which showed not to be that hard. I assumed I could gather more evidence if the issue would show up again. I can imagine Kent or Gabriel could be interested in some post-mortem analysis, but that's up to them.

Rolf
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html