btrfs device delete missing - Input/output error
Quoted 6 lines
The first way to try to recover the current volume would be to overwrite LBA 2262535088 which you should only do with the filesystem unmounted. That's the sector causing the read error. If this is a 512 byte drive:
dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088
It is a 512 byte drive.
Unfortunately overwriting that fails - which is quite weird, given that there are 0 reallocated sectors, according to SMART.
Is there a way to determine what's stored there (i.e. if it's a file?).
# dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088 dd: writing to `/dev/sdb': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 2.88439 s, 0.0 kB/s
# dmesg -c
[ 8177.713212] ata4.00: exception Emask 0x0 SAct 0x400 SErr 0x0 action 0x0
[ 8177.713285] ata4.00: irq_stat 0x40000008
[ 8177.713349] ata4.00: failed command: READ FPDMA QUEUED
[ 8177.713419] ata4.00: cmd 60/08:50:b0:8b:db/00:00:86:00:00/40 tag 10 ncq 4096 in
[ 8177.713419] res 41/40:08:b0:8b:db/00:00:86:00:00/00 Emask 0x409 (media error) <F>
[ 8177.713662] ata4.00: status: { DRDY ERR }
[ 8177.713725] ata4.00: error: { UNC }
[ 8177.755099] ata4.00: configured for UDMA/133
[ 8177.755175] sd 3:0:0:0: [sdb]
[ 8177.755252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 8177.755318] sd 3:0:0:0: [sdb]
[ 8177.755380] Sense Key : Medium Error [current] [descriptor]
[ 8177.755449] Descriptor sense data with sense descriptors (in hex):
[ 8177.755516] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[ 8177.755600] 86 db 8b b0
[ 8177.755667] sd 3:0:0:0: [sdb]
[ 8177.755729] Add. Sense: Unrecovered read error - auto reallocate failed
[ 8177.757316] sd 3:0:0:0: [sdb] CDB:
[ 8177.757377] Read(16): 88 00 00 00 00 00 86 db 8b b0 00 00 00 08 00 00
[ 8177.757463] end_request: I/O error, dev sdb, sector 2262535088
[ 8177.757542] ata4: EH completeSignature · 2 lines
Tomasz Chmielewski http://www.sslrack.com
Quoted 22 lines · Tomasz Chmielewski
Quoted 6 lines
The first way to try to recover the current volume would be to overwrite LBA 2262535088 which you should only do with the filesystem unmounted. That's the sector causing the read error. If this is a 512 byte drive:
dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088
It is a 512 byte drive.
Unfortunately overwriting that fails - which is quite weird, given that there are 0 reallocated sectors, according to SMART.
Is there a way to determine what's stored there (i.e. if it's a file?).
# dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088 dd: writing to `/dev/sdb': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 2.88439 s, 0.0 kB/s
Quoted 4 lines
# dmesg -c [ 8177.713212] ata4.00: exception Emask 0x0 SAct 0x400 SErr 0x0 action 0x0 [ 8177.713285] ata4.00: irq_stat 0x40000008 [ 8177.713349] ata4.00: failed command: READ FPDMA QUEUED
You're getting a read error when writing. This is expected when writing 512 bytes to a 4k sector.
What do you get for
parted /dev/sdb u s p
Signature · 1 line
Chris Murphy
A message that is not in the archive (sent off-list or not yet received): CAJCQCtRh7HU9yd-pxgXzXebVcjukQh6Lyhzfcd6bBJppyCxChQ@mail.gmail.com
Quoted 5 lines · Chris Murphy
You're getting a read error when writing. This is expected when writing 512 bytes to a 4k sector.
What do you get for
parted/dev/sdbu s p
Right - so we're getting these:
# parted /dev/sdb u s p Model: ATA ST3000DM001-9YN1 (scsi) Disk /dev/sdb: 5860533168s Sector size (logical/physical): 512B/4096B Partition Table: gpt
Number Start End Size File system Name Flags 5 2048s 4095s 2048s bios_grub 1 4096s 67112959s 67108864s raid 2 67112960s 68161535s 1048576s raid 3 68161536s 2215645183s 2147483648s raid 4 2215645184s 5860533134s 3644887951s btrfs raid
Signature · 2 lines
Tomasz Chmielewski http://www.sslrack.com
Quoted 9 lines · Tomasz Chmielewski
Quoted 5 lines · Chris Murphy
You're getting a read error when writing. This is expected when writing 512 bytes to a 4k sector.
What do you get for
parted/dev/sdbu s p
Quoted 6 lines
Right - so we're getting these:
# parted /dev/sdb u s p Model: ATA ST3000DM001-9YN1 (scsi) Disk /dev/sdb: 5860533168s Sector size (logical/physical): 512B/4096B
OK it's a 4096 byte physical sector drive so you have to use the bs=4096 command with the proper seek value (which is based on the bs value).
Signature · 1 line
Chris Murphy
Quoted 10 lines · Chris Murphy
Quoted 6 lines
Right - so we're getting these:
# parted /dev/sdb u s p Model: ATA ST3000DM001-9YN1 (scsi) Disk /dev/sdb: 5860533168s Sector size (logical/physical): 512B/4096BOK it's a 4096 byte physical sector drive so you have to use the bs=4096 command with the proper seek value (which is based on the bs value).
Yep.
So here is what I did:
# echo 2262535088 / 8|bc # (512 * 8 = 4096) 282816886
* verify if it's this place
# dd if=/dev/sdb of=/dev/null bs=4096 count=1 skip=282816886 dd: reading `/dev/sdb': Input/output error 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.83002 s, 0.0 kB/s
* overwrite it:
dd if=/dev/zero of=/dev/sdb bs=4096 count=1 seek=282816886 1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 3.0277e-05 s, 135 MB/s
* try to read it:
# dd if=/dev/sdb of=/dev/null bs=4096 count=1 skip=282816886 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 2.6204e-05 s, 156 MB/s
* try to read with the old skip value (repeat for 2262535088 - 2262535095, or use a different count value - this is where we were getting the errors, it's also 8 * 512):
dd if=/dev/sdb of=/dev/null count=1 skip=2262535088
...
dd if=/dev/sdb of=/dev/null count=1 skip=2262535095
* Unfortunately this is still an error for btrfs, because the checksum does not match:
# time btrfs device delete missing /home ERROR: error removing the device 'missing' - Input/output error
# dmesg -c [84200.109774] BTRFS info (device sda4): relocating block group 1375492636672 flags 17 [84203.635559] BTRFS info (device sda4): csum failed ino 261 off 384262144 csum 2566472073 expected csum 4193010590 [84203.650980] BTRFS info (device sda4): csum failed ino 261 off 384262144 csum 2566472073 expected csum 4193010590 [84203.651444] BTRFS info (device sda4): csum failed ino 261 off 384262144 csum 2566472073 expected csum 4193010590
* remounting with nodatacow (nodatasum) does not help (since it's for new data)
* let's find the inode printing the error - turned out to be a directory - so created a new one, moved the data from the "corrupt one", removed the directory with that inode:
# find /home -mount -inum 261 /home/backup/ma-int/weekly
* repeat for any other found:
# dmesg -c [85197.300494] BTRFS info (device sda4): relocating block group 1375492636672 flags 17 [85200.448713] BTRFS info (device sda4): csum failed ino 267 off 384262144 csum 2566472073 expected csum 4193010590 [85200.472581] BTRFS info (device sda4): csum failed ino 267 off 384262144 csum 2566472073 expected csum 4193010590 [85200.473551] BTRFS info (device sda4): csum failed ino 267 off 384262144 csum 2566472073 expected csum 4193010590
* unfortunately it never ends - let's say we have /home/backup/ma-int/weekly, which we "fixed" with:
mkdir /home/backup/ma-int/weekly.tmp
mv /home/backup/ma-int/weekly/* /home/backup/ma-int/weekly.tmp rmdir /home/backup/ma-int/weekly
After we run again btrfs device delete missing /home, the newly created directory eventually (/home/backup/ma-int/weekly.tmp) is being detected as "csum failed ino ...".
Signature · 2 lines
Tomasz Chmielewski http://www.sslrack.com
Quoted 3 lines · Tomasz Chmielewski
After we run again btrfs device delete missing /home, the newly created directory eventually (
/home/backup/ma-int/weekly.tmp) is being detected as "csum failed ino ...".
Unmount the volume and run btrfs check, without repair:
1.
If that comes up clean then chances are this is file corruption and you can mount and do:
btrfs scrub start -r <MP>
This will show you filenames for corrupt files you'll need to locate backups to restore from eventually. Then unmount it again and do:
btrfs check --init-csum-tree <dev>
This will remove the current checksum tree and calculate new checksums (only very recent btrfs-progs does this, I tested with 3.17.3), so any files that are corrupt will no longer be flagged as corrupt, and you should be able to remove the missing device and resume the balance needed to restore normal operation for the mirrored volume. If you don't have recent btrfs-progs, the initialization of the csum tree will strip the csums entirely, and you'll need to recreate the file system (or live without checksums).
2.
I'm going to bet btrfs check doesn't come up clean, based on the additional problems you reported. In this case it's not file corruption, but filesystem corruption. Even though the mirror isn't intact, there's a possibility this small hole in the file system can be repaired with very recent fsck patches from Josef. You'd have to check whether the patches he's submitted in the last month or so to the list have been incorporated into btrfs-progs integration branch; and build a btrfs-progs that has them. Then use --repair and cross fingers.
Certainly before 2. and probably even before 1, you ought to go ahead and get anything off the volume you care about, seeing as right now it's at least mounting. It'd be good to know either way if it works. And if it doesn't work, take a btrfs-image of the fs, Josef might be interested in testing against it seeing as only a 4KB hole was punched into the file system.
Signature · 1 line
Chris Murphy
Quoted 10 lines · Chris Murphy
Quoted 3 lines · Tomasz Chmielewski
After we run again btrfs device delete missing /home, the newly created directory eventually (
/home/backup/ma-int/weekly.tmp) is being detected as "csum failed ino ...".If that comes up clean then chances are this is file corruption and you can mount and do:
btrfs scrub start -r <MP>
Actually, this seemed to be enough.
I didn't realize that scrub would print the names of corrupted files.
After removing them, "btrfs device delete missing" is finally making some progress (previously, it would break at 300+GB; still running now):
# btrfs fi show /home
Label: none uuid: 92e93437-cc9f-475d-a739-085f3270896b
Total devices 3 FS bytes used 1.40TiB
devid 2 size 1.70TiB used 1.42TiB path /dev/sdb4
devid 3 size 1.70TiB used 807.00GiB path /dev/sda4
*** Some devices missingBtrfs v3.14.2
Thanks for your hints!
Signature · 2 lines
Tomasz Chmielewski http://www.sslrack.com